Power assist brakes run off engine vacuum. Perhaps you have a vacuum leak in the power brakes? Can you consistently influence the idle with the brake pedal? If just touching the brakes affects the idle - i.e., it has an effect like and on/off switch, this could be attributable to electric power draw for the brake lights, and weak ignition components. If you can control the idle with varying amounts of pressure on the brake, then I would look at vacuum.

If the brakes don't consistently affect idle, then I would look elsewhere - possibly MAF, FPR, IAC, leaky vacuum lines, etc. The IAC might not operate properly under braking because I believe that the valve opens and closes fore/aft, and momentum while braking might throw off the operation of a weak valve.