There is also a filter on the brake booster, on the other (non-engine) side of the diaphrame. It's described in the Volvo workshop manual. You remove the master cylinder, disconnect the booster from the pedal, unbolt the booster, and pull it away a bit. Then you just slide out the filter from where the pedal connects to the booster (in the center). I'm typing this from memory of what I've read last weekend, but should be close to accurate. Perhaps your filter is blocked, which means vacuum on the engine side will result in vacuum on the other side, thus no booster effect. Then the filter suddenly opens and allows air in to move the diaphrame which assists braking. How does that sound to you?