The booster uses engine vacuum, not pressure, to help the driver apply less foot pressure to the master cylinder. When there is no vacuum, the pedal pressure to activate the brake calipers becomes very difficult - like my 1967 that had the booster removed.

Is the sound you hear always present when the engine is running or only after engine shutdown? If it is always noisy during engine operation, than a seal is probably worn or deformed at the firewall end, and should be fixed one of these days. If it is only noisy after the engine is shut down, don't worry too much.

Klaus
--
Always willing to listen, just not able to take direction.