There is a vacuum servo on the driver's side of blower case that controls the diversion of air between defrost and floor. It has vacuum applied to the floor servo (yellow hose) whenever air is directed out of dash vents and this side of doulble diaphragm has a seal on the servo rod that breaks and leaks the vacuum away. When you are on boost the vacuum available from engine is nil and the contained vac leaks away and the system goes to floor/defrost where the air flow is much lower at the same blower speed.

You have two choices: 1) replace the servo at considerable $$$

or 2) disconnect the yellow hose and have compromised floor airflow when that is selected and buy electric hunter's socks and a lot of spare sock batteries and take a nice vacation with the remaining savings.

You decide.