Right, essentially when the windows are down you're creating a low pressure system in the car versus outside. The low pressure in the car "sucks" air in through the vents. It is especially true if you open the moonroof partially (or all the way). No air comes in through the roof but you'll have a better suction affect through the vents with the AC off.

Happens in most cars. I like it in my Integra in the spring since the air (even with the AC off) is cooled if set in cold mode or warmed if on the hot side of the temp slide lever.

Alberto