Hmm.. On mine the only one marked with yellow goes to the sending unit. If that was yours, unplugging it would not allow your car to run for more than a second or two (if there was residual pressure in the line, enough to start it but not keep running)

Another wire is for fuel level sensor, if you unplug it, does your fuel gauge still work? If it does, do you have any codes? If you get a P0450 the wire you're unplugging is for the fuel pressure sensor.

Now as to why your car doesn't start when its plugged in, I'm going to assume perhaps you've got a short/open somewhere and the current is going elsewhere, and not to the pump. Then again, it might have been a bad connection in the fuel pump wire and the movement gets it to make contact, thus allowing you to run..

That's enough speculating for now...
--
If you're not driving it "like its stolen," are you really driving?