The starting circuit is very complex and the wiring diagram shows a number of
black boxes. You need the Volvo theory writeup for the circuit which is somewhere, but not in the wiring diagram.

My best guess is the Ignition switch module since it has several black boxes
indicated, maybe one is the key delay function? The ignition sw module depending on your model V50 is almost $200 at the first place I checked.

There's a Keyless module KVM under the drivers seat if it's an auto tranny
or contact immobilizer if it's a stick shift , There's an engine control module
special to various model cars and a tranny control Module Ach!!!!
It's not clear where the delay function lives.

Unless you can find a donor car in a boneyard, this is not the kind of problem
that you throw parts at. It can get increasingly ugly for your wallet.
$200 for an ignition switch! I need an airsick bag.

I'll ask my local factory trained Volvo Maven what he thinks.

Bill