I almost got stuck at work but after a half hour of trying it cranked over and started. The next day in the garage I started troubleshooting. The next day it would not crank at all making it easier to troubleshoot vs an intermittant problem. The battery checked out fine. The click under the hood was not coming from the starter solenoid area. I checked and cleaned the small wire going to the starter. Still no good. I hooked a voltmeter between small wire for starter and ground. When I tried to start it no voltage. I checked the antenna ring to make sure the connections was seated properly. No change.

I looked over the wiring diagram and noticed the starter motor relay located in position FMA4 in the fusebox under the hood. I swapped the starter motor relay with the engine management system relay position FMA2. When I tried to crank the car I got 12+ volts on the voltmeter. I swapped back and no voltage. I hooked the small wire with spade connector back to the starter. With the swapped relay it would immediatley crank over. I swapped back to the original relay and no crank. I borrowed the starter motor relay from our 07 XC90. Once intsalled the S80 started and ran fine.
I was lucky the bad relay acted up long enough for me to troubleshoot. After bringing the bad relay inside and shaking it a bit it would also work.

I'm off to the dealer today to buy a new relay.