Yes I see that I was thinking of the M45 that is in my 78. The 1980 body of your yellow car got my mind stuck. I did not know that you have an 84 transmission in there.

According to Bentley, the M45 became a M46 in 83.

I see #47 as a paw lever that passes in front of the switch. I see a flat spring #53 that may hold that paw back. If the switch was screwed down in-line with the spring it would make a better transition under the plunger for less wear. Unfortunately it appears they went for a side ways or horizontal mounting. That might have worn the plunger. Add the gasket, which moves the switch to far back, points to a narrow range of movement to close the switch.
I understand your frustration but most likely you have found an engineering/production flaw. They missed that dimension and tossed the washer.

The other part about the spring pulling the switch open is a rip!
The shift rod rotation must be wiggling the spool behind the gear. That groove that the lever fits down into must barely let the lever shift back under the pressure of the plunger.

The only outside mechanical solution is remove the washer, lengthen the plunger (who knows how) or if you have a lathe, face the shoulder back on the switch to allow for the copper gasket, provided there is enough body casing.

I would go chemically and put sealant on it.

Phil