Missing ball or weak spring would make selection of 3rd or 4th pretty indistinct (doesn't "click" into gear, although much of the "click" is the dogs engaging the gears) and shifter would feel like it wanted to slip.
Bad bush on tranny cover (for lever) would make all gears pretty difficult to get good engagement (and you may bash hand on dash if really bad).
Ditto for bush on bottom of lever.
Worn dogs/retaining spring on the output shaft on which both 3rd and 4th ride is a likely culprit. Freshening the tranny (once you are home) is the only fix for that.

Ugh, I'm explaining this poorly.

Until then, holding the gear lever by the PO would have worn the selector and fork much more than the amount of wear it will get on the way home. Draining the oil and inspecting the bung plug for filings will give you an indication of level of wear. Suspect you will find lots of brass in there from the syncro cones as well.

No loud rumblys means you likely have a good rebuild project, especially if no issues with 1/2 (most common gears to lunch) as the layshaft is probably OK.

Popping the cover off the "spare" will at least let you take a look at what you have there. Take tools and oil for the possibility of a roadside swap if all goes "bang." Shouldn't, as the boxes are pretty darn robust.

--
Mike!