I can't recall which end was the 'big' end in mine - the numbered end or not. But in my case they slid in relatively easily over 90% of the length, and had to be tapped in the last bit. It might also make a difference which end of the case you are trying - I suspect those holes might not quite be the same size either. In other words the holes might be large and small to match up to the shaft large and small ends, so that it is snug on both ends when driven in that last inch or so.

So try both ends of the shaft in both holes on the case. If the theory is correct, only 1 out of the 4 will work.