Have you tried pulling the handbrake as it's making this noise in reverse? If the noise changes or goes away, you'll have isolated it to a brake issue. My guess is that you'll find the problem in those rear brakes.

Not needing a puller to take the drum off is not good.. it means it's not tight. Could be a worn drum where it mates with the axle, or the axle itself, or a combination of both, or maybe the surfaces are not clean or they are rusted. If you continue to drive it like that, you will ruin that woodruff key, as well as the slots that accept it in both the brake drum and the axle.

Make sure the mating surfaces are clean, and get a longer wrench for tightening the castle nut.

--
-Matt I ♥ my ♂