I have not polished a lot of aluminum, but have done a lot of wood finishing with clear finishes. I believe using too high a speed buffer on soft aluminum may be part of the problem. I would buff by hand, (I know not what you want to hear), and if possible using succesivelly (I know I spelt that wrong!) finer grades of polishing or buffing compounds.
There are some really good metal polishing products two are "Flitz" and "Simecrome". also somepeople swear by "Neverdull" which is a cotton wadding that is treated with? it comes in a can. (I am not sure it works on aluminum the label will say, It works great on brass.
You could always start with 0000 steel wool and some soap or wax and then go to one of the finer grade metal polishes.
It is a long process, do a little at a time and listen to a game or watch TV.
...I just thought of this ....If you are trying to polish something that has a grainy pattern or finish, it'll take forever.
Hopefully I have been of some help.
CU