Jeff,

You're onto it.

Proper gasoline combustion isn't an explosion, but a controlled burn that happens over a short amount of time. We want the resulting pressure to peak when the piston is a few degrees past top dead center (TDC) and on its way back down. In order for that to happen, we need to spark the burn before top dead center (BTDC).

If we spark it too late (retarded timing), we don't develop full pressure (in fact, the burn snuffs out before it's complete) and we don't make good power or get good economy. If we spark it too early (overly advanced timing), we develop too much pressure, and the fuel destabilizes and explodes violently. Octane rating is a measure of how much pressure the fuel can stand without that happening.

Such explosions make a sound ranging from a mild, rhythmic ticking (pinging) to a less rhythmic diesel-like clatter (detonation). At the pinging stage, you're wasting energy but not really hurting much, except for overheating the combution chambers, which promotes detonation. Detonation breaks things pretty quickly, so must always be avoided. (Diesels run on detonation, as you surmised, which is why they're so ruggedly built.)

Because the piston goes by the optimum point in less time as engine speeds increase, we have to spark off the charge earlier as RPMs go up... to a point. The difference between the idle RPM timing (let's say 15 degrees BTDC) and the high RPM timing (let's say 35 BTDC) is the amount of timing advance (in this case, 20 degrees). This gets more complicated than I care to get into right now (burn rate isn't a constant by any means), but that's the basic idea.

A high-compression engine operates on higher pressure than a low-compression engine, so it needs fuel that's more resistant to detonation. The correct timing is the correct timing -- you don't get any advantage from advancing it beyond that. If the fuel you're using can tolerate the pressures developed with the correct timing, there's no reason to buy a higher grade.