I too, managed to destroy a couple of those spacer. I was lucky enough to have a couple spares.

What I've successfully done this several times is this:

1) Use a large flathead screwdriver to wedge the spacer away from the handle.
2) Use a long, strong, thin tool to push the clips out
3) Handle will pop off.

The tool is the key and it helps to have the proper tool, but you can get by with one of those odd-looking putty knives (the one with a big circular cut out of the side and a triangular corner on the other side.

A cheapo one, from your hardwares store's "cheap chinese tool" bin is better because it will be more flexible. I useed the triangular point to push out the clip, and later I tool the tool to a grinder to make more of a hook on the curved side.

The result works okay, but it's hardly fool proof. It's better than the stiff wire and pliers I used to use. I can take a photo of it if you like.

You can find those clips in the "HELP" section of your local hardware store, so don't freak if you lose one.