That's exactly what you do. I just did it myself about a week ago. Here are some instructions on removing the door panel:


Re: Driver's window [S70]⏮]
Believe it or not the V70/S70 panel is very easy:
1. Using a tiny flat blade screwdriver, pry the small plastic cover located in the hand well off.
2. Remove the torx screw located behind the cover
3. Carefully pry off the triangular plastic cover located directly inside from the rear view mirror.
4. Carefully, pry the retangular ring around the interior door handle out(you may have to wait until step 5 is partially complete)
5. Using your fingers only, carefully pry the door panel, starting at the bottom away from the door frame--it is secured with nylon door clips--they shoud snap out at remain attached to the panel. If you were unsucessful with the handle trim, it should now be beginning to pull away from the "cup behind the handle" and you should be able to pull/slide it out.
6. Once the panel is several inches away from the door frame, reach inside and unplug the pin connectors attached to the trunk/gas switch and the three connectors attached to the door lock/window switch. Also unscrew the lock knob.
7. Continue to pry the panel away until all of the nylon clips are clear of their seats in the door frame.
8. Carefully and slowly, lift up on the panel--the top of it will pull free of the recess that it fits in--it can be a bit stuck, but should come free without too much hassle.