Hi Bikergal,
I just went through the exact same thing, only mine was the driver's side and it was dangling in my face! Do you have the lighted vanity mirror? If so, it ain't cheap. I called around to dealerships and nobody had the part for less than $90, so I got tricky and ordered it for $77 through a friend who has a shop. Here's the problem though, I still had to take it to someone who knew what they were doing, because my friend never replaced one on a V40 and said the instructions involved popping off the padded pillar and pulling some wiring through to connect the mirror light. If you don't pop off the pillar correctly, the clips holding it could break or something, then you'd have two problems instead of one, and since he wasn't Volvo familiar, he didn't want to mess anything up. So I left the old broken one on and took it to the dealership, along with the new mirror still in the OEM box. I had just bought the car so I was pissed, not just because the part cracked & broke, but because it was a serious safety hazard while driving. I complained to the dealer about it and asked if it was included in the warranty (it had been a certified used Volvo w/the V.I.P extended warranty). Here's the other tricky part, if you have the warranty, you have to word it just right according to the dealership to get it covered, they know how to do this. I guess the dealership wanted to promote good relations, so they told the ext. warranty dept. that the visor would not stay in place (true!)and had an internal fault in the lighted mirror(technically true, but with a 'twist'). The lighted mirror actually worked fine until I had enough of the thing dangling in my face, (great idea you had about the velcro... I taped mine and it wouldn't hold long) so I unscrewed it, and as I attempted to remove the damn thing, the electric lead to the light came out. The way it is designed, there is no way to get inside to reinsert the metal contact (and the lead wire does indeed go up into the pillar). The dealer techs had to realize that it didn't just come undone on it's own, but they've seen problems with these before and they got it covered for me, including a credit for the prepayment of the visor. Without the warranty, the cost was $102.38 including labor. If you don't have the warranty but you're on the handy side, maybe you could find a V40 or an S40 in a junkyard; you might be able to diddle with the pillar to see how it comes off and where the wiring is. Then, you could just order the mirror and do it yourself.

p.s. Make sure the dealer gets the color code of your mirror right, mine was the wrong color the first time.

Good Luck!
Mo