On close inspection it looks like what I thought was just a little mud spray on the passenger side door and front fender is in reality something like white paint or plaster. It looks like the previous owner drove through a puddle of paint (for lack of a better term)

it's really hard to notice, but I'd like to take a shot at removing it. But, any solvent I use will get the car's paint as well. I'd like to try rubbing compound, but I'm afraid of collateral damage.

Is there any other thing I can try? (short of a trip to a body shop) I'm tempted to just ignore it.