You can check it with a voltmeter but you need a voltmeter that is meant to test auto computer systems. It has to have at least 10 megaohm resistance if I remember right to protect the 02 sensor and the computer.. A good O2 sensor should be putting out about .4 volts when hot, assuming the fuel mixture is right. If there is no voltage the sensor is toast.
But you don't really have to do this. Your car has onboard diagnostics, and a non functioning O2 sensor generates an error code. I don't think it casues the check engine light to come, so you have to manually check the code reader.