I am wondering could it be a fualty voltage regulator?

That is possible. The nominal voltage of a 12 V battery is...well...12 V, but the charging voltage should be around 14.4 V max. Typically there will be around 13.5 V when the engine's running.
Without voltage regulator (VR) the alternator would put out varying voltages depending on revs and load. The VR performs the task of stabilizing the output. That means keeping the voltage close to 14 V at the B+ terminal (from just above idle-revs to max. revs).
The reason this has dropped to 13.5 V at the battery terminals is wire resistance, but 13.5 V is still good enough to keep it charged.
If the VR doesn't work properly, it could probably under- or overcharge a battery.

...or only very dimly lit while running

That should only happen with some electrics switched on (e.g. lights and fan). When everything is switched off, I would not like to see the AMP-light at all.

I was under the impression that the regulator was built-in or is on the alternator?

That is possible, but in the early days of alternators they had an external VR. These were mechanical and too big to fit inside the alternator housing. Nowadays they are solid-state items small enough to go in there. My 1970 Amazon has the former type of alternator.
When the alternator has an external VR, you will see at least three terminals (B+, 61 and F), and there will be a device mounted in the vicinity that looks like a big relay.

I dont have a multimeter or anything to check voltage with, tho i will have to pick one

Buy a digital one. Much easier to use than an analogue multimeter. An auto-range multimeter is even easier to use. A (digital) multimeter usually also has the possibilty to measure resistance, AC volts, current and continuity.