IIRC, there is a ground wire on the backside of the passenger fender that runs from the passenger fender ground bolt down to the frame, where as there isn't one on the drivers side, this is likely why the voltage drops., and is why when adding accessories grounding them to the body usually ends up w/them not working, or not working correctly.

You could have a engine to body/frame strap that isn't connected, from past engine work, or you could have a degraded ground wire in a harness somewhere, this happened to me. If you can't find one on the drivers side fender, you can simply add one. Run it up from the frame to the coil ground(on the drivers fender), this should give you the same 12.3 volts on a bolt on the drivers fender as it does n the passengers.

I would start with cleaning the terminals, the ground strap ends, and check that all of the ground straps are attached, then add additional straps.