That is not an unusual problem - mine does the same thing. A thread exists somewhere on this forum where the problem was eliminated by wiring in a relay for the starter circuit - apparently the factory wiring has the ignition switch handling all the power to the starter solenoid and it becomes less capable of sending full power as contacts wear and connections age. I don't think the problem is your neutral safety switch - my experience has been that they either work correctly or they fail completely.

Play with the search function looking for threads about ignition switches and you should come up with it.