Sounds more like the rear axle is loose since it's all good in 4x4 and all over the place in 2x4. How does it feel when it pulls to the side, is it a definate pull from the front or like the whole car is turning? When you put it in 2x4 there should be no power going to the front wheels so there shouldn't be any power to pull it to the side.

1987 SWB Pajero 2.5 TDI on 31" rubber waiting for a hip replacement and bigger shoes