Same issue here. On mine, the two plastic spacer washers on either side of the pivot rod had broken. I remember staring at it for awhile and deciding that there were too many small pieces that would have to be removed in order to replace the washers, so I decided to live with the slop.

To take the whole center console off (this is from very foggy memory) There are a couple screws inside the storage area between the front seats...I think you have to pull the cupholders out then get at them from above with a long screwdriver. I think there are also some other screws that you can access by removing the ashtray. You will probably have to move the shifter and ebrake through a few different positions as you pull out the console. Just keep taking out screws and then gently wiggling it to try to find the next screw...you'll figure it out.