A worn idler will definitely introduce play into the steering and can cause the darting you're describing.

The play in the idler is vertical. You can just push up on it from underneath and see/feel the shaft move up and down in the housing if it's bad. I used to get a good 1/4" of vertical movement in mine. That movement lets the passenger side tire wander a little on it's own. And because of the angle on the tie-rod, the more your t-bars are cranked, the worse the wander is.

Caster & camber shouldn't be affected all that much, but I can see setting toe being a bear with slop in the idler.

--Dan