Might be the motor as well, worn brushes perhaps:
- http://www.4crawler.com/4x4/CheapTricks/Blower.shtml#MotorRepair

Since the switch is probably the hardest thing to get to, I would probably check everything else (motor, resistor pack, relay) and if all that is OK, then the switch last.