Seems like if you don't put it into 4wd for a while either the switch, the relays, or something else does not always make contact. When you put in in the second time it either freed up or the corrosion was rubbed off the contact. Mine was doing the same thing and now I try to shift it in at least 1x a month and no problems since.

1997 T-100 4X4 Xcab,Warn Hubs
02 Camry LE
2008 Yaris