I have the same exact thing and I believe that it is the starter contacts. It always eventually starts so I have been putting off checking it out but someday I will take the starter out and replace the contacts I think that it ends up being pretty cheap. So check that out but also I had the clutch button on the floor behind the pedal pop out before so that it didnt recognize that the clutch was depressed when trying to start, I guess make sure that is alright too. Good luck.

98' Toyota Tacoma SR5
Manual Xtra Cab, Slight Lift
Maine Rules!