The extra info helps a lot.

Here is my long-winded 2 cents worth.

Since you've clarified the wiring, and the problem occurs even with it off, power consumption of the stereo gear shouldn't be an issue here. I thought it might be overloading the fuse if it was connected to the light circuit.

Your 97 should be close to my 2000. In that case, the only things on the turn signal fuse are the turn signal lights themselves, the indicator lights in the dash, and the relay that causes them to flash. The relay is used for both left and right lights, so I'd discount that as a problem.

First, I'd check in the areas where any stereo installation work was done to make sure no wires were inadvertently damaged (like cuts to the insulation, a wire being pinched between something) and that your stereo connections are really correct. (Make sure there are no connections that would be grounding the +12v line to either the turn signal lights themselves or the indicators in the dash). Since you said your new gear is connected either to the battery or to the harness for the factory gear miswiring to something unrelated like the turn signals doesn't seem real likely though. I suppose you could completely disconnect your connections to the battery and the other equipment from the factory harness and see what happens.

It sounds to me more like a short from +12V to ground in the wiring to the left turn signal lights somewhere downstream of the switch. It may be just a coincidence that it happened after your install. The temperature related aspect is sort of strange, although things expand and contract and get softer or harder as the temperature goes up and down that could have something to do with it. (A pinched wire might happen only when some part of the vehicle expands, eventually chafing the wire enough for it to always short out for example.)

I wouldn't replace the turn signal switch until I knew more. That sounds like an expensive shot in the dark.

The actual turn signal part of that switch on the column does nothing more than supply +12v to either the right or left turn signals and indicator lights. The hazard switch also turns on the turn signals (both sets at the same time). You could try turning that on and see what happens - if that fuse blows as well, it points even more strongly to something besides the turn signal switch. I suppose the short could be in one of the light housings, so you could look at the left side turn signal lights to see if something is loose or there is a lot of corrosion. I've never had a bulb short out, but I suppose its possible, so look at the bulbs as well.

If none of this works, I don't have any good suggestions for easy things to do. If it comes down to finding a short somewhere, without some special test equipment I think its a matter of trail and error to try to disconnect the wiring at the connectors that are available to narrow down where the problem is. I wouldn't try that without the wiring diagrams - tracing wires is just about impossible otherwise. You'd also need a meter to check resistances at the minimum.