I haven't read up on the 3VZ specifically, but I'm imagining it's set up like the 22RE. On that, from what I've read, the physical timing is only used for startup and if some part of the ignition system fails, otherwise the timing is set and controlled by the computer (obviously within the limits of cap/rotor contact). I don't recall if the 3VZ has something like the timing jumper that the 22REs have, where it sets control of the timing back to the physical side so that you can adjust it accurately, but it wouldn't surprise me.
I know on my wife's car (03 Hyundai), the timing behaves like you describe when I stick a timing light on it. It has no distributor at all, and everything is controlled by the computer, so I'm assuming I'm just watching the computer play with the timing.
Or I could be completely off base with all those theories, but I do think it's normal behavior.
I cannot say with any certainty - either way because I have read a bazillion posts and it has been said both ways. From my own deduction i surmise that this engine uses a Base + timing system. The distributor does in fact have 3 hall effect pickups in it. There is a computer diagnostic port that you "jump" that turns off the computer control while you set the "Base Timing". And I have learned from messing with it, that your base hard setting will effect the engine running. So I assume it is in fact a base plus system.
That being said, this engine also uses a knock sensor that the computer uses to retard the timing if you are pinging or knocking. I don't believe that a proper tuned engine should have a bouncy random spark when the computer timing is bypassed. And I have not seen this on my other Toyota's save my old 2.4 but that unit needed a timing set. And it was the same hot or cold. Don't really know, all I know is that I don't have any low end torque at all. And It just does not get up and go like it use too. I can see all this on an engine with 250-300K on it but at 170K she is still just a teenager!