Y'know something? Just try to drive it with the gas cap loosened for a few miles and see what it does;
If you drive too long with the cap loosened, that will turn on the check engine light and set error code P0440. The ECU via the evap system measures tank pressure and expects a slight vacuum to be pulled on the tank. The ECU will set the above error if the vacuum is not present after 20 minutes of engine run time.
The vacuum sound you hear when the cap is loosened is the slight vaccum pulled on the tank by using engine intake manifold vacuum via the evaporative emission control system.. This is normal operation.
Normal pressure developed by the in tank fuel pump is 38 - 44 psi per the FSM. A fuel rail pressure check to verify the fuel pump and pressure regulator are OK wouldn't be a bad idea due to the following.
Unless Toyota has added a fuel pressure (rail) sensor in later models, there is no sensor to measure and detect low fuel rail pressure.. So you could have this problem and error codes might not be set. OBDII codes P0190 - 194 are defined for this purpose, but not implemented on some (if not all) Toyotas. It would seem to me that it would at least set a P0171 (running lean) code however.
Had an '85 Cressida 5M that would occasionally loose power w/o explanation.. also experienced difficult starting in cold weather, which I later found out to be caused by low fuel pump pressure. The in tank fuel pump eventually failed..
One other situation I've experienced but haven't mentioned involves the ECU itself.. If for some reason the ECU memory were to be loosing its stored long term fuel trim value, it will definitely stumble and lose power similar to what you described..
I have experienced this occasionally, when I've been working on the vehicle and had the battery disconnected (thus erasing the long term trim values stored in the ECU).. I've done this many times (due to welding, numerous electrical mods, etc). Every so often when I restart the engine and attempt to drive off after having had the battery disconnected, the engine has no power even with throttle way down similar to your description. After the engine stumbles along for 15 seconds or so, it eventually gets enough info to recompute the long term trim and begins to run OK.
The following suggestions are easy to do, and probably not the problem, BUT worth checking....Pull the EFI fuse (it's a 20amp fuse on my 2000 4Runner) and check just to make sure it's in good condition and reseat it a few times to make sure the electrical contacts are clean and making good contact. Power is always on to the ECU memory via this fuse (even when vehicle is not running) so the memory is kept alive with the long term trim values.
I have experienced low voltage fuses that have been overloaded and running hot to only crack the fuseable link instead of blowing the link metal away and they will continue to supply power, but cause strange, intermittent problems.. so while you have the fuse out, look at it very closely, looking for any imperfections or darkening along the link.
You might also pull and reseat the MAIN EFI Relay.. This relay supplies power (from the same 20amp EFI fuse mentioned above) to the fuel pump.. Any intermittent in this circuit would cause the fuel pump to lose power.
And check your battery terminal clamps to make sure they are clean and tight.
btw, if you want more info what the "long term trim" values mentioned above do,
read this article... Lastly, I'd start asking around and see if there isn't a dealer somewhere near you that has a hot shot OBDII guy that can figure this out..