The fuel filter is back by the fuel tank, just in front of it on the inside part of the frame. There should be a bracket holding it on.
I wouldn't think a replacement fuel pump would cause the filter to go bad, but you never know.
The cold idle diaphram (do you mean AAP diaphram?) might be part of it, though I would think it would run OK warmed up. When mine went bad I had trouble idling when cold.
You can pick up a carb rebuild kit cheaper than getting that one part from Toyota (or used to be able to) and it will have one in there.

1986 Pickup extracab longbed "Woody edition"
SAS - 33's, 5.29's, lockright rear.

02 IS300 SportCross - my "DD"
94 2wd v6 extracab
04 Outback Wagon- wife's/family car