Maybe the EGR valve/modulator/BVSV is acting up? Mybe try plugging the vacuum line to the EGR and see if anything changes. The BVSV opens once the engine warms up to start the EGR modulator working.