You could try pressure testing the cooling system. Put a pressure gauge on the radiator and look for the pressure to rise with changing engine RPM as it warms up and runs rough. That could indicate that engine compression is getting forced into the cooling system. Or you can get a chemical test kit at NAPA, for example, to test the coolant for exhaust gas contamination.