There are two types of sealant, one silicone and one more of a paste. Neither are permanent, the silicone melts away and the paste flakes off after a while. They work for short term repairs or for smog issues.

It sounds like you have rust deposits on one or both of the flanges that is preventing them from sealing up, have remnants of the old doughnut gasket, or got gaskets that were made wrong. I would take it apart and see what it looks like, pick up a new set of gaskets and see if it doesn't seal up.

When you have the down pipe off, make a template of the flange, to use to make a paper gasket in the future, if nothing changes after pulling it apart. You can pick up high temp sheets of gasket material and cut your own gasket, they run about $8 for an 8x11 sheet. This is what I had to do for the supra I am restoring, after making my own turbo elbow, since I don't have the tools to machine a recess for the doughnut gasket. It seems to be working so far.