ATF or Motor Oil? ATF expands seals as part of its function. Perhaps the material, though fine with oil, does not react well with ATF and expands too much.
If you true the outer cover, it will not leak with the oem O-ring, no sealant needed.
Oops - disregard, you said transmission; I was thinking chaincase. (I couldn't delete this post - sorry)