Phantom Sounds Randomly Appear During Long Silent Gaps

I use Adobe Podcast to clean up and enhance my clients podcast files and it has been an amazing service to offer our clients. If there are multiple speakers, we process their audio files seperately to ensure the software isn't confused by the multiple voices during overlaping conversations. We've noticed phantom sounds randomly appearing during long silent gaps in the audio when the one speaker we're processing is listening to the other speaker. We have to go in and cut those bits out which adds an additional step to the workflow.
 **No, it's not picking up the other speaker at a low level and trying to enhance it, these audio files are clean and completely isolated from each other.**

I know ai processing is being used to clean up the audio by replicating the speakers voice to help improve the quality, but it sounds like that system gets confused during those long silences and creates these phantom vocals. Is this something that has to do with my settings or a glitch that is known and being worked on? Any tips or feedback on this would be much appreciated!