When recording in Adobe Audacity, every few minutes i get voice feedback while live recording, help!

So in short, I have tried using different mics including Blue Yeti and Shure MV7, and get the same issue. I am recording at a bit rate of 48kHz and tried different I/O buffer sizes, but I like to stick to lower ones (i.e. 32 or 64 to reduce the audio latency In my headset. An example during my recording would be something like me saying "The man woke up to go to work at 6 am in the morning". Out of nowhere as I am saying "in the morning" I'll hear feedback of myself slightly delayed right after saying that just a bit louder saying "IN THE MORNING" and it will throw me off. So it'll be like saying "In the morning" and then hearing "IN THE MORNING" a second time right after even though I only said it once.

The odd thing is that even though I hear that twice, when I go back to listen to it in the recording, it all sounds normal and I am only hearing those few-word feedbacks which are louder, only every now and again while I am actively recording and NOT in the recorded track. It is a very strange thing and I couldn't find anyone else online with the same issue. I hope I was able to explain it well, if anyone can help me figure out how to fix this, I would greatly appreciate it as it would throw my guests and myself off during a podcast recording.

Thank you in advance.