huge amounts of noise during vocal pauses on podcasts

So I record a podcast and when i edit the episodes and export them, i get a massive amount of noise in the parts where nobody is speaking. Its so bad that i literally have to go through the episode and fix every single area where someone isnt speaking and the audio sounds terrible. Noise reduction doesnt fix this. What can I do?