Plugin/tool for tightening audio without cutting video

I am looking for a tool to help speed up my video editing process in Premiere. We produce instructional videos with a top-down camera demonstrating an artistic skill. The instructor often speaks slowly while she's performing the skill, or there are unnaturally long pauses between words. I've been unlinking the audio from the video, and manually cutting the audio track to tighten up the audio and remove silences. Then I'll determine if I can cut out any of the video track, or speed up the video track to match the remaining length of the audio after silences have been cut. 

 

I looked into the Firecut plugin and it seems like the  "Remove Silences" tool has potential, but it's unclear if it can accomplish exactly what I need. I need to be able to edit the audio and video tracks separately and have full control over the video. The end user needs to be able to follow along and understand the video demonstration without it being chopped up. 

 

I often make 100 cuts or more in a 3-5 minute video, and it is extremely time-consuming. Any tools, plugins, tips, or tricks you can suggest to help speed up my workflow?