Delete silence issue

[shaky camera]

I thought I was following everything as I should have (including level scan)but when I remove the silences, most of the audio that follows (to the right of) a removed section is clipped. Therefore I need to pull out the clipped audio from the left. Sometimes this can be a considerable chunk.
Why would that be? What have I done wrong?

Thanks