Auto Silence Detection Not Working Right When Selecting Long Audio Portion

When I select a small portion of audio (a minute or two)  for audition to detect silences. It works.

When I select the whole audio file (an hour long) and tell it to detect silent parts. It's results are durastically different.

What is going on?
First image is it detecting silence well.

The second image has highlighted audio that audition is thinking is silence when it clearly isn't