Audio playback levels is randomly too low or too high when on JKL double speed.

I'm editing long form audio files on multitrack, sometimes I speed it up (modified my settings so that JKL shuttle is half speed) and my levels are randomly either too low or too high, not the actual levels at normal playback speed. 

Am I alone in this? It's making it hard for me to monitor levels when at 1.5 or 2x playback speed.

Thanks.