It is getting slower and slower because frames are dropping and it is not playing at realtime even though the timeline bar is green.
I have tried on MAC book pro playing the audio with 120 FPS and faced the same issue because if you open the info panel and than playback the composition it will show that playback is not realtime and frames are dropping.
And if you off the video(eye icon) in the preview panel and than playback..Audio should playback fine.
Try Pre-render from > Main Menu > Composition > Pre-render and then playback in your system.