Loss of Audio Quality

Some of my users complains that while viewing the live streaming of my application, the audio quality deteroriates over time. The audio tends to sounds jerky for a while and then the audio is completely lost. This happens when the video quality remains constantly acceptable, hence I don't think its a internet connection issue.
I am encoding my audio at 11025Hz and 20kbps. I wonder if anyone has encountered a similar problem or has a solution to this. Thanks!