Live, unbuffered RTMP Audio Delayed

I had a few live audio sessions running on AMS 5 last night.  Most went fine.

In once session (the clients use a buffer of zero in order to prevent any buffering) the audio was delay.  The audio would stop for an extended period of time, then rush into the session (i.e. you would hear 20 seconds of audio in 1 second).

The audio is recorded on the server and is perfect.  Therefore, there was clearly an issue on the AMS side distributing the audio to 50+ users.  There were another 50+ users in other sessions receiving live audio which did not have issues.

My assumption is the particular application instance handling the session with the delayed audio was not getting enough resources to handle the load?  Memory or CPU must have been bottlenecked, causing a performance issue?

I'm wondering if there are settings which are most relevant to increasing performance in this scenario.

There were no errors in any logs related to performance.