Progressive performance drop due to Replays

I have a rather complex puppet with a number of replays. For example, a replay to apply a transform behavior to all components of the face to simulate a nod or looking down.

My PC can comfortably handle a framerate of more than 24 in the record and perform tabs. However, while testing the replays, I noticed that at one point it the frame rate dropped below 24. From there, as I continued testing replays, the frame rate continued to drop in direct relation to the number of replays that had been triggered (Note that all are less than 1 second in length). Refreshing the scene resets the frame rate to above 24, but triggering a certain number of replays make it progressively drop again. 

I've also noticed that as performance dramatically decreases, the memory consumption of Character Animator as well as CPU usage does NOT increase.