Puppet moves slowly after increasing the layers(Windows)

I know people will think the computer is the issue but honestly my character moves great and runs at 30 FPS when just loaded up as a basic character.

The FPS issues comes when I load up a version of the character that has a couple swap sets for the arms to animate different actions.  Basically it seems the more layers my project has the slower Character Animator runs even when those layers are hidden and not being used.  Anyone found a work around for this?  I want to be able include many different actions in the character so the end user can use it for streaming or videos and not have to use multiple characters with few animations.

Turning off things doesn't make a difference and I have had this issue with other characters, basically as the number of layers increases, displayed or not, Character animator begins to slow to a crawl.  PC is I7-6900K, 64Gb ram, 1080TI, triple SSD raid-0. Seems more linked to number of layers in a character than computer capability.  Any tips appreciated, kind of just bringing this up since it seems odd for the program to slow so much due to static layers that aren't displayed.  Surely this isn't working as intended

Title was edited by: Mod