Hands are detaching from arms when using leader/follower...is there a workaround?

I'm wanting to use the leader/follower behaviour to have several arms positions. I'm also wanting seperate hands that can be triggered to change.

I have successfully set up leader/follower with auto-swap triggers. 
I have also successfully set up triggers for the different hands.

Is there any way to get both things working together or do you need to create two puppets?

When I have leader/follower on it means that the hand are no longer attached.