The arm is moving independently leaving the hands behind.

Hi, I have an issue with my puppet's arms. Draggables move everything that's on the same layer, but since my hands aren't on the same layer as the arms, the draggable is only working with the arms and the hands get left behind. The issue would probably get fixed if I put the hands on the same layer as the arms, but since I want to make a swap set with the hands and don't want the arms to disappear with them, they need to stay on different layers. I use Illustrator to edit the vector file if that changes anything
     Is there a way to fix this?