Hidden Layer Is Being Dragged

[ebenheizerm28787053]

So here's the thing:

I have:

1. Direction Front (The Body and Bottom facing to the front)

2. Direction Right (The Body and Bottom facing to the right side of the character)

I put the two layers inside a Swap Set and have the Direction Front layer as the default, which means the Direction Right layer is now hidden until I trigger it.

In the Direction Right layer, I have an arm which I rigged with sticks and draggable points, like usual. And so is the same with the Direction Front layer, which is obvious of course, hehehe.

When I do some dragging with the Direction Front layer, the draggable area in the Direction Right layer is also dragged.

How to make hidden layers undraggable without deleting the draggable points?

Thank you!