apply 3D rotation on alpha matte masked layer

So I have this upscaled layer which needs a mask out and a 3D rotation.
3D rotation is easily achieved on the single masked out layer, but that mask gives it shaggy edges.
Using alpha matte to mask out can leave the edge sharp & clean, but 3D rotation won't apply. I've tried using null object to control the 3D rotation (got the both mask and layer parented), but then the layer won't do the alpha matte mask.

Is there a way to apply 3D rotation on alpha matte masked layer? 

Or is there solution to have a upscaled layer masked out sharply together with 3D rotation?

Thanks!