apply 3D rotation on alpha matte masked layer

[Kaka5C21]

So I have this upscaled layer which needs a mask out and a 3D rotation.
3D rotation is easily achieved on the single masked out layer, but that mask gives it shaggy edges.
Using alpha matte to mask out can leave the edge sharp & clean, but 3D rotation won't apply. I've tried using null object to control the 3D rotation (got the both mask and layer parented), but then the layer won't do the alpha matte mask.

Is there a way to apply 3D rotation on alpha matte masked layer? 

Or is there solution to have a upscaled layer masked out sharply together with 3D rotation?

Thanks!

[Mylenium]

Pre-compose the rotating layer, use the result as a matte.

Mylenium

[Kaka5C21]

Seems like the only working way is to have the layer masked with sharp edge and together applying 3D rotation is to pre compose. Thanks

[Rick Gerard]

Try applying Set Matte instead of using a track matte. 

The other option would be to apply a mask directly to the layer.

[Kaka5C21]

Thanks for the tip "Set Matte", I tried but it was unable to mask out the area I wanted (not center of comp), it only masked correctly with the "collapsed transformations" enabled, however that conflicts with the 3D layer. I used ramdon picture to show the process (see attached)

Applying mask directly results in shaggy edges (becoz of the scaled resolution), yet I want to keep the edge sharp.

Seems like the only working way is to pre compose and rotate. Thanks for the suggestion.

[Rick Gerard]

Put your set matte layer in the nested comp. (the pre-comp)