Artifacts when transforming or warping layer masks in Photoshop

[Dr Missile]

Whenever I transform or warp a layer mask via a selection it creates opaque pixels at the edges of the transformed selection. This occurs even if I have the transparent mask color correctly set. Anyone know of a way of preventing this?

Original image (white is trandparent):

Original shape

Warping shape

Resulting artifact

Layer setup

{Thread renamed by moderator}

[Sahil.Chawla]

Hi there,

That seems like a strange behavior, could you let us know the version of Photoshop you're working on?

Could you please try enabling "Legacy compositing" located in Photoshop's Preferences > Performance, then relaunch Photoshop and let us know if it helps?

Does it only happen with this file or other files too?

Regards,
Sahil

[Dr Missile]

Thank you for the suggestion, Sahil. Enabling legacy compositing unfortunately doesn't fix this. This happens with all files and has been happening with Photoshop for as long as I can remember (5 years ish?)c across multiple PCs running multiple versions of windows (7 through 10). I'm currently on latest photoshop (21.0.1)

[S_Gans]

Can you try warping the mask without making a selection with a selection tool? That may help.

[Dr Missile]

Thanks for the suggetion S_Gans. Actually transforming the contents would be ideal, but some masks don't allow me to transform just the contents. Rather it only allows me to transfrom the entire canvas area. Which is why i'm trying to use the selection to begin with. Obviously mask and layer are not linked.

your suggestion works on some masks but not others. Not sure what forces a mask to use the entire area rather than the masked selection are... perhaps there's some construction history that gets lost or mask pixels outside of the actual image bounds that cause this?

[Dr Missile]

Actually I think I found what causes it to transform the entire canvas: If I fill the mask with white - the contents will never be independently transformable. I'm guessing that not a lot of people interact with masks like this so it might be an un-reported bug.

Sahil, thoughts?

[Michael Bullo]

Have you tried turning your layer into a Smart Object before applying transformations?

[Dr Missile]

Smart objects aren't practical for the way I'm using the masks, but thanks for the suggestion 🙂