How do I invert my Vector mask in a layer mask?

[Tyler_Hudelson3744]

I'm trying to invert my Vector mask (The right one) while it's inside of my Layer mask (the middle one). I can't seem to invert. 

I want to invert this patter. I want the rounded recatangle be blank and the outside to have the pattern.

[Trevor.Dennis · Accepted solution]

Select the mask in the Paths panel.

Select the Direct Selection tool (white arrow) and drag a window around the path so that all points become solid.

Then use the Path Operations drop down in the Options bar, and select Subtract Front Shape.

The mask will be inverted.

[Stephen Marsh]

It's one of those things that are not intuitive and go against my logic, as you don't need to actually create a second vector path on the canvas edges in order to perform the "subtract front shape" path operation against - as there is only one path. Even after the path operation is performed, there is only one vector path.

[Trevor.Dennis]

Steven, I didn't know how to do it from memory, although I assumed it would be via Path Operations.  So it came down to trying it out a step at a time.  I don't think I have ever had a need to do it before.

[Stephen Marsh]

@Trevor.Dennis - exactly the same here! I Googled the answer before your post and also assumed it would be via the path operations. I too have never had to do this.

[Tyler_Hudelson3744]

Thank you so much!

[rayek.elfin]

This is one of those Photoshop 'moments' where it becomes quite obvious that certain functions in Photoshop plainly show their age through obscurity and legacy limitations in core functionality.

To compare, PhotoLine allows the user to check a simple invert switch to control the vector mask's path(s) effect on masking. 

Also, Photoshop's frustrating inability to apply more than one vector layer mask (without the use of awkward clipping masks) once more showcases how little Photoshop's core layer functionality has changed in the past 25 years. Competitors allow for multiple layer masks and vector masks, and clipping masks are not the answer depending on the context. (PhotoLine doesn't differentiate between clipping mask layers or layer mask layers, for example.)

PhotoLine turns this into an easy to use feature, and therefore a feature that is regularly used by myself in comping.

 Of course, other software such as Krita and Affinity Photo also do not limit the layer masking the way Photoshop does.

It would be great if the Photoshop devs would finally update the layer stack functionality and push it into the 21st century 😄

[Trevor.Dennis]

It really isn't a problem. Just use nested Smart Objects.

[rayek.elfin]

Is it possible to edit the contents of Smart Objects in-situ and in real-time (affecting the parent document within context)?

Answer to both: no. Smart Objects are an awkward work-around to 'solve' this. 

In the above example I could have used virtual layers (cloned instances) of the original vector mask in PhotoLine as well. Any change to the original mask would cascade to all the other masks.

Is it possible in Photoshop to instance masks directly? No. It is in After Effects and in other apps.

All of these shortcomings are limitations that have been solved in competing software. 

As I stated earlier: Photoshop's layer stack is showing its age. In my opinion so many things could be improved, and much is low hanging fruit.

It's too bad and unfortunate for Photoshop users.

[Claire H.]

Hi @rayek.elfin, Thank you for sharing your opinion on ways you think Photoshop could be made better. If you'd like to share this idea with the broader community, you can post it in the Ideas section. That way, others can vote on the features they’d love to see, and it helps our team prioritize what matters most. Thanks! ^CH