How to check if a layer's vector mask is enabled or disabled?

[adenry]

I need to check if a layer's masks are disabled (by shift-clicking them in Photoshop, bringing a big red "X" on them).

So I came up with these functions:

function vectorMaskEnabled( _layerID ){
     var ref = new ActionReference();          
     ref.putIdentifier(charIDToTypeID('Lyr '), _layerID);
     return executeActionGet( ref ).getBoolean( stringIDToTypeID( 'vectorMaskEnabled' ) );
}

function rasterMaskEnabled( _layerID ){
     var ref = new ActionReference();
     ref.putIdentifier(charIDToTypeID('Lyr '), _layerID);
     return executeActionGet( ref ).getBoolean( stringIDToTypeID( 'userMaskEnabled' ) );
}

For whatever reason, the rasterMaskEnabled function works, but the vectorMaskEnabled does not. It seems like "vectorMaskEnabled" is always true when there is a vector mask, regardless of its state (enabled or disabled), where "userMaskEnabled" works as expected.

Any idea why is that happening and how can I fix it? I just need to know if the vector mask is enabled or not.

[SuperMerlin]

"vectorMaskEnabled" is not accessible, you would need to request that Adobe add this key.

The only way that I can think of that will change is the bounds of the layer, so if you can get the bounds when you know it not disabled you will have a test.

[wststreet]

I was struggling with this for days.

I found in someone's code on the internet, they did a very clever trick.

You need to save the current document history in a variable:

var history = this.app.currentDocument.activeStateHistory;

After that manually enable the vector mask.

Then compare the history variable with the current state

if (history == this.app.currentDocument.activeStateHistory) {
    // state never changed, vector mask was already enabled
    return true;
} else {
    // state changed, meaning it was originally disabled
    // revert changes and return false
    this.app.currentDocument.activeStateHistory = history;
    return false;
}

[c.pfaffenbichler]

Apart from »this.app.currentDocument« or »activeStateHistory« not being Photoshop-conform it represents a valid work-around so long as »suspendHistory« is not »on«.

[Jarda Bereza]

I logged bug #PS-10234

Add reminder in your calendar at this day in the future.

I guess sometimes around 2020 (+- 2 years)  it could be fixed... maybe

[Jarda Bereza]

c.pfaffenbichler​ i invented valid workaround

function isVectorMaskEnabled(){
    var doc = app.activeDocument;
    var lastIndex = doc.pathItems.length;
    // if there is no item mask can be disabled or not even exists
    // so you could want check for existing vector mask before your call this function
    if(lastIndex===0){return false}
    // vector mask should be last in paths list. If is not selected then it has "normal" kind. So we select last path.
    var desc = new ActionDescriptor();
    var ref = new ActionReference();
    ref.putIndex( charIDToTypeID( "Path" ), lastIndex );
    desc.putReference( charIDToTypeID( "null" ), ref );
    executeAction( charIDToTypeID( "slct" ), desc, DialogModes.NO );
    // now we check which kind is most bottom path
    var lastPathInList = doc.pathItems[lastIndex-1];
    var isVectorMaskEnabled = lastPathInList.kind === PathKind.VECTORMASK;
    return isVectorMaskEnabled;
}
alert(isVectorMaskEnabled());

[Jarda Bereza]

So... we are lucky and logged bug was fixed after only 5 months in PS CC 2018.1

Since this version action descriptor return real world value.

[r-bin]

And what about "vectorMaskLinked" and "filterMaskEnable" ?

[Jarda Bereza]

"vectorMaskLinked" not exists in layer descriptor. You need workaround.

FilterMask is in layer descriptor

I think filterMask can't be linked/unlinked

[r-bin]

But I wrote about "filterMaskEnable".

P.S. So they will fix one ID every 10 years?

[Jarda Bereza]

Yes and I am showing you image with proof that it works 😉

(I hope you should see it)

If you are lucky, your bugs will be fixed in 3-6 months. But most likely it will be 1-3 years. Or more.

Feature requests 2-6 years. It's not certain number of IDs in time period. Reported bugs just have a big delay.

[r-bin]

Yes, I already saw it. Thank you.