How to check if a layer's vector mask is enabled or disabled?

So... we are lucky and logged bug was fixed after only 5 months in PS CC 2018.1

Since this version action descriptor return real world value.

c.pfaffenbichler​ i invented valid workaround

function isVectorMaskEnabled(){
    var doc = app.activeDocument;
    var lastIndex = doc.pathItems.length;
    
    // if there is no item mask can be disabled or not even exists
    // so you could want check for existing vector mask before your call this function
    if(lastIndex===0){return false}
    
    // vector mask should be last in paths list. If is not selected then it has "normal" kind. So we select last path.
    var desc = new ActionDescriptor();
    var ref = new ActionReference();
    ref.putIndex( charIDToTypeID( "Path" ), lastIndex );
    desc.putReference( charIDToTypeID( "null" ), ref );
    executeAction( charIDToTypeID( "slct" ), desc, DialogModes.NO );
    // now we check which kind is most bottom path
    var lastPathInList = doc.pathItems[lastIndex-1];
    var isVectorMaskEnabled = lastPathInList.kind === PathKind.VECTORMASK;
    return isVectorMaskEnabled;
}
alert(isVectorMaskEnabled());

I logged bug #PS-10234

Add reminder in your calendar at this day in the future.

I guess sometimes around 2020 (+- 2 years)  it could be fixed... maybe

I was struggling with this for days.

I found in someone's code on the internet, they did a very clever trick.

You need to save the current document history in a variable:

var history = this.app.currentDocument.activeStateHistory;

After that manually enable the vector mask.

Then compare the history variable with the current state

if (history == this.app.currentDocument.activeStateHistory) {
    // state never changed, vector mask was already enabled
    return true;
} else {
    // state changed, meaning it was originally disabled
    // revert changes and return false
    this.app.currentDocument.activeStateHistory = history;
    return false;
}

"vectorMaskEnabled" is not accessible, you would need to request that Adobe add this key.

The only way that I can think of that will change is the bounds of the layer, so if you can get the bounds when you know it not disabled you will have a test.