Selection toggle

[Hoffbrau]

I need to create a script that will toggle between the layer's mask and the grayscale [or rgb] channel: essentially it will check to see which is currently selected, then select the opposite. How do you querry to see what is selected currently, what's the best way to go about doing this? I'd like to tie this script to a function key and speed up my workflow.

Thanks in advance.

[Paul Riggott]

Is this what you are after? ...

toggleEditMask();
function toggleEditMask() {
try{
var desc = new ActionDescriptor();
var ref = new ActionReference();
ref.putEnumerated( charIDToTypeID('Chnl'), charIDToTypeID('Ordn'), charIDToTypeID('Msk ') );
var Bool = executeActionGet(ref).getBoolean(stringIDToTypeID('visible'));
desc.putReference( charIDToTypeID('null'), ref );
desc.putBoolean( charIDToTypeID('MkVs'), !Bool );
executeAction( charIDToTypeID('slct'), desc, DialogModes.NO );
}catch(e){}
};
 

[Hoffbrau]

Thanks Paul  - that was close. Actually this solved another issue that I was going to ask about. I'm curious - what was your method for solving this problem - did you use the script listener?

[Paul Riggott]

In part I used the ScriptListener, but also examined the keys of the descriptor using ExtendScript Toolkit.

[Hoffbrau]

Paul

So as I said - this creates a toggle for the mask to it's 'black and white state' - that's great - something I definitelly needed, but I also need another toggle to make the whole mask turn on and off. I captured doing this in the following bit of listener code - I clicked the shift clicked the mask to turn it off, then clicked it again turn it back on. What do I replace in your code above to make this work? Thanks so much for the help!

// =======================================================

var idsetd = charIDToTypeID( "setd" );

    var desc5 = new ActionDescriptor();

    var idnull = charIDToTypeID( "null" );

        var ref6 = new ActionReference();

        var idLyr = charIDToTypeID( "Lyr " );

        var idOrdn = charIDToTypeID( "Ordn" );

        var idTrgt = charIDToTypeID( "Trgt" );

        ref6.putEnumerated( idLyr, idOrdn, idTrgt );

    desc5.putReference( idnull, ref6 );

    var idT = charIDToTypeID( "T   " );

        var desc6 = new ActionDescriptor();

        var idUsrM = charIDToTypeID( "UsrM" );

        desc6.putBoolean( idUsrM, false );

    var idLyr = charIDToTypeID( "Lyr " );

    desc5.putObject( idT, idLyr, desc6 );

executeAction( idsetd, desc5, DialogModes.NO );

// =======================================================

var idsetd = charIDToTypeID( "setd" );

    var desc7 = new ActionDescriptor();

    var idnull = charIDToTypeID( "null" );

        var ref7 = new ActionReference();

        var idLyr = charIDToTypeID( "Lyr " );

        var idOrdn = charIDToTypeID( "Ordn" );

        var idTrgt = charIDToTypeID( "Trgt" );

        ref7.putEnumerated( idLyr, idOrdn, idTrgt );

    desc7.putReference( idnull, ref7 );

    var idT = charIDToTypeID( "T   " );

        var desc8 = new ActionDescriptor();

        var idUsrM = charIDToTypeID( "UsrM" );

        desc8.putBoolean( idUsrM, true );

    var idLyr = charIDToTypeID( "Lyr " );

    desc7.putObject( idT, idLyr, desc8 );

executeAction( idsetd, desc7, DialogModes.NO );

[Michael_L_Hale]

As Paul said, making a toggle needs more than scriptlistener code. All that plugin does is record most user actions. To make a toggle you need to know the current state of the mask. Here is how I would go about making a toggle mask function.

Start with the scriptlistener log you posted. Notice that other than the variable numbering the only difference in the code to enable/disable the mask is the line desc8.putBoolean( idUsrM, true );

So with that knowledge, I first start by cleaning up the log to make it shorter and reduce the number of variables. Something like this,

var desc = new ActionDescriptor();

var ref = new ActionReference();

ref.putEnumerated( charIDToTypeID( "Lyr " ), charIDToTypeID( "Ordn" ), charIDToTypeID( "Trgt" ) );

desc.putReference( charIDToTypeID( "null" ), ref );

var desc1 = new ActionDescriptor();

desc1.putBoolean( charIDToTypeID( "UsrM" ), false );

desc.putObject( charIDToTypeID( "T   " ), charIDToTypeID( "Lyr " ), desc1 );

executeAction( charIDToTypeID( "setd" ), desc, DialogModes.NO );

From there it is easy to create a function to either enable or disable the mask.  But to determine the current state of the mask we need to add a line that gets that property from the layer.

var desc = new ActionDescriptor();

        var ref = new ActionReference();

        ref.putEnumerated( charIDToTypeID( "Lyr " ), charIDToTypeID( "Ordn" ), charIDToTypeID( "Trgt" ) );

    var Bool = executeActionGet(ref).getBoolean(stringIDToTypeID('userMaskEnabled'));

    desc.putReference( charIDToTypeID( "null" ), ref );

    var desc1 = new ActionDescriptor();

     desc1.putBoolean( charIDToTypeID( "UsrM" ), !Bool );

    desc.putObject( charIDToTypeID( "T   " ), charIDToTypeID( "Lyr " ), desc1 );

    executeAction( charIDToTypeID( "setd" ), desc, DialogModes.NO );

[Hoffbrau]

Michael - wow, thanks for your answer and a look at your method. That didn't just solve the problem - that answered *a bunch* of questions for future issues. So in the spirit of 'teaching me to fish' as it were...what's the best learning resource you could give a java ps scritpt newbie [particular book, website]? I've gotten a book on basic Photoshop scripting and used it to frankenstien a bunch of scripts together without real knowledge about what I'm doing. You guys are doing things like 'try' 'catch' that I've never seen before...

[Michael_L_Hale]

I don't think there are really any good books or web tutorials on Photoshop scripting. The only book I am aware of was written during the Photoshop 7/CS timeframe. The web tutorials I know about are also outdated. Deke McClelland has a course on Photoshop Scripting at lynda.com but that is also old and has other problems. I think your best resoucre for Photoshop scripting is here or PS-Scripts.com.

Simon Allardice has a course of basic programming( using javascript ) and on basic javascript at lynda.com that I think would be helpful for someone without much programming/javascript experience.

A try/catch block is standard javascript and is not limited to just Adobe's ExtendScript. It is used when you expect there may be an error and want to handle that error without stopping the script.

In the case of the code in this thread you ideally would want the script to first check that there is an open document. Then it would check that the activeLayer has a channel mask. Only then would it toggle the mask.

The try/catch lets you bypass those checks. If there is no document open or the layer doesn't have a mask the function would do nothing. The downside is it doesn't let the user know why it didn't work. At lease as written here. You could put some kind of alert in the catch part of the code.

[cbuliarca]

Hello everyone, first I would like to say that this forum and the  ps-scripts.com forum helped me a lot in understanding the java scripting  for photoshop, I don't have words to express my gratitude for the  people that want to help and guide us newbies for free. Thank you so  much guys.

Now, I would like to contribute to your toogle  script, I've taken the script that you already posted and played with it  and I've noticed that on a document with a lot of layers the toogle is  kind of slow, so I've changed it a bit with the help of xtool's Getter  script , and I've added one step in the toogle so the toogle goes  from RGB to selecting the mask and than making the layer mask visible

This is what I came up with:

function isChannelMaskSelected(){
    var ret = false;
    if(hasMask() == true){
        if(getChannelMaskSel() == true)
        {
            ret = true;
        }
    else{ret = false}
    }
    return ret;
};
function hasMask(){
        var hasMask = false;
        var ref = new ActionReference();
        var keyHasMask = app.stringIDToTypeID( 'hasUserMask' );
        ref.putEnumerated( app.charIDToTypeID( 'Lyr ' ), app.charIDToTypeID( 'Ordn' ), app.charIDToTypeID( 'Trgt' ) );
        var desc = executeActionGet( ref );
        if ( desc.hasKey( keyHasMask ) ) {
            hasMask = true;
        }
    return hasMask;
}
function selectComponentChannel() {
    try{
        var map = {}
        map[DocumentMode.GRAYSCALE] = charIDToTypeID('Blck');
        map[DocumentMode.RGB] = charIDToTypeID('RGB ');
        map[DocumentMode.CMYK] = charIDToTypeID('CMYK');
        map[DocumentMode.LAB] = charIDToTypeID('Lab ');
        var desc = new ActionDescriptor();
            var ref = new ActionReference();
            ref.putEnumerated( charIDToTypeID('Chnl'), charIDToTypeID('Chnl'), map[app.activeDocument.mode] );
        desc.putReference( charIDToTypeID('null'), ref );
        executeAction( charIDToTypeID('slct'), desc, DialogModes.NO );
    }catch(e){}
};
function selectChannelMask(visible){
  try {
    var desc = new ActionDescriptor();
    var ref = new ActionReference();
    ref.putEnumerated( charIDToTypeID( "Chnl" ), charIDToTypeID( "Chnl" ), charIDToTypeID( "Msk " ) );
    desc.putReference( charIDToTypeID( "null" ), ref );
    desc.putBoolean( charIDToTypeID( "MkVs" ), visible );
    executeAction( charIDToTypeID( "slct" ), desc, DialogModes.NO );
    return true;
  } catch (e) { return false;}
};
function getChannelMaskVis(){
    var isVisible = false;
        var ref = new ActionReference();
        var keyVisChannels = app.stringIDToTypeID( 'visibleChannels' );
        ref.putEnumerated( app.charIDToTypeID( 'Lyr ' ), app.charIDToTypeID( 'Ordn' ), app.charIDToTypeID( 'Trgt' ) );
        var desc = executeActionGet( ref );
        if ( desc.hasKey( keyVisChannels ) ) {
            list = desc.getList(keyVisChannels);
            if(list.count > 1)
            {
                isVisible = false;
            }
            else{isVisible = true}
        }
    return isVisible;
}
function getChannelMaskSel(){
    var isMaskSelected = false;
        var ref = new ActionReference();
        var keyVisChannels = app.stringIDToTypeID( 'targetChannels' );
        ref.putEnumerated( app.charIDToTypeID( 'Lyr ' ), app.charIDToTypeID( 'Ordn' ), app.charIDToTypeID( 'Trgt' ) );
        var desc = executeActionGet( ref );
        if ( desc.hasKey( keyVisChannels ) ) {
            list = desc.getList(keyVisChannels);
            if(list.count > 1)
            {
                isMaskSelected = false;
            }
            else{isMaskSelected = true}
        }
    return isMaskSelected;
}
function toggleChannelMask(){
    var maskSelected = isChannelMaskSelected();
    if(maskSelected == true){
        if(getChannelMaskVis() == true)
        {
            selectComponentChannel();
        }else{selectChannelMask(true)};
    }else{
          selectChannelMask(false);
    }
};
toggleChannelMask();

[Michael_L_Hale]

I've noticed that on a document with a lot of layers the toogle is  kind of slow

The Photoshop Object Model can be slow when working with a lot of layers. This only works with one layer( the activeLayer ) and the DOM part only access the document object, not any layer objects. I wouldn't expect replacing the Photoshop Object Model code with Action Manager to make any speed speed improvement. Nor am I seeing any speed increase in my testing. How many layers are you finding are needed before there is a speed difference.

[cbuliarca]

Hello Michael,

so I have a lot of layers inside a lot of layerSets ( but I see it doesn't matter actyually if the layers are inside the layerSets), Ive made a test and with for example 72 layers there is a small wait like half a second (if I increse the layer numbers the waiting time is bigger), but this only happens when the selection toogle has to go from RGB to the mask.

[Michael_L_Hale]

I am not seeing the speed difference here even with more than 100 layers

But I do like the idea of determining if the mask is selected without using a try/catch block. But the function getChannelMaskSel() has problems. It will also return true if the document is in quick mask mode, if one of the component channels is selected, or if an alpha/spot channel is selected. That may not matter for what it is doing here but I could see it causing problems if you then tried to do something thinking the mask was selected.

[Michael_L_Hale]

I think this fixes those problems

function getChannelMaskSel(){
    var ref = new ActionReference();
    ref.putProperty( charIDToTypeID( 'Prpr' ), stringIDToTypeID('numberOfChannels') );
    ref.putEnumerated( charIDToTypeID( "Dcmn" ), charIDToTypeID( "Ordn" ), charIDToTypeID( "Trgt" ) );
    var desc = executeActionGet(ref);
    var numberOfChannels = desc.getInteger(stringIDToTypeID('numberOfChannels'));
    var ref = new ActionReference();
    ref.putProperty( charIDToTypeID( 'Prpr' ), stringIDToTypeID( 'targetChannels' ) );
    ref.putEnumerated( app.charIDToTypeID( 'Lyr ' ), app.charIDToTypeID( 'Ordn' ), app.charIDToTypeID( 'Trgt' ) );
    var desc = executeActionGet( ref );
    var list = desc.getList( stringIDToTypeID( 'targetChannels' ) );
    var firstIndex = list.getReference(0).getIndex();
    return list.count == 1 && firstIndex == numberOfChannels+1;
};

[cbuliarca]

Hello Michael,

that's great the function that you just corrected it's perfect, thank you very much, you opened my eyes once again . I think you should write a book with all your knowledge about the AM scripting. Your help is greatly appreciated.

About the layers number that's odd, maybe it's just me with the problem, I've tested the DOM script on ps5 as well and I have the same problem, but now with your adjustment it work's like a charm. Thank you once again.

[Michael_L_Hale · Accepted solution]

Save the following script then create an action that runs the script and assign a keyboard shortcut to that action.

function isChannelMaskSelected(){
    try{
        app.activeDocument.activeChannels;
        return false;
    }catch(e){ return true;}
};
function selectComponentChannel() {
    try{
        var map = {}
        map[DocumentMode.GRAYSCALE] = charIDToTypeID('Blck');
        map[DocumentMode.RGB] = charIDToTypeID('RGB ');
        map[DocumentMode.CMYK] = charIDToTypeID('CMYK');
        map[DocumentMode.LAB] = charIDToTypeID('Lab ');
        var desc = new ActionDescriptor();
            var ref = new ActionReference();
            ref.putEnumerated( charIDToTypeID('Chnl'), charIDToTypeID('Chnl'), map[app.activeDocument.mode] );
        desc.putReference( charIDToTypeID('null'), ref );
        executeAction( charIDToTypeID('slct'), desc, DialogModes.NO );
    }catch(e){}
};
function selectChannelMask(){
  try {
    var desc = new ActionDescriptor();
    var ref = new ActionReference();
    ref.putEnumerated( charIDToTypeID( "Chnl" ), charIDToTypeID( "Chnl" ), charIDToTypeID( "Msk " ) );
    desc.putReference( charIDToTypeID( "null" ), ref );
    desc.putBoolean( charIDToTypeID( "MkVs" ), false );
    executeAction( charIDToTypeID( "slct" ), desc, DialogModes.NO );
    return true;
  } catch (e) { return false;}
};
function toggleChannelMask(){
    var maskSelected = isChannelMaskSelected();
    if(maskSelected){
        selectComponentChannel();
    }else{
        selectChannelMask();
    }
};
toggleChannelMask();

[Hoffbrau]

Many Thanks Micheael! That was spot on! What was your methodology in solving the problem? Did you use the script listener at all?

Cheers

Clay

[Michael_L_Hale]

selectComponentChannel() contains scriptlistener/Action Manager code but could be done with the Object Model.

selectChannelMask() also uses scriptlistener and as far as I know can not be done using the Object Model.

isChannelMaskSelected() is pure Object Model.

I already had those three function so it was just a matter of creating toggleChannelMask() to combine those to do what you requested.