ArchViz Post Production Layer Mask Script

[Daniel Marple]

Long time photoshop user, first time poster.

I'm an architectural visualiser and part of my job involves using jpeg geometry black and white masks generated in my renders as group layer masks. 

Currently this involves:

1. Creating a group the main photoshop file.

2. Naming the group to match the corresponding .jpeg generated by the render eg. 'Brickwork'

3. Open the separate Brickwork.jpeg mask

4. Copy all

5. Paste into main photoshop file as the mask of my 'Brickwork' group from (2)

This leaves me with a group correctly named, and with the right mask which allows me to adjust levels, saturation, colour etc of all the brickwork in my render. I would love to be able to automate this process to avoid having to manually do the same 5 step process to the 20+ layer masks in each of my renders. I hope that the below image helps which shows an example with only 7 groups:

Any help would be much appreciated. Please let me know if you have any questions.

Dan

[c.pfaffenbichler]

Can you provide a complete example? (The renderings and the resulting file minus sensitive data, naturally.)

Are the renderings always located relative to the psd or do you want to have Folder-selection? 

Are the mask-renderings grayscale or RGB? 

Why jpg and not a file format without lossy compression? 

[Daniel Marple]

Thanks for your swift reply! To answer your questions:

Can you provide a complete example? (The renderings and the resulting file minus sensitive data, naturally.)
Yes please see attached project file stripped of sensitive data as well as the corresponding project renders.

Are the renderings always located relative to the psd or do you want to have Foder-selection? 

The renders are always in a separate folder called 'TIF' in the same master project folder as the photoshop file which is in 'PSD'. Having folder selection would be useful.

Are the mask-renderings grayscale or RGB? 

They are RGB with an alpha channel.

Why jpg and not a file format without lossy compression?

Apologies, the renderings are actually .tif files not .jpeg.

Please let me know if I can be of any more help.

[Stephen_A_Marsh]

The filename pattern appears to be consistent, but just to double check...

R1_CMBrick0000

Ignore the first 5 characters, ignore the last four characters, which may always be digits and not just any character?

Is that a reasonable assumption?

[Daniel Marple]

Hi Stephen,

Thanks for your help.

The first 5 characters are always the same, annoyingly sometimes the render comes back with the extra four 0's at the end, sometimes it doesn't.

But the folders being 'Brick0000' or 'Brick' would be fantastic, although don't worry if this is too complex an ask.

To be honest I'd be happy with the folder name being 'R1_CMBrick0000' as that would be a small price to pay at this point!

[Stephen_A_Marsh]

It's actually one of the easiest parts! 

[c.pfaffenbichler · Accepted solution]

This is crude but feel free to optimize it further. 

// create groups with masks based on tifs in tif-folder beside psd-folder;
// 2022, use it at your risk;
if (app.documents.length > 0) {
    var myDocument = app.activeDocument;
    var docName = myDocument.name;  
    try {var basename = docName.match(/(.*)\.[^\.]+$/)[1]} 
    catch (e) {var basename = docName};
    try {var docPath = myDocument.path}
    catch (e) {var docPath = "~/Desktop"};
    var theFolder = Folder(Folder(docPath).parent+"/TIF");
    if (Folder(theFolder).exists == false) {var theFolder = Folder.selectDialog ("select a folder containing the images")};
    var theFiles = theFolder.getFiles(checkFor);
// process files;
    for (var m = 0; m < theFiles.length; m++) {
        var theFile = File(theFiles[m]);
        var aRegExp = new RegExp(/\d{1,4}\.tif$/i);
        var groupName = theFile.name.slice(5).replace(aRegExp, "");
var idordinal = stringIDToTypeID( "ordinal" );
var idtargetEnum = stringIDToTypeID( "targetEnum" );
var idset = stringIDToTypeID( "set" );
var idchannel = stringIDToTypeID( "channel" );
var iddocument = stringIDToTypeID( "document" );
var idselection = stringIDToTypeID( "selection" );
var idnull = stringIDToTypeID( "null" );
var idname = stringIDToTypeID( "name" );
// create group;
        var desc5 = new ActionDescriptor();
            var ref1 = new ActionReference();
            var idlayerSection = stringIDToTypeID( "layerSection" );
            ref1.putClass( idlayerSection );
        desc5.putReference( idnull, ref1 );
            var desc6 = new ActionDescriptor();
            desc6.putString( idname, groupName );
        desc5.putObject( stringIDToTypeID( "using" ), idlayerSection, desc6 );
        desc5.putString( idname, "Group 1" );
    executeAction( stringIDToTypeID( "make" ), desc5, DialogModes.NO );
// open file;
        var aFile = app.open(theFiles[m]);
// load selection;
    var desc12 = new ActionDescriptor();
        var ref3 = new ActionReference();
        ref3.putProperty( idchannel, idselection );
    desc12.putReference( idnull, ref3 );
    var idto = stringIDToTypeID( "to" );
        var ref4 = new ActionReference();
        ref4.putEnumerated( idchannel, idchannel, stringIDToTypeID( "red" ) );
    desc12.putReference( idto, ref4 );
executeAction( idset, desc12, DialogModes.NO );
// quick mask;
var desc16 = new ActionDescriptor();
    var ref7 = new ActionReference();
    var idproperty = stringIDToTypeID( "property" );
    var idquickMask = stringIDToTypeID( "quickMask" );
    ref7.putProperty( idproperty, idquickMask );
    ref7.putEnumerated( iddocument, idordinal, idtargetEnum );
desc16.putReference( idnull, ref7 );
executeAction( idset, desc16, DialogModes.NO );
// switch documents;
        activeDocument = myDocument;
// load selection;
    var desc19 = new ActionDescriptor();
        var ref9 = new ActionReference();
        var idchannel = stringIDToTypeID( "channel" );
        ref9.putProperty( idchannel, idselection );
    desc19.putReference( stringIDToTypeID( "null" ), ref9 );
        var ref10 = new ActionReference();
        ref10.putEnumerated( idchannel, idordinal, idtargetEnum );
        ref10.putName( iddocument, theFile.name );
    desc19.putReference( stringIDToTypeID( "to" ), ref10 );
executeAction( idset, desc19, DialogModes.NO );
// make layer mask;
    var desc21 = new ActionDescriptor();
    var idnew = stringIDToTypeID( "new" );
    desc21.putClass( idnew, idchannel );
    var idat = stringIDToTypeID( "at" );
        var ref11 = new ActionReference();
        var idmask = stringIDToTypeID( "mask" );
        ref11.putEnumerated( idchannel, idchannel, idmask );
    desc21.putReference( idat, ref11 );
    desc21.putEnumerated( stringIDToTypeID( "using" ), stringIDToTypeID( "userMaskEnabled" ), stringIDToTypeID( "revealSelection" ) );
executeAction( idmake = stringIDToTypeID( "make" ), desc21, DialogModes.NO );
// close;
        aFile.close(SaveOptions.DONOTSAVECHANGES);
    }
    };
////// check for tif //////
function checkFor (theFile) {
    if (theFile.name.slice(-4) == ".tif") {return true}
    else {return false}
    };

[Daniel Marple]

Thanks for putting this together, much appreciated.

To run the script, do I just copy and paste this somewhere?

[Stephen_A_Marsh]

---

@Daniel Marple wrote:

Thanks for putting this together, much appreciated.

To run the script, do I just copy and paste this somewhere?

---

Quickstart:

1. Copy the code text to the clipboard
2. Open a new blank file in a plain-text editor (not word-processor)
3. Paste the code in
4. Save the text file as .txt
5. Rename the filename extension text from .txt to .jsx
6. Install or browse to the .jsx file to run (explained in the link below)

If these simple instructions are too abbreviated, you may need to read on...

[Daniel Marple]

Thank you. I've managed to run the script which starts fine, but I end up with the following error:

I have just updated my photoshop version and still get the error. My Photoshop is v23.1.1.

[c.pfaffenbichler]

Please post meaningful screenshots. 

[Daniel Marple]

Ah I changed the folder locations and this time it didn't ask me to select a folder and it worked perfectly!

The only thing I'd want changed is being able to pick any render files folder, rather than having it limited to the folder 'TIF'. It would be really useful to be able to pull the .tifs in from anywhere on the server.

Thanks again for the help, much appreciated.

[c.pfaffenbichler · Accepted solution]

Try changing the lines 

    var theFolder = Folder(Folder(docPath).parent+"/TIF");
    if (Folder(theFolder).exists == false) {var theFolder = Folder.selectDialog ("select a folder containing the images")};

to 

    var theFolder = Folder.selectDialog ("select a folder containing the images");

[Daniel Marple]

This is perfect, thanks so much for your help.

[c.pfaffenbichler]

I get this result with the files shown in the Finder-window at the bottom of the screenshot. 

Can you provide the set of images on which this error occurs? 

[Stephen_A_Marsh]

Here is my version, I had fun putting it together, hit many brick walls and had find workarounds etc.

I really wanted to create the layer set masks inside the EFFECTS group, however I just couldn't get there and spent way too much time going down false trails. In the end I just moved the layers, however, as I had to hard code their names, if you used this on different renders with different names the mask sets would not be moved unless they were added to the list. I'd like to do more, but for now I'm mentally exhausted!

/*
Create mask groups from renders.jsx
https://community.adobe.com/t5/photoshop-ecosystem-discussions/archviz-post-production-layer-mask-script/m-p/12726579#M620671
Stephen Marsh, v1.0 - 5th February 2022
*/

//#target photoshop

(function () {

    try {
        var inputFolder = Folder.selectDialog("Select the input folder:");
        if (inputFolder === null) {
            // alert('Script cancelled!');
            return;
        }

        var fileList = inputFolder.getFiles(/\.(tif|tiff)$/i);
        fileList.sort().reverse();
        
        var validateEmptyList = (fileList.length > 0);
        if (validateEmptyList === false) {
            alert("Script cancelled as the input folder is empty!");
            return;
        }
        
        var baseDoc = app.activeDocument.name;
        var savedDisplayDialogs = app.displayDialogs;
        app.displayDialogs = DialogModes.NO;
        app.activeDocument.activeLayer = app.activeDocument.layers[0];

        for (var i = 0; i < fileList.length; i++) {
            open(fileList[i]);
            dupeLayer("_temp", 5);
            var maskName = app.activeDocument.name.replace(/\.[^\.]+$/, '').replace(/^.{5}/, '').replace(/\d{4}$/, '');
            app.activeDocument.close(SaveOptions.DONOTSAVECHANGES);

            tempGroup();
            blankMask();
            applyMask(true);
            removeTempLayer();
            app.runMenuItem(stringIDToTypeID('selectNoLayers'));
            app.activeDocument.activeLayer.name = maskName;
        }

        try {
            app.activeDocument.activeLayer = app.activeDocument.layers["WindowFrames"];
            app.activeDocument.activeLayer.name = "Window Frames";
        } catch (error) {}
        
        app.activeDocument.activeLayer = app.activeDocument.layers[0];
        var effectsGroup = app.activeDocument.layerSets.add();
        effectsGroup.name = "EFFECTS";
        try {
            move("Window Frames");
            move("Stone");
            move("Roof");
            move("Interior");
            move("Glass");
            move("Context");
            move("Brick");
        } catch (error) { }
        app.activeDocument.activeLayer = app.activeDocument.layers[0];

        app.displayDialogs = savedDisplayDialogs;
        app.beep();

    } catch (err) {
        alert("An unexpected error has occurred!");
    }

    ////// FUNCTIONS /////

    function move(layerName) {
	var s2t = function (s) {
		return app.stringIDToTypeID(s);
	};
	var descriptor = new ActionDescriptor();
	var list = new ActionList();
	var reference = new ActionReference();
	var reference2 = new ActionReference();
	reference.putName( s2t( "layer" ), layerName );
	descriptor.putReference( s2t( "null" ), reference );
	reference2.putIndex( s2t( "layer" ), 20 );
	descriptor.putReference( s2t( "to" ), reference2 );
	descriptor.putBoolean( s2t( "adjustment" ), false );
	descriptor.putInteger( s2t( "version" ), 5 );
	list.putInteger( 509 );
	descriptor.putList( s2t( "layerID" ), list );
	executeAction(s2t( "move" ), descriptor, DialogModes.NO);
}

    function removeTempLayer() {
        var s2t = function (s) {
            return app.stringIDToTypeID(s);
        };
        var descriptor = new ActionDescriptor();
        var list = new ActionList();
        var reference = new ActionReference();
        reference.putName(s2t("layer"), "_temp");
        descriptor.putReference(s2t("null"), reference);
        list.putInteger(373);
        descriptor.putList(s2t("layerID"), list);
        executeAction(s2t("delete"), descriptor, DialogModes.NO);
    }

    function applyMask(preserveTransparency) {
        var s2t = function (s) {
            return app.stringIDToTypeID(s);
        };
        var descriptor = new ActionDescriptor();
        var descriptor2 = new ActionDescriptor();
        var reference = new ActionReference();
        reference.putEnumerated(s2t("channel"), s2t("channel"), s2t("RGB"));
        reference.putName(s2t("layer"), "_temp");
        descriptor2.putReference(s2t("to"), reference);
        descriptor2.putBoolean(s2t("preserveTransparency"), preserveTransparency);
        descriptor.putObject(s2t("with"), s2t("calculation"), descriptor2);
        executeAction(s2t("applyImageEvent"), descriptor, DialogModes.NO);
    }

    function blankMask() {
        var s2t = function (s) {
            return app.stringIDToTypeID(s);
        };
        var descriptor = new ActionDescriptor();
        var reference = new ActionReference();
        descriptor.putClass(s2t("new"), s2t("channel"));
        reference.putEnumerated(s2t("channel"), s2t("channel"), s2t("mask"));
        descriptor.putReference(s2t("at"), reference);
        descriptor.putEnumerated(s2t("using"), s2t("userMaskEnabled"), s2t("revealAll"));
        executeAction(s2t("make"), descriptor, DialogModes.NO);
    }

    function tempGroup() {
        var tempGroup = app.activeDocument.layerSets.add();
        tempGroup.name = "_tempGroup";
    }

    function dupeLayer(name2, version) {
        var s2t = function (s) {
            return app.stringIDToTypeID(s);
        };
        var descriptor = new ActionDescriptor();
        var reference = new ActionReference();
        var reference2 = new ActionReference();
        reference.putEnumerated(s2t("layer"), s2t("ordinal"), s2t("targetEnum"));
        descriptor.putReference(s2t("null"), reference);
        reference2.putName(s2t("document"), baseDoc);
        descriptor.putReference(s2t("to"), reference2);
        descriptor.putString(s2t("name"), name2);
        descriptor.putInteger(s2t("version"), version);
        executeAction(s2t("duplicate"), descriptor, DialogModes.NO);
    }

}());

[c.pfaffenbichler]

---

I really wanted to create the layer set masks inside the EFFECTS group,

---

I think the »simplest« work-around might be creating an empty Layer in the Group before adding the other Groups and then removing the Layer. 

[Stephen_A_Marsh]

Thank you, I did come to that conclusion, however I couldn't get the recursive code to correctly add the layer sets into the effects set. Perhaps the temp layer was active when it shouldn't be, I ran out of steam...

[c.pfaffenbichler · Accepted solution]

Does this work? 

(function () {

    try {
        var inputFolder = Folder.selectDialog("Select the input folder:");
        if (inputFolder === null) {
            // alert('Script cancelled!');
            return;
        }

        var fileList = inputFolder.getFiles(/\.(tif|tiff)$/i);
        fileList.sort().reverse();
        
        var validateEmptyList = (fileList.length > 0);
        if (validateEmptyList === false) {
            alert("Script cancelled as the input folder is empty!");
            return;
        }
        
        var baseDoc = app.activeDocument.name;
        var savedDisplayDialogs = app.displayDialogs;
        app.displayDialogs = DialogModes.NO;
        app.activeDocument.activeLayer = app.activeDocument.layers[0];

        var effectsGroup = app.activeDocument.layerSets.add();
        effectsGroup.name = "EFFECTS";


        for (var i = 0; i < fileList.length; i++) {
            open(fileList[i]);
            dupeLayer("_temp", 5);
            var maskName = app.activeDocument.name.replace(/\.[^\.]+$/, '').replace(/^.{5}/, '').replace(/\d{4}$/, '');
            app.activeDocument.close(SaveOptions.DONOTSAVECHANGES);

            var tempGroup = effectsGroup.layerSets.add();
            tempGroup.name = maskName;

            blankMask();
            applyMask(true);
            removeTempLayer();
        }

        app.activeDocument.activeLayer = app.activeDocument.layers[0];

        app.displayDialogs = savedDisplayDialogs;
        app.beep();

    } catch (err) {
        alert("An unexpected error has occurred!");
    }

    ////// FUNCTIONS /////

    function removeTempLayer() {
        var s2t = function (s) {
            return app.stringIDToTypeID(s);
        };
        var descriptor = new ActionDescriptor();
        var list = new ActionList();
        var reference = new ActionReference();
        reference.putName(s2t("layer"), "_temp");
        descriptor.putReference(s2t("null"), reference);
        list.putInteger(373);
        descriptor.putList(s2t("layerID"), list);
        executeAction(s2t("delete"), descriptor, DialogModes.NO);
    }

    function applyMask(preserveTransparency) {
        var s2t = function (s) {
            return app.stringIDToTypeID(s);
        };
        var descriptor = new ActionDescriptor();
        var descriptor2 = new ActionDescriptor();
        var reference = new ActionReference();
        reference.putEnumerated(s2t("channel"), s2t("channel"), s2t("RGB"));
        reference.putName(s2t("layer"), "_temp");
        descriptor2.putReference(s2t("to"), reference);
        descriptor2.putBoolean(s2t("preserveTransparency"), preserveTransparency);
        descriptor.putObject(s2t("with"), s2t("calculation"), descriptor2);
        executeAction(s2t("applyImageEvent"), descriptor, DialogModes.NO);
    }

    function blankMask() {
        var s2t = function (s) {
            return app.stringIDToTypeID(s);
        };
        var descriptor = new ActionDescriptor();
        var reference = new ActionReference();
        descriptor.putClass(s2t("new"), s2t("channel"));
        reference.putEnumerated(s2t("channel"), s2t("channel"), s2t("mask"));
        descriptor.putReference(s2t("at"), reference);
        descriptor.putEnumerated(s2t("using"), s2t("userMaskEnabled"), s2t("revealAll"));
        executeAction(s2t("make"), descriptor, DialogModes.NO);
    }

    function dupeLayer(name2, version) {
        var s2t = function (s) {
            return app.stringIDToTypeID(s);
        };
        var descriptor = new ActionDescriptor();
        var reference = new ActionReference();
        var reference2 = new ActionReference();
        reference.putEnumerated(s2t("layer"), s2t("ordinal"), s2t("targetEnum"));
        descriptor.putReference(s2t("null"), reference);
        reference2.putName(s2t("document"), baseDoc);
        descriptor.putReference(s2t("to"), reference2);
        descriptor.putString(s2t("name"), name2);
        descriptor.putInteger(s2t("version"), version);
        executeAction(s2t("duplicate"), descriptor, DialogModes.NO);
    }

}());

[Stephen_A_Marsh]

@c.pfaffenbichler 

I had another go yesterday, again without success.

Your revision does exactly what I wanted to achieve! Thank you, I'll study your code and try to work out where I was going wrong.

[Daniel Marple]

Thanks for all the effort you guys have put in to this. 

This isn't a priority, but if you get a free moment... The next step would be receiving comments back from a client and having to update the masks on the layers already set up. Rather than creating the groups from scratch, is it possible to update the group masks with the new renders?

[c.pfaffenbichler]

Only if there is an unambiguous way of identifying the corresponding files. 

Manual Folder-selection would seem pretty wasteful. 

[c.pfaffenbichler]

But maybe you should change your workflow – instead of using Layer Masks use Linked Smart Objects (either as the basis of a Clipping Mask or as Knock Out); that way you could simply replace the mask files when rendering them anew from the 3D application and update when you next open the image in Photoshop. 

Edit: