Programmatically convert shape layer to masked color fill layer

[Jack Brannen]

I'm looking for a scripting or actions wizard—

I have an illustration workflow that involves making a drawing in Illustrator and then pasting into Photoshop as many different shape layers. Then I convert these shape layers into masked color fill layers. The whole point of this workflow is being able to use Photoshop filters on the masks to rough up the edges of the shape. This workflow works great for me.  

BUT the process of manually converting each shape layer into a masked color fill layer is a little tedious. What's tricky is that the color fill layer needs to retain the color of the shape layer.

I can use shortcuts and actions to speed this up a little, but it quickly gets unwieldy if I have more than a few dozen layers.

Surely there's some way to automate this?

Here's a screencapture of me doing it manually:

[r-bin · Accepted solution]

Logically, you want to convert the vector mask to a regular one. Try this script for the active shape layer.

try {

// vector masrk to selection (antialias, feather 0)

var d = new ActionDescriptor();  
var r = new ActionReference();  
r.putProperty(stringIDToTypeID("channel"), stringIDToTypeID("selection"));  
d.putReference(stringIDToTypeID("null"), r);  
var r1 = new ActionReference();  
r1.putEnumerated(stringIDToTypeID("path"), stringIDToTypeID("path"), stringIDToTypeID("vectorMask"));
r1.putEnumerated(stringIDToTypeID("layer"), stringIDToTypeID("ordinal"), stringIDToTypeID("targetEnum"));
d.putReference(stringIDToTypeID("to"), r1);  
d.putBoolean(stringIDToTypeID("antiAlias"), true);  
d.putUnitDouble(stringIDToTypeID("feather"), stringIDToTypeID("pixelsUnit"), 0);  
executeAction(stringIDToTypeID("set"), d, DialogModes.NO);  


// create pixel mask from selection 

var d = new ActionDescriptor();
d.putClass(stringIDToTypeID("new"), stringIDToTypeID("channel"));
var r = new ActionReference();
r.putEnumerated(stringIDToTypeID("channel"), stringIDToTypeID("channel"), stringIDToTypeID("mask"));
d.putReference(stringIDToTypeID("at"), r);
d.putEnumerated(stringIDToTypeID("using"), stringIDToTypeID("userMaskOptions"), stringIDToTypeID("revealSelection"));
executeAction(stringIDToTypeID("make"), d, DialogModes.NO);


// delete current vector mask

var d = new ActionDescriptor();
var r = new ActionReference();
r.putEnumerated(stringIDToTypeID("path"), stringIDToTypeID("path"), stringIDToTypeID("vectorMask"));
r.putEnumerated(stringIDToTypeID("layer"), stringIDToTypeID("ordinal"), stringIDToTypeID("targetEnum"));
d.putReference(stringIDToTypeID("null"), r);
executeAction(stringIDToTypeID("delete"), d, DialogModes.NO);


} catch (e) { alert(e.line+ "\n\n" +e); }

[Jack Brannen]

😍 Wow, amazing! This works perfectly. I literally gasped and laughed out loud. This will save me so much time! Thank you, kind internet stranger!

[c.pfaffenbichler · Accepted solution]

Not elegant, but I wonder whether existing Layer Masks may merit consideration.

Edited: updated to the code to exclude Shape Layers with Strokes

// change shape layer vector masks to layer masks;
// 2024, use it at your own risk;
if (app.documents.length > 0) {
var theLayers = collectShapeLayers();
var originalRulerUnits = app.preferences.rulerUnits;
app.preferences.rulerUnits = Units.PIXELS;
// process layers;
for (var m = 0; m < theLayers.length; m++) {
// if layer has no stroke;
if (theLayers[m][3] == false) {
selectLayerByID(theLayers[m][1], false);
// load as selection, apply as mask and delete vector mask;
var idchannel = stringIDToTypeID( "channel" );
var idpath = stringIDToTypeID( "path" );
var idnull = stringIDToTypeID( "null" );
var idvectorMask = stringIDToTypeID( "vectorMask" );
// =======================================================
    var desc5 = new ActionDescriptor();
        var ref1 = new ActionReference();
        ref1.putProperty( idchannel, stringIDToTypeID( "selection" ) );
    desc5.putReference( idnull, ref1 );
        var ref2 = new ActionReference();
        ref2.putEnumerated( idpath, idpath, idvectorMask );
        ref2.putEnumerated( stringIDToTypeID( "layer" ), stringIDToTypeID( "ordinal" ), stringIDToTypeID( "targetEnum" ) );
    desc5.putReference( stringIDToTypeID( "to" ), ref2 );
    desc5.putInteger( stringIDToTypeID( "version" ), 1 );
    desc5.putBoolean( stringIDToTypeID( "vectorMaskParams" ), true );
executeAction( stringIDToTypeID( "set" ), desc5, DialogModes.NO );
if (theLayers[m][2] == false) {
    var desc7 = new ActionDescriptor();
    desc7.putClass( stringIDToTypeID( "new" ), idchannel );
        var ref3 = new ActionReference();
        var idmask = stringIDToTypeID( "mask" );
        ref3.putEnumerated( idchannel, idchannel, idmask );
    desc7.putReference( stringIDToTypeID( "at" ), ref3 );
    desc7.putEnumerated( stringIDToTypeID( "using" ), stringIDToTypeID( "userMaskEnabled" ), stringIDToTypeID( "revealSelection" ) );
executeAction( stringIDToTypeID( "make" ), desc7, DialogModes.NO );
} else {
    var desc10 = new ActionDescriptor();
        var ref6 = new ActionReference();
        ref6.putEnumerated( idchannel, idchannel, stringIDToTypeID( "mask" ) );
    desc10.putReference( stringIDToTypeID( "null" ), ref6 );
    var idmakeVisible = stringIDToTypeID( "makeVisible" );
    desc10.putBoolean( idmakeVisible, false );
executeAction( stringIDToTypeID( "select" ), desc10, DialogModes.NO );
// =======================================================
executeAction( stringIDToTypeID( "inverse" ), undefined, DialogModes.NO );
// =======================================================
    var desc18 = new ActionDescriptor();
        var ref8 = new ActionReference();
        ref8.putProperty( stringIDToTypeID( "color" ), stringIDToTypeID( "colors" ) );
    desc18.putReference( stringIDToTypeID( "null" ), ref8 );
executeAction( stringIDToTypeID( "reset" ), desc18, DialogModes.NO );
// =======================================================
    var desc5 = new ActionDescriptor();
        var ref2 = new ActionReference();
        ref2.putProperty( stringIDToTypeID( "color" ), stringIDToTypeID( "colors" ) );
    desc5.putReference( stringIDToTypeID( "null" ), ref2 );
executeAction( stringIDToTypeID( "exchange" ), desc5, DialogModes.NO );
// =======================================================
    var desc13 = new ActionDescriptor();
    desc13.putEnumerated( stringIDToTypeID( "using" ), stringIDToTypeID( "fillContents" ), stringIDToTypeID( "foregroundColor" ) );
    desc13.putUnitDouble( stringIDToTypeID( "opacity" ),stringIDToTypeID( "percentUnit" ), 100.000000 );
    desc13.putEnumerated( stringIDToTypeID( "mode" ), stringIDToTypeID( "blendMode" ), stringIDToTypeID( "normal" ) );
executeAction( stringIDToTypeID( "fill" ), desc13, DialogModes.NO );
};
// =======================================================
    var desc9 = new ActionDescriptor();
        var ref4 = new ActionReference();
        ref4.putEnumerated( idpath, idpath, idvectorMask );
        ref4.putEnumerated( stringIDToTypeID( "layer" ), stringIDToTypeID( "ordinal" ), stringIDToTypeID( "targetEnum" ) );
    desc9.putReference( idnull, ref4 );
executeAction( stringIDToTypeID( "delete" ), desc9, DialogModes.NO );
};
};
// reset;
app.preferences.rulerUnits = originalRulerUnits;
};
////////////////////////////////////
////// collect shape layers //////
function collectShapeLayers () {
// get number of layers;
var ref = new ActionReference();
ref.putEnumerated( charIDToTypeID("Dcmn"), charIDToTypeID("Ordn"), charIDToTypeID("Trgt") ); 
var applicationDesc = executeActionGet(ref);
var theNumber = applicationDesc.getInteger(stringIDToTypeID("numberOfLayers"));
// process the layers;
var theLayers = new Array;
for (var m = 0; m <= theNumber; m++) {
try {
var ref = new ActionReference();
ref.putIndex( charIDToTypeID( "Lyr " ), m);
var layerDesc = executeActionGet(ref);
var layerSet = typeIDToStringID(layerDesc.getEnumerationValue(stringIDToTypeID("layerSection")));
var isBackground = layerDesc.getBoolean(stringIDToTypeID("background"));
// if not layer group collect values;
if (layerSet != "layerSectionEnd" && layerSet != "layerSectionStart" && isBackground != true) {
var theName = layerDesc.getString(stringIDToTypeID('name'));
var theID = layerDesc.getInteger(stringIDToTypeID('layerID'));
var theIndex = layerDesc.getInteger(stringIDToTypeID('itemIndex'));
var theKind = layerDesc.getInteger(stringIDToTypeID('layerKind'));
var hasLayerMask = layerDesc.hasKey(charIDToTypeID("Usrs"));
var hasStroke = layerDesc.hasKey(stringIDToTypeID("AGMStrokeStyleInfo"));
if (hasStroke == true) {hasStroke = layerDesc.getObjectValue(stringIDToTypeID("AGMStrokeStyleInfo")).getBoolean(stringIDToTypeID("strokeEnabled"))};
if (theKind == 4) {
theLayers.push([theName, theID, hasLayerMask, hasStroke])
}
};
}
catch (e) {};
};
return theLayers;
};
////// based on code by mike hale, via paul riggott //////
function selectLayerByID(id,add){ 
add = undefined ? add = false:add 
var ref = new ActionReference();
ref.putIdentifier(charIDToTypeID("Lyr "), id);
var desc = new ActionDescriptor();
desc.putReference(charIDToTypeID("null"), ref );
if(add) desc.putEnumerated( stringIDToTypeID( "selectionModifier" ), stringIDToTypeID( "selectionModifierType" ), stringIDToTypeID( "addToSelection" ) ); 
desc.putBoolean( charIDToTypeID( "MkVs" ), false ); 
try{
executeAction(charIDToTypeID("slct"), desc, DialogModes.NO );
}catch(e){
alert(e.message); 
}
};

[Jack Brannen]

Thank you for all this work! This one didn't work for me.

[PECourtejoie]

Thank you for the feedback, could you please elaborate how it did not fit your needs, it might help Christof to change/tweak his code.

[Jack Brannen]

@PECourtejoie Thank you, yes. Nothing happens when I run it. 

[c.pfaffenbichler]

Could you please provide a file on which the Script failed? 

[Jack Brannen]

@c.pfaffenbichler Thank you, certainly. I just made a brand new PSD with three circles.

Here's a link, but this is just from my Dropbox so it won't be permanent.

[c.pfaffenbichler]

Thanks! 

The layers’ stroke settings seem to be different (though there is no stroke) from the test file I had created. 

I amended the code in the old post if you want to try it again. 

[Jack Brannen]

Wow, that's amazing! It did all of the layers at once. What a gift! Thank you for working on this. This will enable me to do more than I could before.

[Jack Brannen]

@c.pfaffenbichler I am still using your fantastic script. Something it does not handle at the moment is a compound shape pasted as a layer from Illustrator. Would that be difficult to add?

Steps to reproduce, if needed:

1. Make any compound shape in Illustrator

2. Copy

3. In Photoshop, paste

4. Choose to paste as "Layers"

Then try to run the script.

Thanks a million.

[Stephen_A_Marsh]

@c.pfaffenbichler I am still using your fantastic script. Something it does not handle at the moment is a compound shape pasted as a layer from Illustrator. Would that be difficult to add?

Steps to reproduce, if needed:

1. Make any compound shape in Illustrator

2. Copy

3. In Photoshop, paste

4. Choose to paste as "Layers"

Then try to run the script.

Thanks a million.

By @Jack Brannen

This creates a raster mask for me and the mask is correct and the fill layer can be easily changed. I'm not seeing the issue?

[Jack Brannen]

@Stephen_A_Marsh Thanks for checking. Odd, just tested again and it isn't for me. Let's see what @c.pfaffenbichler says about my test files… I can always record a screencast.

[Stephen_A_Marsh]

@Stephen_A_Marsh Thanks for checking. Odd, just tested again and it isn't for me. Let's see what @c.pfaffenbichler says about my test files… I can always record a screencast.

By @Jack Brannen

It results in a solid fill layer with raster mask for me. I'll also try your test file.

[c.pfaffenbichler]

Could you provide the ai and the psd with the pasted content for testing? 

[Jack Brannen]

@c.pfaffenbichler Yes, thanks! Attached.

[c.pfaffenbichler]

Thank you; Compound Shapes seem to paste as a plain pixel Layer when set to »Layers«, so there is no vector mask. 

[Jack Brannen]

@c.pfaffenbichler @Stephen_A_Marsh So weird that we're all getting different results. I bet there are paste setting involved somewhere.

@c.pfaffenbichler I get a shape layer when I paste a compound shape:

[c.pfaffenbichler]

One more thing came to my mind: 

What about Shape Layers with a Stroke? 

edit: updated the code