Clipping Mask Script issues in PS 2025 on Mac

[polpowsmp]

Hi all,

I've been 'fighting' with scripts to autogenerate about 69K country flag-based NFTs. I got most of it working fine, but the final issue relates to applying a clipping mask to a rectangular flag (to give it rounded corners). The template looks like this (flag of china with a vector mask and effects applied)

 However, after multiple different attempts, finally settling on clipping mask, the output still looks like this:

 Here is a summary:

Brief Summary of the Attempted Methods to Apply Rounded Corners

To give the new flag (Flag - New) rounded corners with a 150-pixel radius in Adobe Photoshop, we’ve attempted the following approach:

1. Create a Rounded Rectangle Selection: We generate a rounded rectangle selection matching the flag’s dimensions (1650x1100 px) with a 150-pixel corner radius using the executeAction method with the setd command, specifying a roundedRect shape.
2. Fill the Selection to Create a Mask Layer: We create a new layer (Flag Round Mask), set the foreground color to white, and fill the selection to produce a solid rounded rectangle shape.
3. Constrain the Mask Layer Bounds: To correct the layer’s bounds (which initially span the entire canvas), we reselect the filled area, copy it, and paste it into a new layer, ensuring the bounds match the flag’s dimensions.
4. Clip the Flag to the Mask: We place the flag (Flag - New) inside a group (Flag - New Group) and clip this group to the Flag Round Mask layer using the grouped property, aiming to mask the flag with rounded corners.
5. Apply Effects: We copy effects (e.g., Drop Shadow, Color Overlay) from the original Template Mask Shape to the Flag - New Group to maintain the visual style.

Despite these steps, the flag’s corners remain square in the output, indicating that the clipping mask or selection process isn’t working as expected in Photoshop 25.5.0 on macOS 15.4.1.

Here’s how we’re currently scripting the clipping mask in the replaceFlag function of your photoshop.jsx file, with detailed steps and code snippets:

1. Setup and Flag Positioning

We first position the new flag (Flag - New) at the correct coordinates (X: 928, Y: 1297) with dimensions 1650x1100 px, centered within a 1900x1300 px area. The flag is placed inside a group named Flag - New Group:

// Create a new group for the flag
var newFlagGroup = doc.layerSets.add();
newFlagGroup.name = "Flag - New Group";
newFlagLayer.move(newFlagGroup, ElementPlacement.INSIDE);
log("Created Flag - New Group and moved Flag - New inside it");

2. Create the Rounded Rectangle Selection

We create a rounded rectangle selection matching the flag’s bounds with a 150-pixel corner radius using the executeAction method:

log("Creating rounded rectangle selection with 150px corner radius");
try {
var selDesc = new ActionDescriptor();
var ref = new ActionReference();
ref.putProperty(charIDToTypeID("Chnl"), charIDToTypeID("fsel"));
selDesc.putReference(charIDToTypeID("null"), ref);

var shapeDesc = new ActionDescriptor();
var rectDesc = new ActionDescriptor();
rectDesc.putUnitDouble(charIDToTypeID("Top "), charIDToTypeID("Pxl "), newY); // newY = 1297
rectDesc.putUnitDouble(charIDToTypeID("Left"), charIDToTypeID("Pxl "), newX); // newX = 928
rectDesc.putUnitDouble(charIDToTypeID("Btom"), charIDToTypeID("Pxl "), newY + newHeight); // newY + 1100 = 2397
rectDesc.putUnitDouble(charIDToTypeID("Rght"), charIDToTypeID("Pxl "), newX + newWidth); // newX + 1650 = 2578
rectDesc.putUnitDouble(stringIDToTypeID("cornerRadius"), charIDToTypeID("Pxl "), 150);
shapeDesc.putObject(charIDToTypeID("T "), stringIDToTypeID("roundedRect"), rectDesc);
selDesc.putObject(charIDToTypeID("T "), charIDToTypeID("Shp "), shapeDesc);

executeAction(charIDToTypeID("setd"), selDesc, DialogModes.NO);
log("Successfully created rounded rectangle selection");
} catch (e) {
log("Error creating rounded rectangle selection: " + e);
throw e;
}

• This creates a selection from (928,1297) to (2578,2397) with 150-pixel rounded corners.
• The log confirms this step succeeds, but we’ve observed issues where the selection may not be set correctly in this environment.

3. Create and Fill the Flag Round Mask Layer

We create a new layer (Flag Round Mask), position it directly below Flag - New Group, set the foreground color to white, and fill the selection:

var maskLayer = app.activeDocument.artLayers.add();
maskLayer.name = "Flag Round Mask";
maskLayer.move(newFlagGroup, ElementPlacement.PLACEAFTER); // Move below the group
app.activeDocument.activeLayer = maskLayer;
log("Created new layer for mask: " + maskLayer.name);

// Set foreground color to white (RGB 255,255,255)
setForegroundColor(255, 255, 255);

// Fill selection with white (using DOM method)
try {
app.activeDocument.selection.fill(app.foregroundColor);
log("Successfully filled selection with foreground color");
} catch (e) {
log("Error filling selection with foreground color: " + e);
throw e;
}

• ElementPlacement.PLACEAFTER ensures Flag Round Mask is directly below Flag - New Group in the layer stack, a requirement for clipping masks.

4. Constrain the Flag Round Mask Bounds

The filled layer initially has bounds matching the entire canvas ([0,0→3520,5224]). To fix this, we reselect the area, copy, and paste into a new layer:

try {
doc.selection.select([
[newX, newY],
[newX + newWidth, newY],
[newX + newWidth, newY + newHeight],
[newX, newY + newHeight]
]);
doc.selection.copy();
maskLayer.remove(); // Remove the old layer with incorrect bounds
maskLayer = doc.artLayers.add();
maskLayer.name = "Flag Round Mask";
maskLayer.move(newFlagGroup, ElementPlacement.PLACEAFTER); // Move below the group again
app.activeDocument.activeLayer = maskLayer;
doc.paste();
log("Copied and pasted filled area to constrain Flag Round Mask bounds");
} catch (e) {
log("Error constraining Flag Round Mask bounds: " + e);
throw e;
}

• After this step, the Flag Round Mask bounds are [928,1297→2578,2397], matching the flag’s dimensions.

5. Apply the Clipping Mask

We clip Flag - New Group to Flag Round Mask using the grouped property:

try {
newFlagGroup.grouped = true;
log("Successfully applied clipping mask via DOM grouped property");
log("Flag - New Group grouped property: " + newFlagGroup.grouped);
} catch (e) {
log("Error applying clipping mask via DOM: " + e);
throw e;
}

• newFlagGroup.grouped = true sets Flag - New Group to be clipped to the layer directly below it (Flag Round Mask).
• The log confirms Flag - New Group grouped property: true, indicating the property was set, but the visual output shows square corners, suggesting the mask isn’t effective.

6. Layer Structure Post-Clipping

The layer structure after applying the clipping mask:

- Flag - New Group (Visible, Bounds: [924,1293→2582,2401])
- Flag - New (Visible)
- Flag Round Mask (Visible, Bounds: [928,1297→2578,2397])

Issue and Request for Help

Despite the clipping mask being applied, the flag in the PNG output (1.png) has square corners, not the expected 150-pixel rounded corners. We suspect compatibility issues with Photoshop 25.5.0 on macOS 15.4.1, as previous attempts (e.g., accessing doc.selection.bounds) failed with "No such element" errors. We’d appreciate insights from the Adobe scripting community on:

• Ensuring the rounded rectangle selection is created and filled correctly.
• Verifying that the grouped property works for a LayerSet (Flag - New Group) clipped to an ArtLayer (Flag Round Mask).
• Alternative methods to achieve rounded corners, such as using a layer mask or vector shape, given our environment’s limitations (e.g., Make commands fail).

Any help greatly appreciated!

[Stephen Marsh · Accepted solution]

With your target layer active and above the round cornered shape, you can try:

app.runMenuItem(stringIDToTypeID('groupEvent'));

Or:

var idgroupEvent = stringIDToTypeID( "groupEvent" );
    var desc296 = new ActionDescriptor();
    var idnull = stringIDToTypeID( "null" );
        var ref14 = new ActionReference();
        var idlayer = stringIDToTypeID( "layer" );
        var idordinal = stringIDToTypeID( "ordinal" );
        var idtargetEnum = stringIDToTypeID( "targetEnum" );
        ref14.putEnumerated( idlayer, idordinal, idtargetEnum );
executeAction( idgroupEvent, desc296, DialogModes.NO );

[polpowsmp]

Ok thank you very much - I'll give that a try!

[Stephen Marsh]

Ok thank you very much - I'll give that a try!

By @polpowsmp

You're welcome, let me know how it goes!