Merge Group vs convert to Smart Object and rasterize? [Script]

[dendenis82]

I'd want you share your opinion.
Іs complex group with styles and different overlay modes converted to smart object before rasterizing be more clean than direct rasterizing?
I am using this in a script, it would be faster direct rasterization.  Just want to make sure it's not hallucination that converting to smart object before rasterization is more stable. 

[c.pfaffenbichler]

 Just want to make sure it's not hallucination that converting to smart object before rasterization is more stable. 

It shouldn’t be. 

Could you please post screenshots taken at View > 100% with the pertinent Panels (Toolbar, Layers, Options Bar, …) visible to illustrate what you bean by »more stable«? 

[dendenis82]

As one of the cases I'm facing right now. Can't remember for sure what the others are like, but it refers to complex combinations(blend modes, clip mask, filter masks, objects with styles as mask etc. and whatever it possible to collect in single group).

[r-bin]

Interesting case.

Somehow connected with the fact that there is both a mask and effects on the group.

Maybe even a bug.

This can be avoided by placing such a group in another one, but without a mask and effects, and then merge it. But no one encourages you not to use smart objects.

[dendenis82]

Hm, nice workaround. Placing in another group is still faster than converting to smart object, at least for heavy groups.

[dendenis82]

I noticed even creating new group above the original unstable group, selecting them both and merging do the job. Not only nesting inside new group. But I struggle to write function for this that work: nesting inside new group and merging or merging new group with the original below.
P.S. of course, it can be done via smart objects, but it's speed matters thing, tending to have it close to realtime.

[c.pfaffenbichler]

AM-code for grouping and merging: 

// =======================================================
var desc22 = new ActionDescriptor();
var ref2 = new ActionReference();
ref2.putClass( stringIDToTypeID( "layerSection" ) );
desc22.putReference( stringIDToTypeID( "null" ), ref2 );
var ref3 = new ActionReference();
ref3.putEnumerated( stringIDToTypeID( "layer" ), stringIDToTypeID( "ordinal" ), stringIDToTypeID( "targetEnum" ) );
desc22.putReference( stringIDToTypeID( "from" ), ref3 );
desc22.putString( stringIDToTypeID( "name" ), "Group 1" );
executeAction( stringIDToTypeID( "make" ), desc22, DialogModes.NO );
// =======================================================
executeAction( stringIDToTypeID( "mergeLayersNew" ), undefined, DialogModes.NO );

[dendenis82]

Yes, it works! And it's fast.
Is it possible to keep original name?

[c.pfaffenbichler]

You can evaluate the original name (activeDocument.activeLayer.name) and apply that to the resulting layer. 

[dendenis82]

Thank you!
I've updated the script by working version with classic approach. But still planning to try your short code for grouping and merging.

[c.pfaffenbichler]

As one of the cases I'm facing right now. Can't remember for sure what the others are like, but it refers to complex combinations(blend modes, clip mask, filter masks, objects with styles as mask etc. and whatever it possible to collect in single group).

By @dendenis82

Please provide the file (with just the affected Group and the background) for testing. 

[dendenis82]

Here is affected group.

[c.pfaffenbichler]

Thank you! 

The issue appears to be that the stroke is applied to the content of the Group and the Layer Mask is maintained. 

As @r-bin indicated this could be considered a bug … 

The Group-in-Group-approach seems a good work-around. 

[dendenis82]

I'm trying to make a script with workaround approach, but can't get it working; 
Update: got working version with the same approach. Seems like selection of two groups was missing at the moment before merging;

I have to say, it's tripple faster than approach with converting to smart object and merge approach.

// ?is it time worth to check the group have styles or not; make version with c.pfaffenbichler AM-code for grouping and merging code block , if keeping original name is possible. Make speedtest.
function flattenRootObjectsInCurrentGroup() {
    // Get the active document
    var doc = app.activeDocument;

    // Get the active layer
    var targetLayer = doc.activeLayer;

    // Ensure the active layer is a group
    if (targetLayer.typename !== "LayerSet") {
        // Exit silently if the active layer is not a group
        return;
    }

    // Function to create a new group and merge with the original group
    function handleGroupType(groupLayer) {
        try {
            // Ensure the group layer exists and is valid
            if (!groupLayer || groupLayer.typename !== "LayerSet") {
                throw new Error("Invalid layer group");
            }

            // Create a new group above the current group
            var newGroup = doc.layerSets.add();
            newGroup.name = groupLayer.name; // Keep the same name

            // Position the new group above the original group
            newGroup.move(groupLayer, ElementPlacement.PLACEBEFORE);

            // Select both the original group and the new group
            doc.activeLayer = groupLayer; // Ensure the original group is active
            selectLayer(groupLayer);
            selectLayer(newGroup, true); // Add new group to selection

            // Merge the selected groups
            app.executeAction(stringIDToTypeID("mergeLayers"), undefined, DialogModes.NO);
        } catch (e) {
            $.writeln("Error handling group type: " + e.message);
        }
    }

    // Function to make layer visible and flatten/rasterize
    function processLayer(layer) {
        // Skip if layer is already deleted or is background
        if (!layer || layer.isBackgroundLayer) return;

        // Ensure layer is visible
        layer.visible = true;

        // Handle different layer types
        try {
            if (layer.typename === "LayerSet") {
                // Handle group type
                handleGroupType(layer);
            } else if (layer.typename === "ArtLayer") {
                // If it's a smart object, rasterize
                if (layer.kind === LayerKind.SMARTOBJECT) {
                    layer.rasterize(RasterizeType.ENTIRELAYER);
                } 
                // If layer has styles, rasterize
                else if (layer.layerStyle && layer.layerStyle.enabled) {
                    layer.rasterize(RasterizeType.ENTIRELAYER);
                }
            }
        } catch (e) {
            $.writeln("Error processing layer " + (layer ? layer.name : "Unknown") + ": " + e.message);
        }
    }

    // Helper function to select a layer
    function selectLayer(layer, addToSelection) {
        var desc = new ActionDescriptor();
        var ref = new ActionReference();
        ref.putIdentifier(stringIDToTypeID("layer"), layer.id);
        desc.putReference(stringIDToTypeID("null"), ref);
        if (addToSelection) {
            desc.putEnumerated(stringIDToTypeID("selectionModifier"), stringIDToTypeID("selectionModifierType"), stringIDToTypeID("addToSelection"));
        }
        desc.putBoolean(stringIDToTypeID("makeVisible"), false);
        executeAction(stringIDToTypeID("select"), desc, DialogModes.NO);
    }

    // Store the original layer count
    var originalLayerCount = targetLayer.layers.length;

    // Process layers from top to bottom
    var i = originalLayerCount - 1;
    while (i >= 0) {
        // Use getByIndex to avoid issues with changing layer stack
        var currentLayer = targetLayer.layers[i];
        processLayer(currentLayer);
        i--;
    }
}

// Run the function
flattenRootObjectsInCurrentGroup();

[c.pfaffenbichler]

Another option might be copying the Layer Style, removing it, merging and then applying the Layer Style again. 

// mergeGroupButMaintainLayerStyle
// 2024, use it at your own risk;
if (app.documents.length > 0) {
var myDocument = app.activeDocument;
var selectedLayers = collectSelectedLayersBounds();
if (selectedLayers.length == 1 && activeDocument.activeLayer.typename == "LayerSet") {
// =======================================================
executeAction( stringIDToTypeID( "copyEffects" ), undefined, DialogModes.NO );
// =======================================================
var desc5 = new ActionDescriptor();
var ref1 = new ActionReference();
ref1.putEnumerated( stringIDToTypeID( "layer" ), stringIDToTypeID( "ordinal" ), stringIDToTypeID( "targetEnum" ) );
desc5.putReference( stringIDToTypeID( "null" ), ref1 );
executeAction( stringIDToTypeID( "disableLayerStyle" ), desc5, DialogModes.NO );
// =======================================================
executeAction( stringIDToTypeID( "mergeLayersNew" ), undefined, DialogModes.NO );
// =======================================================
var desc13 = new ActionDescriptor();
desc13.putBoolean( stringIDToTypeID( "allowPasteFXOnLayerSet" ), true );
executeAction( stringIDToTypeID( "pasteEffects" ), desc13, DialogModes.NO );
}
};
////// collect bounds of selected layers //////
function collectSelectedLayersBounds () {
// set to pixels;
var originalRulerUnits = app.preferences.rulerUnits;
app.preferences.rulerUnits = Units.PIXELS;
// get selected layers;
var selectedLayers = new Array;
var ref = new ActionReference();
ref.putEnumerated( charIDToTypeID("Dcmn"), charIDToTypeID("Ordn"), charIDToTypeID("Trgt") );
var desc = executeActionGet(ref);
if (desc.getBoolean(stringIDToTypeID("hasBackgroundLayer")) == true) {var theAdd =0}
else {var theAdd = 1};
if( desc.hasKey( stringIDToTypeID( 'targetLayers' ) ) ){
desc = desc.getList( stringIDToTypeID( 'targetLayers' ));
var c = desc.count;
var selectedLayers = new Array();
// run through selected layers;
for(var i=0;i<c;i++){
var theIndex = desc.getReference( i ).getIndex()+theAdd;
// get id for solid color layers;
try {
var ref = new ActionReference();
ref.putIndex( charIDToTypeID("Lyr "), theIndex ); 
var layerDesc = executeActionGet(ref);
var theName = layerDesc.getString(stringIDToTypeID('name'));
var theIdentifier = layerDesc.getInteger(stringIDToTypeID ("layerID"));
var theBounds = layerDesc.getObjectValue(stringIDToTypeID("bounds"));
var theseBounds = [theBounds.getUnitDoubleValue(stringIDToTypeID("left")), theBounds.getUnitDoubleValue(stringIDToTypeID("top")), theBounds.getUnitDoubleValue(stringIDToTypeID("right")), theBounds.getUnitDoubleValue(stringIDToTypeID("bottom"))];
selectedLayers.push([theName, theIdentifier, theseBounds]);
} catch (e) {};
};
// if only one:
}else{
var ref = new ActionReference();
ref.putEnumerated( charIDToTypeID("Lyr "), charIDToTypeID("Ordn"), charIDToTypeID("Trgt") );
var layerDesc = executeActionGet(ref);
try {
var theName = layerDesc.getString(stringIDToTypeID('name'));
var theIdentifier = layerDesc.getInteger(stringIDToTypeID ("layerID"));
var theBounds = layerDesc.getObjectValue(stringIDToTypeID("bounds"));
var theseBounds = [theBounds.getUnitDoubleValue(stringIDToTypeID("left")), theBounds.getUnitDoubleValue(stringIDToTypeID("top")), theBounds.getUnitDoubleValue(stringIDToTypeID("right")), theBounds.getUnitDoubleValue(stringIDToTypeID("bottom"))];
selectedLayers = [[theName, theIdentifier, theseBounds]]
} catch (e) {};
};
// reset;
app.preferences.rulerUnits = originalRulerUnits;
return selectedLayers;
};

[r-bin]

When you merge a group, you lose (cut off) all the pixels that are outside the canvas. This is the only downside.

Converting to a smart object takes longer and can be affected by blurring due to accidental scaling.

There may also be problems with non-standard profiles and bit modes.

[c.pfaffenbichler]

When you merge a group, you lose (cut off) all the pixels that are outside the canvas. 

This does not appear to happen in all cases. (With Layer Masks on Adjustment Layers it certainly can prove a problem later on.)

[r-bin]

I can't reproduce this (CS6).

I remember that on some version of CC (~2020) they started to partially change the algorithms for merging layers and groups. Because of this, some old scripts stopped working correctly.