Looking to layer approximately 11.4 million images. Could use some script help.

Community Beginner ,
Jul 19, 2021 Jul 19, 2021

Copy link to clipboard

Copied

I should probably start by saying I'm a Photoshop newbie -- I downloaded it to experiment with color grading, as part of a project to remaster NTSC content. 

 

I would like to create a script that performs the following task: 

 

1). Open Image 1A. 

2). Open Image 1B. 

3). Layer Image A on to Image B at 50% opacity. 

4). Flatten the image. 

5). Adjust color balance. 

6). Save result as a new file, Image 1C. 

7). Increment to the next file in the directory (2A, 2B) and perform steps 1-6 again. 

 

It would be *nice* if I had the option to add a third blending option (Image A, B, and C in this case, with final output Image D) but it is not required. It would also be nice to dump the output to a different directory but again, is not required. 

 

I have looked online and found multiple scripts / suggestions for how to merge images side-by-side, but not what I was looking for in terms of creating layers. 

 

I believe doing this through a script is better than through batch actions (if it's even possible that way), but am not sure how to do it. Are there any sites or resources that can teach how to write this kind of script? 

 

TOPICS
Actions and scripting, Windows

Views

152

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jul 19, 2021 Jul 19, 2021

Copy link to clipboard

Copied

You are right, actions can't do this, it is a job for a script and there are many examples of such scripts in the forum. 

 

Steps 1-4 and 6-7 can all be automated, however, more info would be needed if you can't script and require somebody to write this.

 

Step 5 would be a static step, always the same value of colour balance change?

 

Example  exact filenames (say for 3 random sets of images) and required file formats and settings would also be needed.

 

EDIT: You will obviously need storage space and it will likely take working in smaller batches.

 

Photoshop is probably not the best tool for so many images, something command line based with no GUI would be more productive.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jul 19, 2021 Jul 19, 2021

Copy link to clipboard

Copied

Step 5 is a static step. All changes performed would be identical to every image. The actual color processing steps are pretty simple. It's the sheer number of files that's so daunting. 

 

File names would be as follows: 

 

000000A.png - 065142A.png. 

000000B.png - 065142B.png 

 

Storage requirements are not a problem. To do this kind of work I'll need approximately 600GB of storage to handle intermediate files. Intermediate files can be deleted between clips, so I don't need more than that. 

 

"Photoshop is probably not the best tool for so many images, something command line based with no GUI would be more productive."

 

I am open to suggestions. I did not find AviSynth to be effective and have had trouble performing analogous grading in DaVinci Resolve Studio 17. I have also found that DVRS sometimes creates blurred frames when combining output, even when blurred frames should not exist based on the outputs being combined. It would be simpler to use a video editor instead of an image editor, but PS seems to yield better results. Still working on that. 

 

I do not know how to script yet and would appreciate help, but I taught myself AviSynth scripting for this project and have programmed in the distant past. If you say there are examples already in-forum I'll dig around more and try to find them. The threads I found to-date dealt with related but distinct ideas. 

 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jul 19, 2021 Jul 19, 2021

Copy link to clipboard

Copied

Questions:

 

* Are the A and B files in the same folder, or separate input folders?

 

* Input is PNG, is the output PNG? If so, what method (save or export save for web legacy) and what settings?

 

* Where are the output files saved to, is it different to the input?

 

* What are the pixel dimensions of the files?

 

* Are they 8-bit RGB?

 

__________________

 

Possible alternative software to look into could be:

 

ImageMagick (CLI)

GraphicsMagick (CLI)  

NConvert (CLI)

XnConvert: The Best Bulk Image Converter (GUI)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jul 19, 2021 Jul 19, 2021

Copy link to clipboard

Copied

So you have 11.4M Pictures  you have Paired paired  that 5.7M pairs.  You want a script the create the 5.7 layers documents and  5.7 layered PSD files.   That means the script has to process 17.1 million files. How long do you think the script will run.  17.1 million file. That a lot of image file. 

JJMack

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jul 20, 2021 Jul 20, 2021

Copy link to clipboard

Copied

That's what I was thinking.  When I saw him mention 11.4 million images I think I did a recreation of the blinking man meme. I didn't even know that it was even possible to do that many files. I guess I still have things to learn, despite my years of using Photoshop (primarily for graphic design & illustration)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jul 20, 2021 Jul 20, 2021

Copy link to clipboard

Copied

It's 11.4M images in total for the *entirety* of the project. Practically speaking, I'd like to be able to do it in batches of 65,142, but I'm actually not sure Photoshop can handle that many files either. 

 

If I were to complete this project in Photoshop using the method I've described, for both projects I'm working on, the total amount of images processed would come to -- and I swear, I am not exaggerating this -- 45,859,968.

 

That's an approximation -- but it ought to be accurate to within 100,000 images.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jul 20, 2021 Jul 20, 2021

Copy link to clipboard

Copied

The following script will process A and B "paired" files in a single, top level folder and will then export them to PNG in a new folder under the top level folder named "C-Versions" using a C at the end of the original filename.

 

You will need to alter the values in the colorBalance() function. You will also likely need to look at the values in the saveForWebPNG function as well.

 

Compare to your current process to ensure that the end result is correct. Work on a small subset of the original images to ensure that the pair processing is sorting correctly.

 

The script is easily modified if you wish to combine ABC files and output a combined D version, however, more info would be required.

 

/*
Combine File A & B to New File C v1.jsx
Stephen Marsh, 20 July 2021
Looking to layer approximately 11.4 million images. Could use some script help.
https://community.adobe.com/t5/photoshop-ecosystem/looking-to-layer-approximately-11-4-million-images-could-use-some-script-help/td-p/12187411
*/

#target photoshop

var restoreDialogMode = app.displayDialogs;
app.displayDialogs = DialogModes.NO;

main();

function main() {

    inputFolder = Folder.selectDialog("Please select the folder with files to process");
    if (inputFolder === null) return;
    var newPath = Folder(decodeURI(inputFolder + "/C-Versions"));
    if (!newPath.exists) newPath.create();
    var fileList = inputFolder.getFiles(/\.(jpg|jpeg|tif|tiff|png|psd|psb)$/i);
    fileList.sort();

    var setQty = 2;
    var inputCount = fileList.length;
    var cancelScript1 = !(inputCount === 0);
    if (cancelScript1 === false) {
        alert('Zero input files found, script cancelled!');
        return;
    }

    // Thanks to Kukurykus for the advice to test using % modulus
    var cancelScript2 = !(inputCount % setQty);
    alert(inputCount + ' input files stacked into sets of ' + setQty + ' will produce ' + inputCount / setQty + ' output files.');
    if (cancelScript2 === false) {
        alert('Script cancelled as the quantity of input files are not evenly divisible by the set quantity.');
        return;
    }

    while (fileList.length) {
        for (var a = 0; a < setQty; a++) {
            try {
                app.open(fileList.pop());
            } catch (e) { }
        }
        processFiles();
    }

    app.beep();
    alert("Script completed!");

    function processFiles() {
        try {
            var newName = app.documents[0].name.replace(/\.[^\.]+$/, '').replace(/.$/, 'C');
            app.activeDocument = documents[1];
            app.activeDocument.activeLayer.duplicate(documents[0]);
            app.activeDocument = documents[0];
            var doc = app.activeDocument;
            doc.activeLayer.opacity = 50.0;
            doc.flatten();
            colorBalance(true);
            app.displayDialogs = restoreDialogMode;
        } catch (e) { }

        var saveFile = File(newPath + "/" + newName + ".png");
        saveForWebPNG(saveFile);

        while (app.documents.length) {
            app.activeDocument.close(SaveOptions.DONOTSAVECHANGES);
        }
    }
}

// Functions

function colorBalance(preserveLuminosity) {
    var s2t = function (s) {
        return app.stringIDToTypeID(s);
    };
    var descriptor = new ActionDescriptor();
    var list = new ActionList();
    var list2 = new ActionList();
    var list3 = new ActionList();

    list3.putInteger(0); // Red
    list3.putInteger(0); // Green
    list3.putInteger(0); // Blue
    descriptor.putList(s2t("shadowLevels"), list3);

    list.putInteger(0); // Red
    list.putInteger(0); // Green
    list.putInteger(0); // Blue
    descriptor.putList(s2t("midtoneLevels"), list);

    list2.putInteger(0); // Red
    list2.putInteger(0); // Green
    list2.putInteger(0); // Blue
    descriptor.putList(s2t("highlightLevels"), list2);

    descriptor.putBoolean(s2t("preserveLuminosity"), preserveLuminosity);
    executeAction(s2t("colorBalance"), descriptor, DialogModes.NO);
}

function saveForWebPNG(saveFile) {
    var sfwOptions = new ExportOptionsSaveForWeb();
    sfwOptions.format = SaveDocumentType.PNG;
    sfwOptions.PNG8 = false;
    sfwOptions.quality = 100;
    sfwOptions.interlaced = false;
    sfwOptions.transparency = true;
    sfwOptions.includeProfile = false;
    //sfwOptions.matte = true;
    //sfwOptions.matteColor = 0, 0, 0;
    activeDocument.exportDocument(saveFile, ExportType.SAVEFORWEB, sfwOptions);
}

 

Downloading and Installing Adobe Scripts

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jul 20, 2021 Jul 20, 2021

Copy link to clipboard

Copied

@Stephen_A_Marsh : Thank you so much.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jul 20, 2021 Jul 20, 2021

Copy link to clipboard

Copied

How long will the run time be to process the 17,100,000 files

JJMack

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jul 20, 2021 Jul 20, 2021

Copy link to clipboard

Copied

That's turning out to be a sticking point. The script that @Stephen_A_Marsh posted seems to choke when I hand it the entire input directory at once. Said directory contains 61,762 images divided into two sets of 30,881 each (000000A.png / 000000B.png). 

 

When I tested the script on a smaller run of 15 files, it worked perfectly, but when I asked it to ingest the larger directory, it sat on a single thread for 60 minutes before I turned it off. Switching Photoshop to CPU mode did not help. Memory usage remained quite low. The validation message in the script about the final number of images created -- and *thanks* for that -- did not display. 

 

It may not be worth trying to troubleshoot, however, because it appears to take Photoshop about 9 seconds per frame to perform this operation. I'm hoping for something more along the lines of 1s, and that may not be possible. 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jul 20, 2021 Jul 20, 2021

Copy link to clipboard

Copied

I would look into using ImageMagick to do this, it will likely be MUCH faster.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jul 20, 2021 Jul 20, 2021

Copy link to clipboard

Copied

I'm looking into it. Certainly not opposed. Just need to figure out the
scripting.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jul 20, 2021 Jul 20, 2021

Copy link to clipboard

Copied

There are good user communities, you should be able to find a similar post or request help, good luck!

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jul 30, 2021 Jul 30, 2021

Copy link to clipboard

Copied

LATEST

So how did this project turn out?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines