Batch Action Crashes After 7,077 files?

[JoeJoseph]

Hi there, I have a problem with a batch process I'm trying to use to add watermarks. The "watermark" is very simple - plain grey text. However, I have 1.2 million files to process and Photoshop crashes, usually after processing 7,077 files. On average that's only about 450MB, all 1280x720 png files. These are in separate folder of about 55k each. I've tried this on two different computers. Both have drivers and Photoshop up-to-date. I have tried turning off the option to use the video card, as suggested by the troubleshooting guide. I've turned off autosave and tried adjusting the memory limit up and down. No change.

The computer does not freeze. It's just Photoshop that stops. There are no errors in the log file, and it doesn't give me a specific error. It just gives me the option to send or not send a crash report.

Please help. I don't want to have to restart this process every hour or so 170 times. If it's important my primary system is Windows 10, 32 GB memory with an Intel i7-8700k and a GTX 1080ti. I don't think the video card has anything to do with it since I turned off the option to use it to process. I'll try anything, thank you.

[JJMack]

Can you post the details in the crash report.  Also when you were running that batch job you monitoring  Photoshop resource consumption using Windows task manager windows explore properties for your scratch dusks? Was Photoshop using all available RAM and a lot of scratch disk space?  Processing 1.2 million file may be more then your machine resources can handle if Photoshop resource management has problems.  When Photoshop crashed did your machine still have some free resources available. If Photoshop can process a file a second how long will it take Photoshop to process 1,200,000 files  two weeks.

[JoeJoseph]

Can you post the details in the crash report.

I cannot, unfortunately, because it doesn't give me any. It just tells me I can send the crash report or not, but there's no way to view details or anything. It appears to be a very generic message. It looks just like this. Except it's PS and not LR, obviously.

Also when you were running that batch job you monitoring  Photoshop resource consumption using Windows task manager windows explore properties for your scratch dusks? Was Photoshop using all available RAM and a lot of scratch disk space?

Looks like with the graphics card processing off Photoshop is running under capacity. The machine is at about 56% CPU and 36% RAM. I did notice while doing this that Windows Explorer was taking a significant CPU load because it's constantly refreshing the folder to sort by date, so I've backed out of the folder and it's now down to 31% and 37%. Almost all of that is now Photoshop.

When Photoshop crashed did your machine still have some free resources available. If Photoshop can process a file a second how long will it take Photoshop to process 1,200,000 files  two weeks.

To be clear, I'm doing batches of about 55k. I have a hard time believing it's system resources. It also doesn't explain exactly 7,077 images, when the images vary from 5 KB to 350 KB you'd expect it to at least be 7,075 sometimes. The exact file count every time is uncanny.

I'm wondering if there's some limit in Photoshop's internal storage of which images have already been processed? It's odd, because even if I only have 8k images left in the folder (I move them to a completed folder once they've been processed) it won't make it past 7,077.

I think I will be looking at alternatives, but I'd love to not have to learn a new tool if possible.. Two weeks would be fine if I didn't have to constantly restart the process. I'm already to 165k, having restarted the batch process over 20 times.

[JJMack]

There are two type of crash reports system crash reports the Windows and Mac OS create for application failures and Adobe Crash Reports for Photoshop failures the you can send to Adobe  which you can also look at report before sending it to Adobe by clicking on more info.  That Info can be captures and posted. Photoshop editing image is a slow process It is hard to believe you would want to process so many image files and Photoshop place your logo on not many have such volumes of images. Batches of 55,000 image files WOW. I would think it would take Photoshop over a day to batch process 55,000 images.  Photoshop does a lot  of sequential processing many thing are not multi threaded so if your processor have many cores most will be idle during the batch process..

[JoeJoseph]

Thanks for the response JJMack I really appreciate you and others being willing to respond and help out here.

I should also say it's not even the volume of 55k that's causing problems. As the images are processed I sort by date and move them out of the original folder and into a "completed" folder. So whether if there are 7,079 files or 55,000 in the folder when I start the batch process Photoshop will still process exactly 7,077 then crash.

Thanks for pointing out that link actually did something. I thought it was just going to an FAQ or something. Here is what it gives me on the slower of the 2 PCs I'm working on.

Error log on pastebin

It looks like it might be an access thing? I'm not clear on why it would do 7k files then stop. But I will try a batch running as administrator. If you have any other ideas in the meantime I'd love to hear.

[JJMack]

Adobe would need to look at that crash report the access violation was within Photoshop.exe itself not any other external module that Photoshop uses. So I do not get a clue to what Photoshop was actually doing when it bombs out.  Have you tried using any older versions of Photoshop. They have fewer issues than CC 2019 version 20.0.3.

<crash exception="EXCEPTION_ACCESS_VIOLATION" exceptionCode="0xc0000005" instruction="0x00007FF6F4806F19">

<binaryImage start="0x00007FF6EB310000" end="0x00007FF6F896C000" path="adobe\Photoshop.exe"/>

Again Photoshop design  is that of  an interactive image editor is not the approbate tool to batch thousands of image files.  While I have batch thousands I do not remember ever doing more the a few thousand  never 7K.  3K took 1.5 hours on my machine after I reprogrammed my Script to perform better.  Before I made my programming changes Photoshop was eating up all my machines resources RAM and Scratch Disk space.

What does you action look like.

[JoeJoseph]

Yeah as you predicted I get the same errors in admin mode. The action is very simple. 18 point arial font in bold, applied towards the middle bottom of the image. Then it saves and closes. The action I choose on the batch is "none." I have also tried deleting the save/close from the action and choosing "save and close" in the batch dialog, but this doesn't change anything.

I just tried adding a "purge > all" after each file. Excessive, I know, but when I Googled it everyone was talking about memory. I also adjusted the memory downward slightly. Once again, 7,077 files processed.

I'll try another version. I've tried to get other software but that's also been frustrating. About to pay for a bulk watermarking tool, which seems pretty stupid given how easy it seems.

[Stephen Marsh]

I've tried to get other software but that's also been frustrating. About to pay for a bulk watermarking tool, which seems pretty stupid given how easy it seems.

Why beat a dead horse? IMHO Photoshop is not the right tool for the volume of images that are being processed.

Why pay when there are free options (free apart from your time researching + learning curve)?

Have you tried any of the software from my post #2?

P.S. I presume that you have ample hard drive space to hold all of the processed images?

[JoeJoseph]

Have you tried any of the software from my post #2?

Yes, I investigated them all. GraphicsMagick was the most promising, but it will not run on my machine after install. I run the EXE and nothing happens. It's unclear whether the other 3 even do watermarking as they talk mostly about resizing, and the last two are command-line things I do not have time to learn. As for why, my time is valuable, and I have more money than time. It's easily worth the money, but my big concern is whether to trust the image processor from not-very-professional-looking bulk watermarking software. It looks like I'll have to retain the originals and deal with it later.

And yes, I have 800 GB free on the scratch drive (SSD), and I've tried switching to the other drive with 1.5TB free with no difference. The 1.2m images are only 88GB.

[Stephen Marsh]

I have not tested on your volume of images. The other suggested apps can do what you want and have far less overhead than Photoshop, but they too could fall over, I don’t know. The reason that I suggested them was that they do what you require without all of the baggage that comes with Photoshop.

Here is an example from XnConvert, which is a GUI so it will have some overhead over and above a CLI tool (XnConvert can output code for use in the CLI based nConvert, which may have better performance):

[Stephen Marsh · Accepted solution]

OK, I performed a very simple test. I did not close open apps or reboot the computer before the test – nor did I attempt to fine tune Photoshop for better batch performance, so perhaps the results could have potentially been better.

I created 500 png files at 1280 x 720 px in size, approx. 2mb each.

I setup an action in Photoshop to add a text layer, flatten and save as PNG. I then setup the batch and timed it. The run took around 28 minutes 40 seconds to complete the batch of 500 images. This was on a Mac with an SSD.

I then went into my Windows 10 virtual machine that is running on the same Mac, so there is the overhead of running two OS at the same time. I then used XnConvert to perform the same action of watermarking the same 500 source images with text. The run was approximately 3 minutes. I could have downloaded and installed XnConvert on the Mac, however it was already installed in Windows so ran the test there. I doubt that the deciding factor is the OS, it is the program. The right tool for the job, horses for courses and all that.

Photoshop: 28:40

XnConvert: 03:02

There is no guarantee that XnConvert will crash or stop processing after processing thousands of images, however on this simple test I am backing XnConvert over Photoshop for performance and reliability of batching many images with a simple addition of a text watermark. It would also be informative to run the command line nConvert, ImageMagick or GraphicsMagick tools as well as they don’t have the processing overhead of a GUI, however they usually have a longer/steeper learning curve for installation and use.

[JoeJoseph]

Okay XnConvert was surprisingly easy to use and very effective. I was initially thinking it was just for converting since that's most of what their website talks about. The files are larger coming out than I'd like, but I will just Smush them back down. Thank you for your advice. I was able to "convert" 55k files in a much shorter period of time with 0 issues.

[Michael254743562f1m]

Photoshop may not the right choice, but there are limitations to the other options, namely the complexity of certain batch processes we must perform are most seamlessly accomplished through recording in Photoshop.  I'm flirting with the idea of trying to do this in Python built into an executable I can simply click, but seriously, some of the actions I perform are far too complex.  And this at the heart of the issue with photoshop: the more complex the action (with more individual actions) the quicker Photoshop crashes.

It "seems" like a simple thing for Adobe to try and fix, imho, but they're tackling super advanced, and less foundational enhancements to the software as compared to the age-old batch function.  I am having the same problem, as is THOUSANDS of other people as evidenced by a quick Google Search.  

Adobe is a multi billion dollar company, and it defies logic that such a basic and ubiquitous problem for thousands of subscribers paying $50 a month isn't addressed by anyone.  I truly believe a team of 2 people can troubleshoot this and figure it out within a few days or weeks.

Rant over. 

I can tell you I am encountering the same exact problem and it must be some sort of memory/caching type of issue as was stated by others.  And its not the desktop memory size that's a problem either, its 100% software.  I know this because part of my daily routine involves batching thousands of files, and I troubleshooted the heck out of this over the past couple of years, and paid hundreds of dollars to double my memory and the problem still happened at the same exact number of files.  The amount of memory has zero bearing on the crashes.

I also tested it out on the files of different sizes, and it had no impact whether it was a mutiple megabyte file or a few hundred kb file. 

The only thing that has any bearing over the crashes is the number of individual actions within the action I execute.

~7500 files is the most I use, so I haven't tested this on more but here's what I found:

I have an action which has 2 simple actions within it, (select area and crop), and this never causes a crash, even on 7000+ files.  

I have another action which has approx 21 actions within it and Photoshop crashes after approx 3500 files.

And finally I have another action which has 31 actions within it and Photoshop crashes at exactly 700 files every time.  Curiously round number on this one, wouldn't you say?

Also important to note, I have broken up my daily routine to 3 separate machines running the same actions to circumvent the crashing of the "21 action" Action, but that "31 action" Action which I perform on weekends requires me to tediously break up the folders into 11 sub folders and run the action 11 times.  Tedious. Horrible.  The point here being its not a system issue as they are vastly different specs, even different operating systems, and the crashes happen at the same file number.

The problem is with the software. 100%.

[the_king_of_goats]

A few key things I've found that allow me to perform monster batches without crashing Photoshop:

1) Set "history states" to 1 before launching the script. (the default is 50 I think.) Depending on how your script is written, storing all of this info for the full set of batches could just be a huge memory/CPU drain. I've found, for whatever technical reason, setting this to the smallest possible value allows you to run much bigger batches much faster and more effectively.

2) Wipe Photoshop memory in between actions. For example I include these lines in my .jsx code:

              app.purge(PurgeTarget.ALLCACHES); // not sure all 4 are needed, just doing it to be thorough

              app.purge(PurgeTarget.CLIPBOARDCACHE);

              app.purge(PurgeTarget.HISTORYCACHES);

              app.purge(PurgeTarget.UNDOCACHES);

3) Reduce the Photoshop file sizes, where possible. Originally I was using these gigantic documents with tons of complex layers. I realized for my needs, the workflow was standardized, and I could really merge a ton of the layers to where it's now basically, background, and smart object atop it that gets replaced. Then I ALSO reduced the size, from the initial super high resolution, to the required one for the final output.

This functionally seems to keep the engine from running hot, because it's just constantly wiping the stored memory that would otherwise start to pile up, and it runs fresh and fast throughout the full duration.

Those are the main things I've found that allow me to run huge batches without crashing Photoshop or causing it to massively slow and drag as the program gets further down the line. 

[Stephen Marsh]

I would look into using a different tool for this “industrial volume” of images, such as ImageMagick or GraphicsMagick or XnConvert or nConvert etc.