Random Weirdness (invalid file, out of memory etc)

[Guest]

Hello,

I have a rather lengthy ExtendScript that does a bunch of post-processing on custom map tiles for various map rendering services (Google Maps, Bing, Open Street etc). We basically take tiles that are exported from our ArcGIS suite, and pretty them up by adding drop shadows, text halos etc to certain features, and then layering them on top of each other and saving them.

With the standard map tile being 256x256 pixels, and the amount of tiles growing exponentially based on the zoom level we are working with, there ends up being A LOT of tiles... For example, zoom level 8 has 256 folders with 256 tiles in them (2^8x2^8) = 65536 tiles.

The problem I'm running into, is after a random point in time, the script stops functioning. Sometimes Photoshop will claim that it is out of memory, other times it just throws a JavaScript error claiming that it cannot open a tile .PNG file (and all subsequent tile .PNGs in the loop), even though if I stop the script, restart Photoshop, and start the script from where it left off, it continues as normal.

Obviously, using JavaScript, the level of garbage collection and memory management control is pretty limited from a programming standpoint. I can say that when I'm done processing a tile, I do make sure that all documents I opened in regards to processing that tile are closed, and caching operations are cleared:

           //Cleanup

            while(app.documents.length > 0) {

                    app.documents[0].close(SaveOptions.DONOTSAVECHANGES);

            }

            app.purge(PurgeTarget.ALLCACHES);

When I do see these errors, there aren't a giant amount of documents open in the Photoshop window. Everything appears as it should. Process Explorer shows Photoshop using anywhere between 600MB-1800MB of memory during these times. So what I'm saying is, something weird is happening, and I don't know what. I'm hoping some people can shed some light on this for me, as it is a pain babysitting and restarting this script every few hours.

Thanks!

[xbytor2]

You may want to ad a $.gc(); to force garbage collection when you do the purge. That may help some.

If it doesn't, there are ways of automating the babysitting process.  I implemented a couple of solutions for customers that had long running script in PS (2-3 days) so it can be done but it's never particularly easy.

[Guest]

Funny you mention $.gc(), as I read about it just after I posted my initial post. Didn't seem to do anything for me. It still stopped opening perfectly valid tiles about 4 hours after I left the office.

I'd love to hear your solution of automating the babysitting. I was hoping photoshop.quit() after a few iterations would close Photoshop and reopen it again, but that didn't seem to work. At this point I'm willing to try anything if I can't get an actual solution to Photoshop bombing out like it has been.

[xbytor2]

1) Write a shell script that calls PS with your script as an argument

A bash script would look like this:

while  true; do

    photoshop ~/Desktop/test.jsx

done

Also, you would need to specify the full path to the PS app.

2) In your script, do however many iterations you're comfortable with. If it's failing after 4 hours, I'd do it in batches that take no longer than 2 hours.

3) When a set of iterations is done, do photoshop.quit() to end the app.

4) When you run out of stuff to process do something like alert("Done") so the app stops waiting for input and preventing a restart.

5) When you see the alert pop up, kill the shell script and press "OK".

This is about as simple as it gets. There are ways of doing this entirely in jsx but they're a bit more complicated.

[Tom Ruark]

In Task Manager give me the Handles, GDI Objects and USER Objects count for each. I am seeing USER Objects go to 10000 which will cause Ps to behave badly and require a restart. I hope it is just my environment!

[Guest]

Handles - 16,987

USER Objects - 1,886

GDI Objects - 552

Memory:

Private Bytes: 1,384,608 K

Working Set: 600,392 K

Over the 6 hours it ran, Handles and Private Bytes slowly crept up to where they are when it stopped (could no longer open any tiles).

[Guest]

Another instance:

Handles - 14,400

GDI Objects - 545

USER Objects - 1,879

Memory:

Private Bytes - 870,800K

Working Set - 804,356K

[Tom Ruark]

Send me your System Info output or post here. Can you try to shut off all of your third party plug-ins and try again?

[Guest]

Not using any plug-ins that I'm aware of... Basically got a copy of CS5 to install so I could write the script (I'm just a developer). This time I got a "Could not save a copy of ... because there is not enough memory (RAM)" exception:

Handles - 16,442

USER Objects - 1,892

GDI Objects - 561

Memory:

Private Bytes - 1,582,040K

Working Set - 625,820K

My System Information:

Adobe Photoshop Version: 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch]) x64

Operating System: Windows NT

Version: 6.2

System architecture: Intel CPU Family:6, Model:14, Stepping:5 with MMX, SSE Integer, SSE FP, SSE2, SSE3, SSE4.1, SSE4.2

Physical processor count: 8

Processor speed: 2793 MHz

Built-in memory: 12247 MB

Free memory: 5769 MB

Memory available to Photoshop: 10870 MB

Memory used by Photoshop: 60 %

Image tile size: 128K

Image cache levels: 4

OpenGL Drawing: Enabled.

OpenGL Drawing Mode: Basic

OpenGL Allow Normal Mode: True.

OpenGL Allow Advanced Mode: False.

OpenGL Crash File: Not Detected.

OpenGL Allow Old GPUs: Not Detected.

Video Card Vendor: NVIDIA Corporation

Video Card Renderer: GeForce 310/PCIe/SSE2

Display: 2

Display Bounds:=  top: 0, left: 1680, bottom: 1050, right: 3360

Display: 1

Display Bounds:=  top: 0, left: 0, bottom: 1050, right: 1680

Video Card Number: 1

Video Card: NVIDIA GeForce 310

Driver Version: 9.18.13.4181

Driver Date: 20150817000000.000000-000

Video Card Driver: nvd3dumx.dll,nvwgf2umx.dll,nvwgf2umx.dll,nvd3dum,nvwgf2um,nvwgf2um

Video Mode: 1680 x 1050 x 4294967296 colors

Video Card Caption: NVIDIA GeForce 310

Video Card Memory: 512 MB

Serial number: 95458123392184773369

Application folder: C:\Program Files\Adobe\Adobe Photoshop CS5.1 (64 Bit)\

Temporary file path: C:\Users\CGSOBO~1.SHO\AppData\Local\Temp\

Photoshop scratch has async I/O enabled

Scratch volume(s):

  Startup, 237.4G, 100.7G free

Primary Plug-ins folder: C:\Program Files\Adobe\Adobe Photoshop CS5.1 (64 Bit)\Plug-ins\

Additional Plug-ins folder: not set

Installed components:

   A3DLIBS.dll   A3DLIB Dynamic Link Library   9.2.0.112  

   ACE.dll   ACE 2010/12/13-23:37:10   64.449933   64.449933

   adbeape.dll   Adobe APE 2011/01/17-12:03:36   64.452786   64.452786

   AdobeLinguistic.dll   Adobe Linguisitc Library   5.0.0  

   AdobeOwl.dll   Adobe Owl 2010/06/03-13:43:23   3.0.93   61.433187

   AdobeOwlCanvas.dll   Adobe Owl Canvas   3.0.68   61.2954

   AdobePDFL.dll   PDFL 2010/12/13-23:37:10   64.341419   64.341419

   AdobePIP.dll   Adobe Product Improvement Program   5.5.0.1265  

   AdobeXMP.dll   Adobe XMP Core   5.0   64.140949

   AdobeXMPFiles.dll   Adobe XMP Files   5.0   64.140949

   AdobeXMPScript.dll   Adobe XMP Script   5.0   64.140949

   adobe_caps.dll   Adobe CAPS   4,0,42,0  

   adobe_OOBE_Launcher.dll   Adobe OOBE Launcher   2.0.0.36 (BuildVersion: 2.0; BuildDate: Mon Jan 24 2011 21:49:00)   1.000000

   AFlame.dll   AFlame 2011/01/10-23:33:35   64.444140   64.444140

   AFlamingo.dll   AFlamingo 2011/01/10-23:33:35   64.436825   64.436825

   AGM.dll   AGM 2010/12/13-23:37:10   64.449933   64.449933

   ahclient.dll    AdobeHelp Dynamic Link Library   1,6,0,20  

   aif_core.dll   AIF   2.0   53.422628

   aif_ogl.dll   AIF   2.0   53.422628

   amtlib.dll   AMTLib (64 Bit)   4.0.0.21 (BuildVersion: 4.0; BuildDate: Mon Jan 24 2011 21:49:00)   1.000000

   amtservices.dll   AMTServices (64 Bit)   4.0.0.21 (BuildVersion: 4.0; BuildDate: Mon Jan 24 2011 21:49:00)   1.000000

   ARE.dll   ARE 2010/12/13-23:37:10   64.449933   64.449933

   asneu.dll    AsnEndUser Dynamic Link Library   1, 7, 0, 1  

   AXE8SharedExpat.dll   AXE8SharedExpat 2011/01/10-23:33:35   64.436825   64.436825

   AXEDOMCore.dll   AXEDOMCore 2011/01/10-23:33:35   64.436825   64.436825

   Bib.dll   BIB 2010/12/13-23:37:10   64.449933   64.449933

   BIBUtils.dll   BIBUtils 2010/12/13-23:37:10   64.449933   64.449933

   boost_threads.dll   DVA Product   5.0.0  

   cg.dll   NVIDIA Cg Runtime   2.0.0015  

   cgGL.dll   NVIDIA Cg Runtime   2.0.0015  

   CoolType.dll   CoolType 2010/12/13-23:37:10   64.449933   64.449933

   data_flow.dll   AIF   2.0   53.422628

   dvaadameve.dll   DVA Product   5.0.0  

   dvacore.dll   DVA Product   5.0.0  

   dvaui.dll   DVA Product   5.0.0  

   ExtendScript.dll   ExtendScript 2011/01/17-17:14:10   61.452840   61.452840

   FileInfo.dll   Adobe XMP FileInfo   5.0   64.140949

   icucnv36.dll   International Components for Unicode 2009/06/17-13:21:03    Build gtlib_main.9896  

   icudt36.dll   International Components for Unicode 2009/06/17-13:21:03    Build gtlib_main.9896  

   image_flow.dll   AIF   2.0   53.422628

   image_runtime.dll   AIF   2.0   53.422628

   JP2KLib.dll   JP2KLib 2010/12/13-23:37:10   64.181312   64.181312

   libifcoremd.dll   Intel(r) Visual Fortran Compiler   10.0 (Update A)  

   libmmd.dll   Intel(r) C Compiler, Intel(r) C++ Compiler, Intel(r) Fortran Compiler   10.0  

   LogSession.dll   LogSession   2.1.2.1263  

   MPS.dll   MPS 2010/12/13-23:37:10   64.450375   64.450375

   msvcm80.dll   Microsoft® Visual Studio® 2005   8.00.50727.9185  

   msvcm90.dll   Microsoft® Visual Studio® 2008   9.00.30729.9158  

   msvcp80.dll   Microsoft® Visual Studio® 2005   8.00.50727.9185  

   msvcp90.dll   Microsoft® Visual Studio® 2008   9.00.30729.9158  

   msvcr80.dll   Microsoft® Visual Studio® 2005   8.00.50727.9185  

   msvcr90.dll   Microsoft® Visual Studio® 2008   9.00.30729.9158  

   pdfsettings.dll   Adobe PDFSettings   1.04  

   Photoshop.dll   Adobe Photoshop CS5.1   CS5.1  

   Plugin.dll   Adobe Photoshop CS5   CS5  

   PlugPlug.dll   Adobe(R) CSXS PlugPlug Standard Dll (64 bit)   2.5.0.232  

   PSArt.dll   Adobe Photoshop CS5.1   CS5.1  

   PSViews.dll   Adobe Photoshop CS5.1   CS5.1  

   SCCore.dll   ScCore 2011/01/17-17:14:10   61.452840   61.452840

   tbb.dll   Threading Building Blocks   2, 1, 2009, 0201  

   TfFontMgr.dll   FontMgr   9.3.0.113  

   TfKernel.dll   Kernel   9.3.0.113  

   TFKGEOM.dll   Kernel Geom   9.3.0.113  

   TFUGEOM.dll   Adobe, UGeom©   9.3.0.113  

   updaternotifications.dll   Adobe Updater Notifications Library   2.0.0.15 (BuildVersion: 1.0; BuildDate: BUILDDATETIME)   2.0.0.15

   WRServices.dll   WRServices Thursday January 21 2010 12:13:3   Build 0.11423   0.11423

   wu3d.dll   U3D Writer   9.3.0.113  

Installed plug-ins:

   3D Studio 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Accented Edges 12.0

   ADM 3.11x01

   Angled Strokes 12.0

   Average 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Bas Relief 12.0

   BMP 12.0.2

   Camera Raw 6.3

   Chalk & Charcoal 12.0

   Charcoal 12.0

   Chrome 12.0

   Cineon 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Clouds 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Collada 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Color Halftone 12.0.2

   Colored Pencil 12.0

   CompuServe GIF 12.0.2

   Conté Crayon 12.0

   Craquelure 12.0

   Crop and Straighten Photos 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Crop and Straighten Photos Filter 12.0.2

   Crosshatch 12.0

   Crystallize 12.0.2

   Cutout 12.0

   Dark Strokes 12.0

   De-Interlace 12.0.2

   Dicom 12.0

   Difference Clouds 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Diffuse Glow 12.0

   Displace 12.0.2

   Dry Brush 12.0

   Eazel Acquire 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Embed Watermark 4.0

   Entropy 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Extrude 12.0.2

   FastCore Routines 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Fibers 12.0.2

   Film Grain 12.0

   Filter Gallery 12.0

   Fresco 12.0

   Glass 12.0

   Glowing Edges 12.0

   Google Earth 4 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Grain 12.0

   Graphic Pen 12.0

   Halftone Pattern 12.0

   HDRMergeUI 12.0

   IFF Format 12.0.2

   Ink Outlines 12.0

   JPEG 2000 2.0

   Kurtosis 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Lens Blur 12.0

   Lens Correction 12.0.2

   Lens Flare 12.0.2

   Lighting Effects 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Liquify 12.0.1

   Matlab Operation 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Maximum 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Mean 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Measurement Core 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Median 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Mezzotint 12.0.2

   Minimum 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   MMXCore Routines 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Mosaic Tiles 12.0

   Multiprocessor Support 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Neon Glow 12.0

   Note Paper 12.0

   NTSC Colors 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Ocean Ripple 12.0

   OpenEXR 12.0.2

   Paint Daubs 12.0

   Palette Knife 12.0

   Patchwork 12.0

   Paths to Illustrator 12.0.2

   PCX 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Photocopy 12.0

   Photoshop 3D Engine 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Picture Package Filter 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Pinch 12.0.2

   Pixar 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Plaster 12.0

   Plastic Wrap 12.0

   PNG 12.0.2

   Pointillize 12.0.2

   Polar Coordinates 12.0.2

   Portable Bit Map 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Poster Edges 12.0

   Radial Blur 12.0.2

   Radiance 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Range 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Read Watermark 4.0

   Reticulation 12.0

   Ripple 12.0.2

   Rough Pastels 12.0

   Save for Web & Devices 12.0

   ScriptingSupport 12.1

   Shear 12.0.2

   Skewness 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Smart Blur 12.0.2

   Smudge Stick 12.0

   Solarize 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Spatter 12.0

   Spherize 12.0.2

   Sponge 12.0

   Sprayed Strokes 12.0

   Stained Glass 12.0

   Stamp 12.0

   Standard Deviation 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Sumi-e 12.0

   Summation 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Targa 12.0.2

   Texturizer 12.0

   Tiles 12.0.2

   Torn Edges 12.0

   Twirl 12.0.2

   U3D 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Underpainting 12.0

   Vanishing Point 12.0

   Variance 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Variations 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Water Paper 12.0

   Watercolor 12.0

   Wave 12.0.2

   Wavefront|OBJ 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   WIA Support 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   Wind 12.0.2

   Wireless Bitmap 12.1 (12.1x20110328 [20110328.r.145 2011/03/28:10:30:00 cutoff; r branch])

   ZigZag 12.0.2

Plug-ins that failed to load: NONE

Flash:

   Mini Bridge

   Flash

   Access CS Live

   Flash

   Kuler

   CS Review

Installed TWAIN devices: NONE

[Guest]

Just a FYI, over the holiday weekend I tried removing all extraneous plug-ins (ones installed by default for CS5 - I haven't installed anything else). To make my script work, I needed the following folders in the Plug-ins folder:

Extensions/*.*

Filters/*.*

Not sure which files in those folders would need to be removed to maintain functionality. However, by cutting it down to there, it seems to process about 20 folders with ~256 tiles in each instead of 10 folders now... Still eventually has the same issue though, either stops opening .PNGs, or says it is out of RAM.

[Tom Ruark]

Version 12 I see. I'll have to look to see if this is a known issue or not. You can hold down the shift key while launching to remove all the "external" plug-ins. I see you have two "flash" plugins listed in your system info. I would also try to remove all of those during launch. In preferences, under plug-ins, there is an option to "load extension panels" try turning that off.

[Tom Ruark]

What is the USER object count for Photoshop when you hit this problem?

[Guest]

Thanks for the reply. How do I check the USER object count?

[Guest]

Assume you mean USER Handles in Task Manager/Process explorer? Will report back when it fails.