Graphics PC using 64GB RAM and 3-M.2 drives with over 1.5TB of scratch disk space Out of Memory

[KMI-KWagenknight]

Our Graphics designer uses very large files, sometimes up to 15-20gb and multiple layers that are necessary due to our processing and use of the layer files but even on the smaller 2gb files she is running out of memory and I am wondering if it has to do with the files or her saved textures within PS.

Im in IT so I dont know much about PS so please help guide me through finding out why it crashes during saving and/or other processing of these files as she has lost lots of work hours because of this and I cant imagine how frustrating it must be for her!

All the settings seem fine and recommended, 70% of the 79gb (with cache) of DDR4 -3200 RAM is allocated for PS and then the scratch disks (m.2 SSDs) are set up with the C: drive being lowest and last to be used in the i7-8700k PC. Also the NVidia RTX 2080(updated drivers) is set to be used in the settings. No configuration changes that we have attempted to change are making a difference and at some point during her day it will crash. This seemed to start 3-4 months ago and we have been limping along ever since.

Any help would be appreciated so let me know if you need more info!

Thank you in advance! 

[kevin stohlmeyer]

Is the file 15-20 GB compressed when saved? What file size is Photoshop showing when it is opened?

How much HD space is allocated in the Scratch Disk preferences?

[KMI-KWagenknight]

Thanks for responding! After talking with Sandi, she tells me that the file (tiff) is say 5GB compressed on the drive then when she is working with that file, it is around 22GB.

We have 3 scratch disks set up which are Samsung Pro 970/980 500GB with her C:(OS Drive) being 3rd on the list with around 500gb of free space on the first 2 scratchdisk's before even getting to the 3rd C:Drive.

[D Fosse]

with around 500gb of free space on the first 2 scratchdisk's before even getting to the 3rd C:Drive.

By @KMI-KWagenknight

Not enough. With those file sizes, scratch can easily hit 1TB.

Most people underestimate the disk space needed for scratch. This gets really big really fast. Every history state potentially adds the full uncompressed file size, plus overhead. If there are smart objects, it can explode. And Photoshop reserves scratch space based on anticipated need. It doesn't grow as you go.

At least reduce the number of history states to 10-20, down from the default 50.

Personally, I'm never comfortable with anything less than 1TB scratch space (I actually have more).

[Earth Oliver]

i'm guessing that auto-save is what's causing the issue. When working on large files, the auto-save recovery will cause major problems. 

Either that and/or if they're using embedded SO's, that will also bring down a system... since for whatever reason, eSOs are still saving to the startup instead of the scratch. I've had a FR in for years to change this, but we're still living in the 90s with eSO's.

[D Fosse]

Yes, but one instance of the full file size is still small compared to 50 of them, which is what history can contain. It's true that smart objects are stored in the system temp directory, not the designated scratch disk, but that only becomes a problem is the system drive is already close to full (which would be a general problem for a whole lot of reasons beside Photoshop).

EDIT: on the other hand, if the file contains a lot of smart objects, then yes, that could be a problem.

Either way, it's all down to disk space. Keep your disks tidy and clean out the junk at intervals. A standard configuration of operating system plus applications should normally not take up more than 80-120 GB.

[Lumigraphics]

What troubleshooting have you done? Have you tried a new clean user account on the same computer?

[Derek Cross]

Are you certain you need to produce such large files? Often people producing, for example, posters create them at a much higher resolution than is needed.

[Earth Oliver]

That's really not very helpful.

[Derek Cross]

Why?

[KMI-KWagenknight]

Yes, some of the dimensions of each layer/file can be for a press plate that is 2300mm x 6500mm. She starts out creating a certain size repeat, maybe 600mm x 900mm and then stitches it together for the final size where it can go to production and can be printed. Due to certain parameters with the print head it needs to be a certain dpi so it is exactly the size it needs to be unfortunately. She has been doing this for 15+ years here but this is the first time we have run into this exact problem when we think the resources arent really the main issue and wasnt for so long so we are kind of stumped.

[D Fosse]

What Derek is getting at is that a lot of people think you need 300 ppi at any print size. And so you end up with these mammoth file sizes. But you don't - the bigger it gets, the lower the ppi requirement, because it will be seen from farther away. A large wall-sized banner can be 15 ppi and look splendidly crisp and sharp.

The rule of thumb (at least for a photograph) is that any good file will work at any size, whether magazine spread or billboard. The file is exactly the same, no bigger.

[PECourtejoie]

Hej, Dag, it seems to me that, according to the description of the job, in this case, it might be for wallpaper or fabric designs, which are seen from a closer distance than banners.

[D Fosse]

Yes, you're probably right. I've assumed the whole time that the file size was actually necessary, and outlined the hardware requirements for that.

Just wanted to clarify Derek's point, which is valid even if not applicable here.

[KMI-KWagenknight]

Yes I get what you are saying but this has been tested and refined to its necessary parameters as its not for visual but engraving/etching purposes and the way the ink lays and spreads over time etc has to be a certain way or it wont get the desired engraving/etching we need when we are done. We have gone to the minimum requirements due to creating, handling as well as storing these files and everything else so we have looked at making them as small as possible and what we are doing now is that so we have to work with it like we are unfortunately.

[Earth Oliver]

She needs to turn off the auto-save option in the preferences and if she's using embedded smart objects, she'll need to switch to using Linked SOs.

[D Fosse]

That shouldn't be necessary if you have enough disk space. That's what we keep returning to.

When working with massive files like this, you really need to make sure there's enough machine resources to handle it. You can't blame Photoshop.

[PECourtejoie]

Hi, I second the remarks about the scratch size.

I'm surprised about the 5GB TIFFs, I thought they were limited to 4GB. I guess you need tiff to import in other software. PSB is another option, but has only a bearing at save time, AFAIK.

You state 64GB of RAM, then 79 with cache... Is it a cache on the SSD? Did you ever try to disable it? There could be competing memory management.

Did you already peruse this document? 
https://helpx.adobe.com/photoshop/kb/optimize-photoshop-cc-performance.html

Having tons of loaded presets can also eat in the memory...

A good thing to avoid memory fragmentation is to use CTRL+J instead of CTRL+C/V as it used PS's memory and not the operating system's.

[KMI-KWagenknight]

You are correct, that is my mistake in translation. Anything over a certain size is in a .psb file format but most of them we use are tif.

Thats a good point about the page file cache, I will turn that off and see if that helps.

I will ask her about the presets but I know she said there are a ton of brushes and other things she has saved, is that what you mean by presets as that was a worry of hers when we started troubleshooting?

I will also let her know about CTRL + J which Im not sure she uses but will pass it along, thank you!

[PECourtejoie]

@KMI-KWagenknight when did her problem happen?

Which exact version of Photoshop is she using?

There's been some changes in the foundations of Photoshop that might cause some issues, but this very problem seems different than most others reported, so it might not be related...

Hence users are told to verify if checking "Deactivate Native Canvas" in Edit/Preferences/Technology Preview made a difference, it might be worth cheching.

I think that you are part of the IT staff, so I do not think that telling you to make only one change at a time, restarting is warranted, but might be a good reminder for others.

Other troubleshooting documents you might have found: https://helpx.adobe.com/ca/photoshop/kb/basic-troubleshooting.html 

https://helpx.adobe.com/photoshop/kb/photoshop-slow-lags.html