LR Classic - changing the temp (scratch) drive? It fills up my start-up drive completely with some operations

[FrankWPhoto]

Recently, I had been stitching a lot of 16MP photos into HDR and the resulting HDR photos into panoramas (by using LR Classic Merge in the background - ctrl-shift-H and M, since Adobe Camera Raw doesn't allow background merges, even though without any BATCH processing option, this is still a major PITA).

Since there was a lot of the merges queued, my start-up disk almost completely filled up even though I had almost 30GB free on the SSD. I have dedicated SSDs in RAID just for scratch files and working with video, these are much faster than the start-up SSD (256GB) and had ample space. Is there any way to tell LR to use only those for scratch files? Mind you, these were not previews getting generated, the space immediately got reclaimed when I quit LR, these were just temp files or maybe system memory (got 32GB which is not so big nowadays I admit) getting swapped to disk.

Still, inside Photoshop I can at least assign scratch disk to specific drives, which does work when merging big panoramas manually (about the most memory intensive operation I do there), although that does not seem to work for Camera Raw (which seems to rely only on system memory and results in swapping to disk in some operations).

So, is there any way to tell LR to use specific drives for any temporary files?

[Just Shoot Me]

You can set the ACR cache location from the Performance tab of the LR Preferences.

Also purge the video cache. No matter what it is set to it just keeps filling up.

[FrankWPhoto]

Hi, thanks. I did that already, the ACR Cache location is set to my temp ssd raid array right from the start, did no difference. Nevertheless, that seems to be only the CACHE for storing PREVIEWS data, not an actual TEMP location. When doing a memory intensive operation like merging a lot of photos into a panorama, it still seems to eat just main system memory and that either gets swapped to (startup) disk by the OS, or LR itself stores the temp files in TMP or somewhere on the startup disk, disregarding the CACHE setting. Maybe it's temp files that get created right next to the catalog (which is currently stored on my startup disk), maybe it's just memory swapping but even if really so, it would be IMHO bad programming to rely on OS memory swapping only when potentially dealing with panoramas the size of hundreds of megapixels, or not offer the user a different location for the scratch files apart from system tmp, like Photoshop itself does.

I might try it again, do a lsof command on Lightroom, just to see what file operations the LR process does (lsof is an unix command showing all files opened by a process, supported on Mac OS natively).

Sorry, should have stated the system specs first - Mac OS 10.12.6.

[Just Shoot Me]

What exactly do you mean by Temp Location?

There are only 4 places LR stores info to.

1) The catalog file itself and that can be on any drive and in any folder you like.

2) The image file Previews (CatalogName Previews.lrdata) folder and that is stored (and has to be stored) in the same folder as the Catalog File.

3) The Smart Previews folder. If you use smart previews. And that also is stored in the same folder as the catalog.

4) The Video Cache which is stored someplace in your UserName folder and that can't be moved

All the others, other than the video cache, can be moved off the system drive, Windows the C: drive, Mac the Macintosh HD.

If somehow your Windows Temp folder, usually C:\Windows\Temp, is filling up then you can move that to some other drive using the System "Environment Variables". you have to create a Temp folder on some other drive first then point Windows to use that temp folder.

[FrankWPhoto]

Just Shoot Me, unfortunately, what you wrote above is somewhat incorrect*. You forgot the scratch file location, which is number 5. Which gets created when PS/LR/ACR needs more memory for big operations.

By the Temp location I mean the location temporary files (or scratch files) get written to when either Photoshop or Lightroom is running and that get deleted after it quits. In Photoshop, you can set some of the location in Preferences/Performance/Scratch Disks. I always unselect the startup disk and select a dedicated SSD raid array for the scratch disk (but that setting is not honoured by Adobe Camera Raw anyway). No way to do even that in Lightroom.

*: I just did a lsof on Lighroom, and it writes a lot of temporary data to both user cache locations, and system TMPDIR locations. See this old discussion here on the forums: Large Temp File in hd:/private/var/.../Cleanup at Startup/ folder, bloating my backup

The file in the system $TMPDIR location ($TMPDIR/Cleanup at Startup/cr_sdk_[numbers].tmp) seems to be the culprit, according to lsof. After running multiple panorama merges in the queue in LR, it's grown currently to 25GB in size. Even though the merges are concurrent, not parallel, that seems to be pretty bad scratch memory cleanup (not uncommon for PS, where quitting and restarting after some big operations was often the only way to restore performance).

This is the exact scratchdisk location I would like to choose location of. Putting it in system $TMPDIR, while it can grow to hundreds of gigabytes when working with really big panoramas or similar operations, and not letting the user choose, is really strange. Especially, since the only time it gets culled is when quitting LR. There is no scratch culling while LR is running at all. But again, that had been the way with Photoshop for years...

I can definitely disregard the notion that it was just the OS doing memory swapping when it ran out of memory. Even with LR using most available RAM, the system swap is only ±4GB as confirmed in Activity monitor and by sizing the swapfiles in /private/var/vm

Definitely, Lightroom Classic is putting up to 25GB of scratch files somewhere in my $TMPDIR, and deleting them not after ending one operation when the data is no longer needed, but only after quitting. I can't even access the scratch without loging as ROOT, that well hidden is it. Only using lsof which looks a bit deeper can I see it and its size. Not only that, if it indeed is .../Cleanup at Startup/cr_sdk_[...].tmp, the Cleanup at Startup folder is a bit strangely named - it used to be name of a TMP folder for applications back in OS 9...

Unfortunately, exact same situation happens with Adobe Camera Raw, invoked from inside Photoshop, and merging panoramas or HDRs. Similar scratch file gets created in the same location, irrespective of your preferences for scratch disks in PS/ACR.

tl;dr:

Both Adobe Camera Raw and Lightroom Classic on Mac OS (and, from web searches also on Windows) can put a massive scratch file on your startup disk, with intensive operations like panorama merging up to gigabytes or tens of gigabytes in size, that just grows until it consumes all the available space or you quit them. Without any way to choose a better location for the scratch file, like a dedicated, raided, super-speedy SSD or another volume. The only solution is to buy a bigger startup disk... Even if I had 2x1TB m2 SSDs in RAID, or PCI-Express SSD cards with 1TB+/s write speeds, dedicated to just such scratch files like in video editing, it would still put it on the startup disk. That also means that with big operations (like big panorama merging or multiple high-res HDR merging), once Lightroom runs out of free RAM (and that's pretty easy on anything but highest end systems with 128GB RAM), its speed is limited by the speed of your startup disk. And also limited by all the other file operations happening on the startup disk, by the OS, by other programs, whatever. Even if it could use a super-speedy dedicated scratch drive just for itself. Photoshop Adobe Camera Raw behaves the same, even when you set Scratch Disk location elsewhere in PS preferences.

It seems to me like a bug or oversight, especially since neither LR nor PS nor ACR seem to honor the user set Camera Raw Cache / Scratch Disk location for those operations.

[Just Shoot Me]

LR does not have a Scratch file.

ACR does have a Cache folder it can use and both ACR and the RAW editor in LR use the same folder.

Photoshop does use a Scratch drive that can be changed in the PS preferences. But that does nothing for LR or ACR.

If you talking about the Windows Pagefile.sys file that is controlled by the OS.

If you are on a Mac then it is a function of OS X creating and deleting temp files similar to the Windows Pagefile.sys. But as far as I know, as I haven't used a Mac in 5+ years, you can't change the location of that file/folder/whatever in OS X like you can in Windows.

Your # 5

That is a function of the OS and not LR. The OS chooses what gets put in RAM or into a Virtual RAM environment on the HDD if no more RAM is available.

I don't see that on my Windows system. I did 3 HDR merges and the Win Pagefile stayed the exact same size. Physical RAM usage got to 48% after the 3 merges then dropped back down to the normal 16%

I'm not seeing this on my Windows system.

Watching Disk activity with Resource Monitor while doing a merge I see not real increase in overall disk usage.

[FrankWPhoto]

Sorry, but you must have probably misunderstood me, or might be mistaken.

Quotes in italics:

"LR does not have a Scratch file.

ACR does have a Cache folder it can use and both ACR and the RAW editor in LR use the same folder."

LR does not have a scratch file? So could you please tell me, what is the 20GB+ file, that is clearly linked to LR by lsof (that's an unix command that shows all file system access, ports, pipes and nearly everything a program accesses), that gets created in system TMPDIR as, on Mac OS, var/.../T/Cleanup At Startup/cr_sdk_[numbers].tmp (and from few web searches, same in Windows TEMPDIR)? And that gets cleaned up when I quit LR or PS ACR? And whose size clearly depends on the number and complexity of the merges? Sorry, but that seems to me like an Adobe LR scratch file...(especially since it

ACR (and LR ACR) Cache folder is, plainly, only used for the caching of previews, not for scratch files.

"That is a function of the OS and not LR. The OS chooses what gets put in RAM or into a Virtual RAM environment on the HDD if no more RAM is available.

Sure, the system swap is managed by the OS, be it on Windows or Mac. But I specifically confirmed (in two different ways), that the system swap stayed just around 4GB, while my startup disk free space just dropped by 25GB when LR was running and merging.

"If you are on a Mac then it is a function of OS X creating and deleting temp files similar to the Windows Pagefile.sys."

Yes, and these are specifically stored in /private/var/vm folder, called swapfile and serve the same purpose as pagefile on Windows. That's not where the used space went to, as I wrote. It went to a completely different file in TMPDIR folder on the startup drive, linked to Lightroom/Photoshop, not to swap.

"I'm not seeing this on my Windows system.

Watching Disk activity with Resource Monitor while doing a merge I see not real increase in overall disk usage."

You must exceed your RAM for LR/ACR to start writing the scratch file. Depending on your specs, it might not have been visible with just a small merge or few files HDR merge.

I just did a quick test, merging of 23 HDR 16megapixel photos in ACR run from PS (same issue as LR). During the preview creation, no space was used. Once I hit Merge, my disk space started getting down pretty quickly, with Photoshop CC process started showing heavy writes and reads in the Activity Monitor (OS X system app that shows memory use and disk access, just like Resource Monitor). lsof showed the corresponding scratch file created. Merging 23 HDR files into one very big panorama obviously requires a lot of space, the 20GB (plus some RAM) or so used seems pretty apropriate.

If it was OS that was writing some obscure file to undocumented location, Activity Monitor would show Kernel Task or something like that as the process responsible for the writes. Nope, the process writing and reading gigabytes of data to disk was Photoshop CC (and Lightroom CC in the previous instance).

See the screenshot showing writes and reads:

At least on the Mac, that's the way it is.

[Just Shoot Me]

Then you need more RAM and a bigger SSD as your OS system drive.

[FrankWPhoto]

Very helpful. Thanks.

Obviosly, that's not the problem, is it? I could get 128GB RAM and 1TB start-up drive instead of 32GB and 256GB, but the issue would be the same, just visible with somewhat bigger files or more files to merge into a panorama. The problem is that both Photoshop ACR and Lightroom CC Classic cannot utilise already existing resources on my computer - a super speedy raided 1TB SSD array - for a scratch disk, defaulting to a system disk for scratch space, and not telling the user that their choices for cache and scratch disk are completely ignored in that particular case, and that panorama merging is limited only to system start-up disk.

[Pro360Media]

Hey FrankWPhoto,

Did you find a solution to this issue?

[FrankWPhoto]

Unfortunately not. The Cleanup At Startup subfolder which seems to be the location of the invisible temp files is nested deeply in /private/var/folders/..., under a numbered folder whose name might change (not sure about that, if the number is derived from UUID or whatever else), making it somewhat hard to hardlink it without possibly breaking system functionality (by hardlinking the whole /private/var folder or something like that).

[Ryan Smith Photography]

Comment to Just Shoot Me: I found this thread because I'm having issues with over 100gb of space being used when merging hdr panoramas. Then I saw this unhelpful jerk, he is wrong then instead of admitting it says just buy a bigger SSD.  You can't do that easily in a Mac.  If you have nothing helpful to say, maybe don't say anything at all.

[pacoquinha]

I jsut started having the same issue after stitching some photos in LR for the first time 😞
Searching for LR scratch disk brought me here (too)
Maybe the move tmp trick will work for me on windows 10...

Adobe should utilise resources in a better way...

Thanks guys!

[pacoquinha]

Update to my previous comment:

Microsoft Image Composite Editor works well and is easier. It lets you set the tmp directory to wherever you want. It's also a free program from Microsoft

[Hal P Anderson]

You don't say whether you're on Mac or Windows. On Windows, at least, you can tell the system to keep your temporary files on a different disk. Type "move temp folder" into Cortana and go to the web page she suggests. There's probably a way to do it on Mac as well, but a quick view shows it's perhaps more complicated. Search the web for "move temp folder on Mac".

Hal

[FrankWPhoto]

Hi, thanks. Both you and JSM suggested moving the tmp folder. Alas, that's not so easy on Mac OS, and requires some low level tinkering. I am not even sure Adobe programs do honour the $TMPDIR variable or are just hardwired for the location. I might try that later, with a clean install.

Normally, for any application, it would be just fine to put any temp files in the system TMPDIR. But since ACR/LR scratch can grow up to tens of gigabytes very easily, and they both do seem to offer the user the option of selecting a "cache" or "scratch disk" location, it would make some sense to put the actual scratch file in that location as well, since the way it is now, it can be confusing what location is what. On the other hand, TMPDIR gets cleaned by the system on reboot (or the scratch file gets deleted by ACR/LR upon normal quit or upon ACR/LR restart if they crashed). Still, I would prefer if there was some clear documentation on that - could not find any at all, and had to parse through lsof output just to find what and where is eating up my free space.

[GyBacsi]

Moving the windows temp to other hdd was the only solution for me on Windows. Thanks!

[simone_zini97]

Have you tried creating an hard link from the directory you are using to where you want to store this kind of files? An hard link is like a normal link but it is used also from programs. I use it on Windows because I have a really small and pretty old SSD as C drive, only 64 GB, it is like a mirror, OS and programs think they are using it from C drive, instead everything is done where you want it to be redirected, in my case on my bigger and newer 750 GB SSD E: drive. I know it exists also on mac os but I don't know exactly how to do it, try do some research about creating hard links on mac os.

I also suggest you to check them sometimes because they might get corrupted. For example I moved my /Local folder on E: drive but when I update programs the hard link breaks and I have to rebuild it. For this i have written a little batch file that does the boring job for me.

It's pretty easy to notice if it breaks: the disk fills up . I also suggest a tool to discover big files and folders, on windows it is called WinDirStat but I know there are really similar programs for Mac OS and Android

[FrankWPhoto]

Hi, thanks, but see my reply above to pro360media why hardlinking it might be difficult/dangerous.

[rizzlevizzle]

The curious thing here on my end is that once Lightroom eats up all this space during a merge, it doesn't give it back to me after quitting Lightroom. The only way for me to get the space back is to restart my computer entirely (2013 MacBook Pro, 16GB RAM, macOS Mojave). I can't believe that this doesn't impact more people on a regular basis. Among Lightroom's zillion other problems, I'd say this is a pretty major one.

[FrankWPhoto]

So, from your reply I assume the bug is still there, even after half a year (I haven't done any major merging after noticing the issue, so I don't know)? Adobe memory/scratch management at its finest...:(

And yeah, it's only really noticeable if you are merging a lot of big photos into HDR/Panoramas, where the scratch grows rapidly into tens of gigabytes. Hiding the scratch in a hidden file in hidden folder on the system disk instead of respecting user-set scratch location is pretty bad.

[Jao vdL]

rizzlevizzle  wrote

The curious thing here on my end is that once Lightroom eats up all this space during a merge, it doesn't give it back to me after quitting Lightroom. The only way for me to get the space back is to restart my computer entirely (2013 MacBook Pro, 16GB RAM, macOS Mojave). I can't believe that this doesn't impact more people on a regular basis. Among Lightroom's zillion other problems, I'd say this is a pretty major one.

The problem (apart from the fact that virtual memory can only be on the startup disk without some major dangerous hacks) here is that Apple ships Mac Book Pros with 256 GB SSDs as main disks. This is not nearly enough for anything professionally. I have a similar MBP and upgraded its SSD years ago to 1TB and all these disk space management issues disappear (as long as you keep it more than 25% free). It is cheap and easy and your MBP probably can be upgraded. I recommend other world computing (Apple Mac Upgrades - RAM, SSD Flash, External Drives and More​, click on My upgrades and select your machine) for these sort of SSD upgrades but you can find similar kits on amazon and other retailers too. I like the kit with the little external housing that you can put your old SSD in and get a little fast USB3 SSD drive in the process.

[DdeGannes]

Quote "Since there was a lot of the merges queued, my start-up disk almost completely filled up even though I had almost 30GB free on the SSD. I have dedicated SSDs in RAID just for scratch files and working with video, these are much faster than the start-up SSD (256GB) and had ample space. Is there any way to tell LR to use only those for scratch files? Mind you, these"

This is the root cause of your problem the 30GB is insufficient free space on the 256GB SSD drive. You should have at least 20% free space on the drive at all times when you go below that level you will begin to experience significant affect on your system's performance.

See the link below for the Adobe site.

optimize-performance-lightroom.html

This is a clip from the article.

Make sure that you have a large enough hard drive and enough free space

Working with too little free hard-disk space can cause poor performance. Make sure that the hard drive that stores your Lightroom catalog, previews, and image files is at least 20% free.

See Lightroom system requirements to find out the minimum amount of free hard-disk space you need for your version of Lightroom.

[FrankWPhoto]

Not helpful at all. Did you actually even read the thread? There was plenty of free space, and I had a several hundreds GB free ona second SSD, which was set as scratch drive for PS/LR. The bug is that PS/LR IS NOT HONOURING THE SCRATCH LOCATION. Same problem would manifest with 30% free on the system drive, with 90% free, just the same, only a little later in the multiHDR merge.

[DdeGannes]

What I was referring to is the 30GB of the 256GB SSD drive, that calculates to just under 12% and you risk loosing the disk completely.