cr_sdk_########.tmp grows very large (User/AppData/Local/Temp) when batch AI Denoising

[daehxxiD]

See: https://community.adobe.com/t5/lightroom-classic-discussions/cr-sdk-00554910-tmp-grows-very-large-us...

I've observed a very strange behaviour with LrC when using the new Enhance Noise Reduction feature.

For some reason, when I run the noise reduction on a large selection of photos (200+) (photos filtered by HighISO metadata i.e. anything above 3200) at some point Lightroom seems to start writing to my boot drive like crazy:

This results in a very large file, that fills my C: drive completely:

Curiously, the file-size is only visible in the explorer, once lightroom stops the AI Denoising because it "ran out of memory" or if the User cancels the process once. Keep in mind, that this trashing of the OS-Drive DOES NOT happen prior to some arbitrary number of photos (another community member tested with 35 photos). I haven't pinpointed exactly after how many photos the app starts to misbehave, but it is probably around 100-150+ photos.

C should only hold OS and Apps, I've instructed Lightroom and Photoshop to Cache on a separate drive. I'm 99% certain this is a bug and not user error, as behaviour like this should not be expected. Unfortunately this makes using the feature a bit of a chore with many photos.

I let it run last night again, and Lightroom popped up an error-message detailing, that there was not enough memory to complete the denoising for 321 of 600 photos. Memory usage for both RAM and VRAM was however at only around 50% at best. Trying to save the log (clicking on "Save as..." in the error dialog) resulted in an empty .txt.

My System:

Ryzen 9 3900X

32GB DDR4

RTX 3060 12GB (NVIDIA Studio Driver 535.98)

68GB Free on C: drive

Windows 10 Pro 22H2

Lightroom Classic 12.4 (I've first observed the Bug in 12.3)

Edit: Another important info: I'm using DNG-Raws as the source files, that were converted from Sony A7 IV .ARWs by Lightroom/DNG Converter.

[DdeGannes]

This may be your problem "68GB Free on C: drive".

What is the full size of your C: Drive, you need at least 20% of free space on the drive for efficient performance?

[daehxxiD]

The drive is 256GB, perfectly reasonable for an OS-Drive in my opinion. So 68GB is around 26%. This is not a "my problem" problem, this is, I'd wager, a bug that needs fixing. No app should ever start gobbling up 60+ GB on an SSD, especially not on the OS drive (all sorts of bad things can happen if the OS drive is full) and in such an inexplicable fashion (Why does it only do it if I do more than, say,  200 photos?).

Again: The normal and expected behavior is that LR would use the memory it needs to process the files. Preferably it would use volatile memory (RAM/VRAM). If that runs out, it should use the virtual memory provided by the OS and/or use the user-defined Cache folder. In my case, LR seems to be perfectly content with the ressources at and and only works in volatile memory; at least as long as the processed photos are below ~200 (I have not determined the exact cut-off point yet)... However, go above that and LR (or the Camera Raw SDK, judging from the file-name) starts gobbling up OS-drive space like there is no tomorrow. It does NOT use any persistent storage-space (aside from the one occupied by the generated .png next to the original photo) prior to that elusive cut-off point.

[DdeGannes]

Just some thoughts.

The Denoise feature will create a new DNG file which will be 4 times the size of the original DNG and will be saved in the same folder of the source file. This begs the question where are your photos stored and is there sufficient free disk space for the new images?

In addition new previews will be built for the Enhanced Dnoised images and will be saved in the folder that contains your Catalog file. This begs the question where is the folder your Catalog file with Previews and other support data files stored?

What edits if any are you doing with the originals prior to the Denoise feature, recommendation is to do the Dnoise early in the process?

There is info available for the new Enhance Denoise feature in the Thread at the top of the forum announcing the update of LrC 12.4.

[daehxxiD]

Oh, that's interesting! I didn't realize the enhanced dng was 4 times the size, but it is indeed 😮

But I think it should not matter as the answers to your questions are as following:

1) This begs the question where are your photos stored and is there sufficient free disk space for the new images?

My photos are stored on a 6TB NAS network drive and this still has 2TB+ free, so space should not be an issue

2) This begs the question where is the folder your Catalog file with Previews and other support data files stored?

My Catalog is on a separate SSD in the system (Catalog-Size ~800MB, Folder-size with Previews ~5.33GB); the drive still has 160GB+ of space, so that should not be an issue.

3) What edits if any are you doing with the originals prior to the Denoise feature, recommendation is to do the Dnoise early in the process?

Yes, I had read that in the blog post, so this time I did the Denoise-Step as the very first thing after Import (and conversion to DNG). That said, on a prior project I actually did it after the fact (i.e. selected and edited my photos, and ran the denoise on High-ISO images prior to export). This was needed, because I could not just enhance all high-ISO photos in one go, because of the described bug or faulty behavior (but did actually work quite well!).

[GoldingD]

You might want to consider redirecting Windows OS to place the temp files on a different drive.

[daehxxiD]

That's actually a really good idea. I moved it to the HDD in the system; hopfully this won't slow down other apps too much. Still would only regard this as a workaround, and not a solution. The moment this is fixed, I'm restoring the default PATH  to TMP and TEMP 🙂

Would anyone be so kind as to test this and see if they can reproduce the issue? I think the only way we can get Adobe to notice this is if there are a couple of mentions of this issue.

[daehxxiD]

Side-note: 

Observe the HDD usage graph compared to the GPU usage graph:

It seems like every time before or after an image is processed, there is a large write operation happening on the HDD (it was harder to spot on the SSD, because it completes it so fast). My theory is, that this is either the input into the nerual network or the output of it and all of that is somehow aggregated into the cr_sdk_###-file. Essentially what is happening, is that Lightroom just simply forgets to clean out the completed files, for some reason.

[bogdans14041187]

That file is not working as intended.

For example whem lightroom is idle, that file keeps increasing no matter what i do lol.

I usually restart the app a few times. Also i am using an 3rd SSD for this temp file...But still after a while the 3rd ssd that i use only for this file starts to hit the 100% usage and then everything freezes for a while. I will try next to add 32GB ram, to get at least 64GB.

[wonderful_Elixir0D4C]

Did you solve this problem? I'm having the same issue and found this post by searching in google "cr sdk tmp" because I found that this is the file that keeps getting bigger and bigger. 

Exactly the same problem, Intel i7-12700K 32GB RAM DDR5 RTX 3060 12GB with latest drive, SSD 512GB with more that 40% free disk. When I put between 50-150 photos to noise reduction, it starts using more a more space till it stops. I can't leave my PC at night processing more than 200 photos because this problems starts. 

[wonderful_Elixir0D4C]

Here is the infamous file that has no explanation to this date! the file keeps getting bigger and bigger till it fills all disk no matter how much space you have, this one is "only" 32gb because I had to apply AI Denoising in batches and then restart LR... It's weird keep selecting 150 photos, apply AI Denoising, then restart. Select 150 photos, AI Denoising, restart. Someone please help to fix this bug because the file can grow to +100gb if you let it go... I don't thing developers wanted this to work this way.

[daehxxiD]

Hi, no I have not really fixed this yet. My workaround was putting the "%TEMP%" folder onto a large secondary NVME drive as suggested by @GoldingD  and just live with Adobe writing hundreds of gigabytes when I use the AI denoise in batch.

I still agree this should be fixed, but I don't think this is on Adobe's radar.

[R38974687y0h0]

Is there harm in deleting these files? I'm noticing a few different instances of this file being created and all >1GB in size. 

[daehxxiD]

The file should be deleted after you close lightroom.