P: DNG Converter running out of memory (version 17.1.0.2100)

Report · Dec 26, 2024

I've reported on memory issues with the DNG Converter before, but it's getting worse and someone needs to tell the developers.

Today it failed to convert 11 images because it ran out of memory. This is on a 64Gb system with 32 threads (AMD Ryzen 9 3950X 16-core)

From the outside looking in - I'm pretty sure there is a memory leak. I'm guessing that it opens the Sony ARW files and doesn't free up memory after converting and writing the DNG to disk. So the more files it processes, the more memory it uses and pretty soon it runs out of memory. It had successfully converted 42 files before it started showing the "not enough memory" errors.

Before anyone suspects there was something wrong with the 11 images that it failed to process the first time, I re-ran the DNG converter and JUST those 11 images and it finished all of them very quickly and without error.

This is with version version 17.1.0.2100 set to:
Camera Raw 16.0 and later

JPEG Preview: Full Size
Embed fast load data

Don't use lossy compression
Preserve Pixel Count
Don't embed original

Report · Dec 13, 2023

I walked away after loading tonight's session and after about a half hour came back to discover the DNG Converter still had not finished converting about 900 files. That is crazy long on my system. I noticed that moving my mouse was jerky, and that the conversion itself appeared frozen. So I aborted it. Moved all of the finished files and restarted with only 211 files left to go.

After it had finished about 100 of the images, the system was again extremely slow and the conversions were taking forever. So I brought up Task Manager. You can see what it looked like in the screen capture down below (100% cpu utilization and 87% memory). This is on a Windows 11 machine with 64GB of memory and an AMD 3950 with 16 cores (32 threads) all of them pegged at full utilization.

I suspect the problem is a memory leak causing the DNG Converter to usurp all available memory and then the system just drives itself into the ground when there's very little memory left.

This occured using version 16.0.1.1683 with the following settings:
JPEG Preview: Full Size
Embed Fast Load Data

Don't use lossy compression
Preserve Pixel Count
Don't embed original

And this happened while processing files from my fairly new Sony a7iv (not 'R').

I have had a Sony a7Riv (different from above - this one is a 60mp camera) for a couple of years now and have experienced "slow-ish" conversions, but nothing approaching this level of sluggishness nor taking over the system CPU and memory the way this one is. So the problem might be a combination of the new camera (a7iv) and the settings I'm using (see above).

Here is a screen shot of what Task Manager looked like after converting about 100 of the a7iv images:

Report · Dec 14, 2023

Moving to Camera Raw forum, so that @Rikk Flohr: Photography can notice it.

Report · Dec 20, 2023

Just reporting back... I noticed there was a newer version today so I downloaded it. Version 16.1.0.1728.

Same problem. It very quickly zooms up to 100% memory and cpu utilization. I let it convert a couple hundred, cancel the conversion. Kill it via Task Manager (because it takes FOREVER to exit even though the UI has closed). Then restart it on the rest of the files. It's tedious restarting multiple times, but it's faster than letting it get stuck forever at full cpu/memory utilization.

Report · Dec 21, 2023

Does Lightroom, Adobe Camera RAW, or Bridge (using the Export feature) have the same problem?

Report · Dec 21, 2023

I can't answer that. I use the DNG Converter to convert all of my raw files to dng *before* importing to Lightroom, I've been doing it this way since 2018 when I switched to Sony cameras. Maybe someone who uses one of the other workflows can try it out?

Because it looks like a memory leak, I do believe the quickest solution would be to get one of the developers to look into ways that memory might not get freed after an image is converted.

Report · Dec 22, 2023

So do a test?

Report · Mar 17, 2024

I am having the same problems but with a M3 Max Macbook pro and Nikon Z8 files. It is just ridiculously slow. I am trying to convert 6000 at a time some days and it has become unworkable. Like over 12 hours to convert them. Its almost as if the app has to be in front on screen to work properly. When left unattended and my screens go to sleep it seems to just slow right down. The whole system is set for max performance at all times. It is baffling and extremely frustrating. It was always a time consuming thing but this is crazy. My conversion settings are the same as yours. My version is 15.3.1.1455 but given what you said the version doesn't really seem to matter at the moment.

Report · Mar 17, 2024

Maybe do the conversion in Bridge or Lightroom instead?

Report · Mar 17, 2024

I did a test with 200 50mb images using conversion from Bridge and the standalone converter. From Bridge it took 4 minutes 23secs and the standalone converter too 7 mins 30 secs so over 6000 odd files that would be a 90 minute saving which is well worth having. It will be interesting to see if the Bridge conversionslows down over time because the standalone is. It took way more than 4 hours to do 6000 from my last job. More like 14 hours. I confess I didn't actually know you could convert straight from Bridge. I have been using the dng converter since it was first launched. Thanks for pointing me in that direction.

Report · Apr 22, 2024

To update my last entry. It seems that the standalone DNG converter slows right down when my screens go to sleep. It's odd beccause everything is set not to sleep or reduce power etc. I watched it convert 1500 files in about 90 minutes, left it running and went to bed with about 3500 still to conert. I got up about 6 hours later, woke the screen and it still had 200 to convert! I ran the same test using the converter in Bridge on the exact same files and all 5000 file took about 4 or so hours. So the dng standalone is seemingly affected by the screens going to sleep which seems very weird. It also quietens right down once the screens go to sleep so if the fans ere running during conversion they would fairly quickly shut off once the screens went to sleep. All really odd.

Report · May 04, 2024

Update. My previous test that suggested that the DNG converter in Bridge was faster is actually not the case. I had the preview set to medium which makes the whole thing faster. I need full size previews so it actually is no better than the standalone converter. What I cannot understand is that if the converter is not the foreground app it appears to slow right right down. I set it to convert 1700 50MB files at 3am. By 8.30 am there were still 600 to convert. My computer is not set to sleep or reduce power when not in use but waking the screen and making the converter "front" app meant that the remaining 600 took less than hour to convert. This is so frustrating because I cannot seem to find a way to stop this slowdown. It seems to be a memory leak but when I am sitting in front of it and I make the app come to the front it all speeds up again!!

Report · May 10, 2024

Bump! This issue seriously needs to be addressed. I redownloaded v14.4 and it is x4 faster as I measured it. The current version v16 seems to reduce file size more compared to the older versions which is nice, but the program frequently crashes now too!

Report · May 26, 2024

A further update. 6500 50MB raw files just took 12 hours to convert! This is on an M3 Max with 36GB ram reading and writing to a 4TB SSD with a read/write speed of 3200Mb/s. This just makes the converter completely useless. The behaviour is really weird. If it isn't the front active window it seems to almost go to sleep then surges and causes the fan to run full blast for a few minites then it calms down or slows down again. 12 hours is absurd.