Convert to DNG in 13.0.1 is very slow, CPU/Memory intensive and crashes

[yphoto123]

I have been scanning old negatives with VueScan, converting them from Negative to Positive in LR for nearly 4 years now.  The last step in my process is to convert the files to DNG in LR.  Since 13.x this step has slowed to a crawl and even crashed LR (sent a bug report to Adobe when it crashed).  

I select a group of photos to convert (10-20) to DNG, I then watch high utilization and memory usage jump from about 10gig to 36gig or so. Converting 10 images in the past took maybe a minute.  Now it's at least a minute for 1 file.  I made an attempt to convert 22 files this morning, LR crashed after about 18.

I have tested this with no other apps open, without GPU acceleration with the same results. Something has definitely changed in this version.

Has anyone experienced this issue?

The other bug I have is dual monitor zooming in Develop mode, something I use every day, now broken. This has been acknowledged in other posts here. Another bug dealing with plugins was quickly fixed in 13.0.1. Terrible update as far as I am concerned.  I have made too many changes to my catalogs in 13.x to try to go back to an older version.  The quality team has serious work to do.

[Rikk Flohr: Photography]

We’ve reviewed the email address you use with this forum and find no crash reports in our system.

Did you see a crash dialog?

Do you use a different email address when filling out the crash report?

Was the crash dialog Adobe’s? Microsoft’s?

FWIW, I convert to DNG daily and have done several batches of 100 or more in the past 2 weeks with no issues with regard to speed or stability. 

[yphoto123]

I just had it crash again trying to do 10 images.

I use a different email address though I don't think it is wise to post that in a public forum. 

It is Adobe's crash dialog. 

I am sure I will have it happen again today if there is anything you want me to do to diagnose it.  Again, this has been a routine for me for four years, this issue started with LR 13.

These files I am converting are very large, that is the whole reason for the conversion 400meg out of Vuescan, about 25meg after conversion. Maybe that size is the difference in your batches to mine?

Thank you for your response Rikk!

[yphoto123]

I just checked and it is the same email address.  I am sure in the next hour it will crash again, I will do another report.

[Rikk Flohr: Photography]

Screenshot the report after you've entered your email and post the screenshot here. That should be safe.

[yphoto123]

Converting 10 images and it crashed, Adobe Error screen shot, I did submit and confirm my email address is what was entered here.  I am not looking to advertise this email address on a forum so I blurred it out.

 

 

[D Fosse]

What format are they out of the scanner? Not that it answers your question, but DNG is just a wrapper. A TIFF converted to DNG is just a TIFF inside a DNG. So until you get this sorted, are you sure you really need to do this conversion?

I too convert all my 60 MP raw files to DNG. Not quite as big as yours, but stll quite big - and there can be 300 of them in one go. I haven't had any slowdowns or other problems.

[yphoto123]

They are a RAW file straight from the scanner. 48bit RGB, 10000dpi from 35mm film.  I started this film project back when COVID started.  I have converted over 20,000 files in the 4 years and still going.  The disk they are stored on and converted on is an M.2. The only reason I do the conversion is the huge file size reduction.  In the past I would do 100 in a batch without an issue.

Thank you for message!

[D Fosse]

I don't want to split hairs, but 48 bit RGB is not a raw file. That's a rendered RGB file, encoded into a color space. A raw file is linear mosaic data from a camera sensor, usually 14-bit single channel.

The point I'm still trying to make is that DNG may not serve a real purpose here, and it may be more efficient to skip that step.

I still agree that if something used to be fast and now is considerably slower, then yes, there is a problem that needs to be fixed. No argument there. But in the meantime, you may be able to bypass the problem for now.

EDIT - You mention file size - if there's a reduction in file size, there's compression. Compression is always slow, depending on the algorithm, so maybe look at the settings.. There are many ways to compress.

[yphoto123]

I have done some research on the scanning formats in VueScan:

VueScan reads raw CCD sensor data from scanners and can write this to a raw TIFF file for later reprocessing... VueScan internally keeps all samples in 16-bit linear format...

I do not have the in-depth knowledge you do on the RAW/DNG format and appreciate what you have written.

Not only is the process slow, it crashes LR.

You may be onto something with the compression, I have tried converting with and without lossy compression  in LR. Without compression it was much faster. In the capture below you can see why I am running this conversion. First DNG is from the scan, second converted without compression, third (my normal workflow) with compression.  I definitely want to store a 20meg image over a 403meg.

I took the test a bit further by converting 12 files without the compression option, all 12 processed in about 30 seconds, no huge memory spike.  I converted the same group of images with the lossless compression enabled (in the LR dialog) after about 20 seconds the memory spiked from 12g used to almost 40g before LR crashed.

While I wait for a possible fix, I have moved my images to a folder to be resized at a later time.

Thanks again

[D Fosse]

OK, if VueScan can output linear sensor data I suppose you could call it raw, as long as it's not confused with true mosaic raw data from a camera. It would be similar to what Lightroom outputs from Denoise or Super Resolution, linear RGB.

Anyway, this suggests that the underlying issue is indeed the compression. I think there was a recent change in Lightroom compression algorithms (haven't really looked into that) - but I don't know if that would also extend to DNG conversion from Lightroom.

Either way, compression can be very slow, and on very big files to the point where it's almost unmanageable. Just as an example, saving a 5-10 GB PSB in Photoshop can go from 15 seconds uncompressed to 2-3 minutes compressed. Theres just a lot of math to process.

[yphoto123]

Seems Adobe did make changes to the compression and this is likely the cause of my issue.  Hope this can be reported to them.  On page https://www.lightroomqueen.com/whats-new-in-lightroom-2023-10/

I am cool with this change, if it worked.

Do you know the best avenue to report this to them? Should I call?

[yphoto123]

I think I have useful information.  I came across Adobe DNG converter and decided to see what happens with this tool. Trying the same conversion it crashes with the same settings. Maxes out the memory and crashes.  BUT... If I use Camera RAW 14 and later instead of 15.3 and later my conversions are successful, no spike, no crash and done in about 20 seconds for 12 images.  I am not sure if I can test an older Camera RAW with LR, I will look into that later today.

[yphoto123]

Doesn't look like you can downgrade Camera RAW.  When I install it, 16 is still reported to be the installed version. Oh well.

[DdeGannes]

@yphoto123 , LrC version 13 has Camera Raw 16 builtin. Each upgrade has its own version of Camera Raw.

LrC version 12 has Camera Raw 15

builtin.

Lightroom version 1 was released in Feb 2007 and Camera Raw builtin was Camera Raw 4.

[Petr Klapper]

The new JPG XL DNGlossy conversion is horribly slow (personally using DNG Converter on Windows). 

While I appreciate smaller file size, the around 4x slower conversion is too much. Part of the problem I'm sure is also the fact the CPU is not fully utilized for some reason like with the older internal JPG compression. And seems to get even worse a bit when running longer batches.

It would be great to be able choose the compression type until the new process is really optimized.

[yphoto123]

I have had support inside my computer on several occasions since this thread started.  They updated drivers, windows updates and running fixes on each c++ redistributable program on my PC.  These were not out of date more than 2 weeks.  They were able to convert (slow) but succeeded. Immediately closed their ticket.  Two days later I started a conversion of 40 images, fails constantly and my ticket is closed.  Today same issue.  Very frustrating.  I believe my DNGs are larger than most people, so I am experiencing something others likely don't.

I agree the ability to choose the compression type would likely allow me to convert my images in one batch instead of dozens of crashes hoping to convert a couple images at a time.  I have submitted 6 or 7 crash reports today alone.  😞

[52Photos]

I'm not sure on the scanning etc, however I can confirm there is most certainly an issue with speed in this version.  The upgrade to a new dB version has killed any speed lightroom had in the previous version.

[trunkafunk]

I am experiencing this problem now. my typtical import process on my older intel imac was 8x faster than a brand new Mac Studio (M2).  

[Lumigraphics]

Did you migrate your apps from the older computer or reinstall from scratch?

[trunkafunk]

I migrated.. Sometimes LRC uses multiple processors, sometimes it appears to be single threaded and deadlocked between multiple operations. 

[Lumigraphics]

Reinstall from scratch so you get all Apple Silicon code. You don't want migrated apps on your new computer.

[Lumigraphics]

Just scan to TIFF and forget using DNG format, it does nothing for you. I have an Epson Photo scanner and Vuescan and just save to TIFF which is perfect.

[Lumigraphics]

OH and FWIW, a DNG is a fully-compliant TIFF file. What you are seeing is a byproduct of trying to batch compress huge files, your are probably running out of memory. Can you run them through Photoshop and save out to a compressed TIFF?

[yphoto123]

Initially it was maxing out my 64GB of RAM and 100% CPU.  After uninstalling/reinstalling LR per Adobe's request that no longer seems to be the issue.  Memory sits around 65% when converting.  Sometimes converting just 1 image causes the crash and utilization is not high.  Also, just the DNG converter with the latest Camera RAW selected will crash, though the 14.x version will complete properly.