P: Regarding JXL Compression in derived Merge/Enhance DNG Images

[tigre58]

Why with LR CC v.13, the files panorama files are now saved as lossy files?

[madbrain76]

There is a CODEC for JXL files that allows previews in Windows explorer, but I have not found one that allows previews for the panorama and HDR DNGs with JPEG-XL compression.

The problem is not just about previews, though. I also can't edit these DNG files in Topaz Photo AI; because they don't support JXL compression. An intermediary conversion is needed.

[Olaf314551382lie]

Ich habe Lightroom Classic Version 13.0.1 und Camera Raw 16.0 auf meinem neuen Rechner installiert.

Seitdem bekomme ich VERLUSTREICH komprimierte Panoramen und HDRs mit deutlich reduzierter Auflösung und Dateigröße (Dateityp "Digitales Negativ/Verlustreich komprimiert". Ich möchte jedoch verlustfreie DNG-Dateien erzeugen.

Mache ich etwas falsch oder haben andere das gleiche Problem? Anwendungsfehler oder Bug?

Über Tipps bin ich sehr dankbar!

Viele Grüße

Olaf

[kglad]

in the future, to find the best place to post your message, use the list here, https://community.adobe.com/

p.s. i don't think the adobe website, and forums in particular, are easy to navigate, so don't spend a lot of time searching that forum list. do your best and we'll move the post (like this one has already been moved) if it helps you get responses.

<"moved from cc desktop">

[reproo2773183]

Hi Olaf,

You are not doing antything wrong.

Lightroom's Panorama and HDR Merges to DNG have always been lossy compressed.

I think the report bit that said type "Digital Negative/Lossless" was the "bug".

You should expect the significantly reduced file size because JPEG XL is a lot more efficient than JPEG.

You should NOT expect significantly reduced resolution.

Maybe check if the Merge is using the less sharp of the originals?

[Olaf314551382lie]

Hi reproo2773183,

Thank you so much for your comment! You are right!

I just tried HDR merging 3 Nikon RAW files (47 MB, 50 MB, 53 MB) using my old computer (LrC Version 12.5 + Camera Raw-Version: 15.5) and got an HDR DNG with a size of 151 MB.

With my new PC (LrC Version 13.0.1 und Camera Raw 16.0) I merged the exact identical NEFs, resulting in an HDR DNG size of just 28 MB.

When comparing both HDRs side by side in 100 % view (or even enlarged to 200 % and virtually exposing + 2 f-stops to better be able to assess the dark parts of the image) - I couldn't identify visible differences, despite the file size reduction by approx. a factor of 5. How is this possible? New/better algorithm?

I guess I was confused by the terminology „Digitales Negativ/Verlustreich komprimiert“, corresponding to the smaller size and „Digitales Negativ/Verlustfrei“ for the original one.

Thanks again for your help!

Best regards,

Olaf

[Ross2511000501vd]

Hello all, I wondered if you can help me.

When taking bracketed shots, I have been merging them in Lightroom as an HDR merge. To give you an examples my RAW files are each 85MB in size. I am generally merging 5 shots. Once it has created the HDR merge, which provides a DNG file, the size is reduced to 14MB for the stacked file. Is this correct or is there something I am missing? I believed that the idea with stacking or bracketing was to retain more data from each shot. Am I wrong in thinking that the size of the output DNG file should be as large as all RAW files put together? Please do let me know any thoughts or if there is something I am missing or have done incorrectly. I have attached a sample photo showing the smaller merged file and the original RAW file.
Many thanks

Ross 

 

 

[reproo2773183]

Stacking is  just how the files are displayed/hidden in Lightroom.

The HDR DNG is a demosaiced Merged file it won't contain 5xRAW data, only one Layer of pixels and that single layer is now compressed, whereas the RAWs may have been lossless.

I think you are assuming a structure similar to 5 Layers in Photoshop 5, thats not whats in you DNG.

You get to keep the extra dynamic range information from bracketing because you effectively have more bits (smaller steps) to play with.

[madbrain76]

I'm not the one you were responding to, but I thought I would respond still.

I'm not personally assuming that any of the original raw data is preserved in the merged file - HDR or Panorama.

What I'm assuming is that the process goes like this.

1. Merge x DNG files into one demosaiced set of pixels, in RAM

2. Compress that data on disk

If LRc were properly designed, you could choose the compression method in step 2, one with lossless JPEG XL, and one with lossless JPEG.

You could then export each output DNG to a TIFF, then use a tool to compare the pixels data, to see exactly how much information was lost.

Unfortunately, the option to choose the compression for photo merge is missing in LRc, and there is thus no way to do that.

I would try to test this by comparing a previously created lossless DNG Panorama,  to a newly recreated one as a loss DNG. Unfortunately,  the merge algorithms have changed. I was unable to recreate one of my existing panorama in the current version of LRc. The merge algorithms change with each version of Lightroom.

It is thus not possible to keep the compression algorithm as the only variable.

A someone who can hear the difference between lossless audio and compressed audio, I think this is a choice that needs to be left to the user. Just because I have macular degeneration and can't personally tell the difference between the lossy and lossless images doesn't mean someone else won't be able to tell.

This is just a braindead decision by Adobe. They need to provide a lossless option. Also, since JPEG-XL doesn't yet have wide acceptance in other software (not even Windows Explorer shows previews), the option to switch between JPEG-lossless and JPEG XL-lossless should be there. Just like it is for regular Export rather than Merge.

[reproo2773183]

What I'm assuming is that the process goes like this.

1. Merge x DNG files into one demosaiced set of pixels, in RAM

2. Compress that data on disk

If LRc were properly designed, you could choose the compression method in step 2, one with lossless JPEG XL, and one with lossless JPEG.

You could then export each output DNG to a TIFF, then use a tool to compare the pixels data, to see exactly how much information was lost.

By @madbrain76

I'm assuming Step 1. accounts for 99% of the overall reduction in quality, which is in itself fairly minor.

Step 2. becomes the high quality set of pixels that you can continue to non-destructively edit in Lightroom.

Because of Rik's post I'm assuming this was previously JPEG (and only 8 bit, and don't know if this is lossy or lossless) but could be wrong on any of this. Now I'm assuming this is now JPEGXL and lossy but possibly 16bit.

When you are done you choose to Export in the format of your choice.

I'm not seeing a downside, the first JPEG compression of an image at best quality is super subtle.

You would be unlucky to see it even if you enlarged to 400%.

My pano workflow almost always involves finishing in Photoshop and there (for ultimate quality) I'd use 16bit PSD or PSB and smart objects. 1 rotatation or scaling will cause more damage than JPEG encoding. The difference with Photoshop is that potentially you're making destructive changes so re-saving as JPEG requires re-compression and cascading degradation.

[madbrain76]

A few things :

a. the compression was definitely lossless when doing Photo merge in older revisions of LRc. My previously created Panoramas and HDR i mages show up under "Digital negative / Lossless" in "Library filters / file type". The new ones I created in LRc 13.51 show up under "Digital negative / lossless".

b.  I verified with exiftool that previously created HDR and Panorama DNG have a 16 bit depth. The original DNG files from my Pentax K-1 II that were merged have a 14 bit depth. There is no 8-bit conversion involved.

c. when you do a Photo merge in LRc, both steps 1 and 2 are combined. It's not possible to do a separate "merge" and "export".  That is the whole problem. You cannot choose lossless instead of lossy. Adobe chooses for you, and discards some of the data.

d. While you and I may not be able to immediately tell the difference, someone printing an image very large might very well notice. That is not my case. However, I sometimes do several levels of processing. For example, with panoramas, LRc has given me problems in the past when trying to merge a large number of images at once. It is sometimes completely unable to do any merge at all in this case. But removing some images makes it work, somehow. I believe it's an LRc bug. What I end up doing in this case to work around it is creating multiple sets of panoramas, and then merging those panoramas. That works. It's not ideal, obviously. And now, there will be further degradation at each step. I sometimes do the same for HDR panoramas, for similar reasons.

Finally, there is sometimes external processing involved, also, with Topaz Photo AI, which also outputs DNG.

These are just some of the reasons why using lossy compression is a bad idea.

As a test, I checked my second biggest HDR Panorama, which was made with previous versions of LRc, and is 419 MB. I exported it as DNG "lossy". The resulting DNG file is only 69 MB.  I'm very skeptical that compressing by a factor of 6 does not lead to any information loss. I exported both to TIFF, and they were 528 and 527MB respectively. Those TIFFs are using ZIP compression. The image is 14744 x 7648 at 16 bitd/channel. , so it would be about 650MB uncompressed.

Anyway, I downloaded Imagemagick to compare those DNG files. Unfortunately, it didn't support either the old lossless DNG, or the new lossy DNG. So, I converted them both to TIFF.

Input:

magick.exe compare -metric AE "TIFF lossless.tif" "TIFF lossy.tif" diff.tiff

Output :

1.11395e+08

ChatGPT said:

Here is the interpretation from ChatGPT :

Interpretation

1. Absolute Error (AE):
   • Value: 1.11395e+08 (which is 111,395,000 in decimal notation).
   • This value indicates the total sum of the absolute differences between all corresponding pixels in the two images. A larger number signifies greater total pixel difference between the images.

What This Means:

• High AE Value: A high AE value indicates significant differences between the two images. It suggests that the images differ substantially in pixel values, which could be due to compression, quality loss, or other image alterations.
• Context: To better understand whether this value is large or small, you might need to consider the size and complexity of the images. For instance, larger images with more details typically result in higher AE values, but relative comparisons are more insightful.

Additional Steps:

1. Visual Inspection: Open the diff.tiff image to visually inspect the differences between TIFF lossless.tif and TIFF lossy.tif. This image will highlight areas where the two images differ.
2. Normalize: To better understand the AE value in relation to image size, consider calculating the average error per pixel or comparing it to AE values from similar comparisons.

If you need to quantify the differences further, you might use other metrics such as Mean Absolute Error (MAE), Root Mean Square Error (RMSE), or Structural Similarity Index (SSIM) for a more detailed analysis.

I reviewed the diff.tiff . It is almost completely red. There is very little of the original photo visible.

I'm not happy about this change at all. Adobe should have allowed lossless JPEG-XL compression, not lossy. DNG is meant to be archival format, and removing some data is simply unacceptable.

[JohanElzenga]

I believe you forget one thing: the DNG that Lightroom creates is linear RGB. That means that 50% of all information is in the highest stop, 25% is in the next stop, etc. No matter how much editing you are still doing when you convert this to a gamma-corrected image before you print that, you will never spread out the information equally over all the stops. The brightest stop will always have some 'information overkill'. That's the nature of the beast called 'digital photography with linear sensors'. If lossy compressions only removes some of that overkill, then it is unlikely you are going to see that, regardless of how large that print is going to be. Mind you, I am not saying that this is how it works, but it could be how it works. And so I think you are overreacting a little by calling this 'unacceptable' without having any real proof why that would be the case.

[John R Ellis]

"The image is 14744 x 7648 at 16 bitd/channel. ... Absolute Error (AE) Value: 1.11395e+08"

That's an average error per pixel per channel of 1.11395e8 / (14744 * 7648 * 3) ~~ 0.33. In a 16-bit file, the channel values range from 0 to 2^16-1, or 0 to 65,535. So the average error per pixel per channel is about 0.33 / 65,536 = 0.0005%, or 1 part in 200,000.   

Such an average error would be completely imperceptible, unless most of the error was clumped into a much smaller set of pixels.

[madbrain76]

John,

If the changes are so small, how do you then explain the resulting diff image looking nearly all red ? Doesn't that mean that most of the pixels in the image (98.7% of them) have different values, for at least one if not all of the color channels ?

Doesn't this error compound every time you do a merge, if you are doing things like merging multiple panoramas or HDR ?

Unfortunately, Adobe did not provide a slider for the quality/compression ratio for DNG in JPEG-XL mode, even in "export mode".  There is only a checkbox for lossless or lossy. for DNG export

However, when exporting to JPEG-XL, there is a quality slider, as well as a "lossless quality" checkbox.

I just exported the original lossless DNG, which was 419MB, to lossless JPEG-XL . It resulted in a 347MB JXL file.

Then, I tried 0% quality. I got a 1.51MB file.

I then tried 50% quality. This resulted in a file that's only 6.10 MB ! Extremely surprising.

My next test was 75% . That yielded a 10.2MB file.

Afterwards, I did 90%. I got 16MB.

98% produced a 36.8MB file.

99% produced 43.7 MB.

That's a rather massive difference between 99% JXL and 100% quality JXL - 43.7 MB to 347 MB.

The 43.7 MB is still less than the 69MB DNG in "lossy" mode. Perhaps there is a fractional quality setting in between 99 and 100% that's being used for DNG lossy during Photo merge.

I then ran Imagemagick on the 0%, 50% and 99% files.

Surprisingly, the ratio reporterd by Imagemagick for the 0% 1.51MB file is nearly the same as the 99% 347 MB file. I can tell the difference visually between the 0% and lossles files, though even there it's not entirely obvious - it can best be seen on text. This image does not have any people in it - it's a handheld shot of night cityscape and is not especially sharp, even though it's one of my favorites,  but I would assume the very high compression ratio could create problems when doing further processing such as AI face recovery in Topaz. Time to test that hypothesis with other files.

[John R Ellis]

"If the changes are so small, how do you then explain the resulting diff image looking nearly all red?"

I was just interpreting what the AE number meant. Without seeing the two DNG files and the difference file computed by "magick", I can't speak to what you're seeing. 

[madbrain76]

John,

You can check out the images from

https://drive.google.com/drive/folders/1Zq_6zNtcAmphIzPWbF-1u3zRfOsdgpLM

The original files are the DNG , but ImageMagick cannot process either of them. You can use the TIFF conversions. I have also included JXL conversions, since they allow the rate of compression to be selected.

I did not include any diff image file,  as my google drive is nearing its 15GB capacity just from two decades worth of emails, let alone any images. I will be deleting the files in the near future.

[reproo2773183]

a. the compression was definitely lossless when doing Photo merge in older revisions of LRc. My previously created Panoramas and HDR i mages show up under "Digital negative / Lossless" in "Library filters / file type". The new ones I created in LRc 13.51 show up under "Digital negative / lossless".

By @madbrain76

I don't beleive this to be true anymore.

Please read page 1.

Previously "derived DNG's used JPEG" and this wasn't 16bit because Rikk Flohr crossed that part out.

I've also read on another thread that the "Digital negative / Lossless" returned information was incorrect and a bug.

Original JPEG encoding does not support 16bit. Which is why I asked the question to Rikk, thats why I assume now that at some point the Merge is 8bit.

The End Merge is 16bit but if you paste 8bit image into a 16bit document all that happens is the numbers get more decimal places.

Your File sizes to me don't make sense to be uncompessed 419MB DNG becomes 527MB LZW TIFF and 650MB opened in Photoshop. The DNG must be compressed but even if original JPEG supported 16bit, which it doesn't, I don't think a lossless JPEG would be so much smaller than LZW.

Your arguments are sound but at this point I think they're based on assuming the previous method was end to end lossless, and I don't believe it was.

[madbrain76]

I don't beleive this to be true anymore.

Please read page 1.

Previously "derived DNG's used JPEG" and this wasn't 16bit because Rikk Flohr crossed that part out.

I've also read on another thread that the "Digital negative / Lossless" returned information was incorrect and a bug.

Original JPEG encoding does not support 16bit. Which is why I asked the question to Rikk, thats why I assume now that at some point the Merge is 8bit.

The End Merge is 16bit but if you paste 8bit image into a 16bit document all that happens is the numbers get more decimal places.

The main reason I joined this thread is that I was skeptical of some of the statements that have been made.  I would like to follow the evidence and either corroborate them or disprove them.

Regarding the 16-bit issue, it seems a bit odd that LRc was in the past creating HDR images with a path limited to 8 bits per color channel. The process still output a 16-bit/channel DNG file, so there must have been some reason for using that. I don't have any tool handy that can analyze these files and see if there are 8 unused bits for color values in the output files. ChatGPT says that part of the merging process in LRc was 8-bit for performance reasons. It seems to me that this would be independent of the compression used. Ie. the merge is done first, and the image is subsequently compressed, either losslessly or lossily.

Anyway, I thought I would try to find out for myself what the actual behavior is.  I exported the my "DNG lossless" HDR  panorama, made with previous versions of LRc, to both 8-bit and 16-bit TIFFs using LRc. The 16-bit version is 528MB, while the the 8-bit version is just 129MB.

magick compare -metric AE "TIFF lossless.tif" "TIFF lossless 8-bit.tif" diff8bit.tif
1.1268e+08

I also exported the same panorama to a JPEG at 100% quality and 8-bit. The JPEG is just 49 MB.

magick compare -metric AE "TIFF lossless.tif" "JPEG lossless.jpg" diffjpeg.tif
1.1272e+08

Both could be explained by the 8 bit color values not being shifted the right way by comparing 8-bit against 16-bit files, so this is not really conclusive. However, the number reported in each case is not the same, which is not normal for two lossless algorithms.

So, I compared the two 8-bit files.

magick compare -metric AE "TIFF lossless 8-bit.tif" "JPEG lossless.jpg" diff8bittiffjpeg.tif
9.68164e+07

It turns out that rendering to a JPEG 100% quality in LRc is not the same as lossless. That sure is confusing ! So, my "JPEG lossless.jpg" needs to be renamed "JPEG 100%.jpg".

Also, it turns out ImageMagick is able to handle one of the DNGs, the original so-called "lossless" one. I ran Imagemagick against the 16-bit TIF exported from LRc.

magick compare -metric AE "DNG lossless.dng" "TIFF lossless 8 to 16 bit.tif" dngtiff.tif
1.12756e+08

Surprisingly, they are also not the same.

I then use LRc to export the 8-bit TIFF back to a 16-bit TIFF. I compared the two.

compare -metric AE "TIFF lossless.tif" "TIFF lossless 8 to 16 bit.tif" diff8to16.tif
1.1267e+08

They were also different.

I then converted that back to 8 bit, again using LRc.

magick compare -metric AE "TIFF lossless 8-bit.tif" "TIFF lossless 8 to 16 to 8 bits.tif" diff8to16to8.tif
1.90508e+07

My conclusion is that starting with an 8-bit TIFF, export it to 16-bit , and then back to 8-bit, is a lossy process in LRc. Strange, since LRc only exports to TIFF with lossless LZW / ZIP or no compression.

Your File sizes to me don't make sense to be uncompessed 419MB DNG becomes 527MB LZW TIFF and 650MB opened in Photoshop. The DNG must be compressed but even if original JPEG supported 16bit, which it doesn't, I don't think a lossless JPEG would be so much smaller than LZW.

Your arguments are sound but at this point I think they're based on assuming the previous method was end to end lossless, and I don't believe it was.

By @reproo2773183

The original panorama DNG , made by LRc 10.2, and designated as "Lossless" in 13.51, is 419MB. I did not say that it was uncompressed. It is losslessly compressed. LRc does not offer an option to disable compression on DNG files when doing an export. ChatGPT says the format does allow uncompressed files. I have never run across one.

If I export the lossless DNG to uncompressed 16-bit TIFF, I get a 645 MB file, just as I predicted.

I don't believe I previously mentioned a  [plain] JPEG file, only DNG, JPEG-XL and TIF.

While the JPEG file format indeed does not support 16-bit/channel, DNG files using the "lossless" option in the current version of Lightroom definitely do support 16-bit/channel, whether one is doing a manual export, or a photo merge. I don't think that fact is in question. You can very easily tell from the EXIF . Both the "lossless" and "lossy" versions of my HDR panoramas DNG contain this, according to exiftool.

[EXIF]          Bits Per Sample                 : 16 16 16

Now, when it comes to compression, Exiftool shows the following for the "lossless" version :

[EXIF]          Compression                     : Adobe Deflate

And for the lossy :

[EXIF]          Compression                     : JPEG XL

What's Adobe deflate ? Here is what ChatGPT says :

Adobe Deflate compression is a lossless compression algorithm used in formats like TIFF and PDF. It is based on the same algorithm used in the ZIP file format, called the DEFLATE algorithm, which combines LZ77 compression and Huffman coding to reduce file size without losing any data.

Adobe Deflate is often an option when saving TIFF files, offering efficient compression while preserving the full quality of the image. Since it’s lossless, all the original image data can be restored perfectly when the file is decompressed. This makes it useful for scenarios where maintaining high image quality is critical, such as in photography, medical imaging, and printing.

It is commonly chosen over other compression methods like LZW (another lossless compression) because it often results in smaller file sizes, especially for larger images.

I don't know if the AI is correct, as it does not cite a source. But this is a lossless compression algorithm. Nothing to do with JPEG, which is typically lossy.

[reproo2773183]

Quick Summary of your Exports

Starting with:

419MB old panorama derived DNG 16bit compressed (probably Adobe Deflate, probably lossless)

Exported to:

645MB 16bit TIFF uncompressed

528MB 16bit TIFF LZW lossless

129MB  8bit  TIFF LZW lossless

49MB     8bit JPEG 100% quality

69MB 16bit DNG lossy (new JPEG XL compression) —> becomes 527MB if exported as TIFF LZW lossless

347MB  JPEG XL lossless

43.7MB JPEG XL 99% quality

36.8MB JPEG XL 98% quality

16.0MB JPEG XL 90% quality

10.2MB JPEG XL 75% quality

6.1MB   JPEG XL 50% quality

1.51MB JPEG XL 00% quality

You’ve stated that you can see the difference between the 1.51MB JPEG XL 00% quality and the Lossless files but its not obvious.

Firstly, You are concerned that someone with better eyesight will be able to tell the difference between the Lossless (oldDNG/TIFF) and the 69MB JXL.

Secondly, You are concerned that if you want to merge 2 of the new panos together, or merge 1JXL pano with one RAW, the quality will be worse than it was before.

I work in Pre-Press so have pixel peeped at JPEGs a lot.

The first conversion to JPEG (old) is almost indistinguishable from the original. Its absolutely good to print from like 999 times out of 1000.

In my experience to find some jpeg noise you need to look for areas adjacent to edges of fine detail, so for example a tall ship’s rigging or guitar strings, roof tiles, and the area next to the edge should have some fade from one shade to another.

Insufficient resolution is much more noticable than JPEG compression.

Downsampling and applying JPEG compression at the same time is actually much worse than just JPEG compression, but even this requires high zoom level pixel peeping that you won’t see in print.

Your second concern is more valid. Any crop or pixel based edit followed by re-save as JPEG will make things worse, each generation of JPEG gets progressively worse. Most images can tolerate 3 generations at 100% quality after that it is image dependant. 

Rotating in Photoshop or Scaling in Photoshop or stretching will reduce quality more than 1 round of JPEG compression.

You should factor in, that for Print you or your Printer will probably use up one generation when exporting to PDF (from say InDesign).

I was concerned that merging 2 panos to a new one wouldn’t work out but I honestly can’t see any difference.

If you did see a difference you can fall back on Photoshop and Merge two 16bit TIFFs.

If you want to experiment further, you could find an image with fine detail and Export to JPEG then actually find an area that gets JPEG noise, once you’ve found one, then Export the same image to JXL lossy and compare.

the same area.