P: Certain raw files export unusually large with JPEG XL

[JtheNinja]

In most cases, exporting AVIF and JPEG XL with corresponding settings produces files of fairly similar size. There's some variance as expected, some images compress a bit better with one format than another.

But certain raw files inexplicably compress very, very poorly with JXL. Here's a sample DNG file that exhibits the problem. https://drive.google.com/file/d/1Cs7TiY7LeA3CGUVCAzqKUl9pLSK6ek7X/view?usp=sharing

It seems common to DNGs produced by the AI denoising function or HDR merge tool, but that is not universal (I've found raws from the same camera that can be enhanced or made into HDR stacks and still compress normally on export)

The settings in the attached screenshot barely compress it at all from the original DNG! Despite said DNG containing over double the amount of pixels, and ostensibly less compression. The AVIF version with the same resolution and quality setting is over 80% smaller than either one. Again, on most images switching to AVIF at the same resolution and quality setting tends to result in fairly similar file sizes, not a 80-90% size reduction. Something about the attached DNG does not seem to be compressing properly on export.

[moved from bugs to discussions according to the community rules  - Mod.]

[GoldingD]

The Enhanced DNG file contains a demosaiced file, which is typically 3 times the original file size as it now contains RGB channels. Additionally, the Enhanced DNG file also contains a copy of the original mosaic data (i.e. a copy of the original raw file (copy from Ian Lyons in ref 1)

They do not behave the same as normal DNG files

1) https://community.adobe.com/t5/lightroom-classic-discussions/why-are-dng-files-produced-bt-denoise-a...

[John R Ellis]

[This post contains formatting and embedded images that don't appear in email. View the post in your Web browser.]

With your sample DNG, I made  a lot of measurements of file sizes, comparing exported JXLs with exported AVIFs and with JXLs produced by Imagemagick (which uses the reference "libjxl" reference library associated with the ISO standard).

For this photo, the JXLs are 2 to 3.2 times larger than the AVIFs and 3.6 to 6.7 times as large as those produced by Imagemagick.

DETAILS:

All the exports were with Prophoto and Bit Depth = 16.

DNG > JXL -- size of .jxl exported from the DNG

TIFF > JXL -- size of .jxl exported from a TIFF losslessly converted from the DNG

DNG > AVIF -- size of .avif exported from the DNG

TIFF > AVIF -- size of .avif exported from a TIFF losslessly converted from the DNG

Magic TIFF > JXL -- size of .jxl converted by Imagemagick from the TIFF 

(*) With Imagemagick, Quality = 100 means "lossless".

Note that Quality isn't directly comparable between formats and converter programs, though it ranges between 1 and 100 for all of them.

Observations:

- JXLs made by Imagemagick from the TIFF are 3.6 to 6.7 times smaller than LR's JXLs.

- AVIFs from the TIFF are 2 to 3.2 times smaller than JXLs.

- JXLs from the TIFF are 1.1 to 1.6 times smaller than from the DNG. I think this is because the DNG contains a Transparency Mask (an alpha channel) added by LR's Photo Merge, whereas that mask gets dropped when the DNG is converted to a TIFF.

[peter_7626]

I've been seeing the same with all of my Nikon raw files (*.NEF) - Jpg-XL exported with lightroom or camera raw results in massive file sizes (compared to jpg) regardless of compression/quality/colour space settings.  At high quality, I'm getting 20mb+ JPG-XL and 8mb equivelent jpgs.  AVIF has no problem, and produces excellent quality images at much smaller file sizes than JPG.

[JtheNinja]

This has nothing to do with my post, I'm talking about the EXPORTED .jxl files, the size of the DNG is not what this discussion is about. Since the buffer is always RGB prior to compression, the fact that the enhanced DNGs are demosaiced is irrelevant.

[John R Ellis]

[This post contains formatting and embedded images that don't appear in email. View the post in your Web browser.]

It's clear LR's JXL export is creating files that are much too large.

I randomly picked three raws from my test library, from a Sony 7M4, Canon R5, and Nikon Z8. I exported 16- and 8-bit JXLs (quality = 70), JPEGs (quality = 70), and AVIFs (10-bit, quality = 70), all Prophoto RGB. I used Imagemagick to create JXLs from the raws and from TIFFs exported from the raws (quality = 90).

As evidence the 16-bit JXLs are too large, consider that they are:

- the same size as the 8-bit JXLs.

- 2.7 - 3.7 times as large as 16-bit JXLs made by Imagemagick.

- 2.3 - 6.4 times as large as 10-bit AVIFs.

- 1.2 - 2.4 times as large as lossy DNGs (linear raws using JPEG XL compression).

- 2.5 - 3.4 times as large as 8-bit JPGs.

The quality values aren't directly comparable between the various formats and tools, but they directionally indicate a problem, especially considering that LR JXLs (quality = 70) are much larger than Imagmack JXLs (quality = 90).

You can download the raws and the output files here:

https://www.dropbox.com/s/vkgtowf0c5d2beh/jxl-size.2024-10-03.zip?dl=0 

The detailed measurements:

[John R Ellis]

@Rikk Flohr: Photography, something is amiss with exporting as JXL. See my previous post for details. Please consider moving to Bugs.

[marc_7730]

Hello,

Always used to export my edited files in Lightroom as 10-bit HEIC files (through plugin) and upload to Synology Photos for quick access. Worked wonderfully.

Until Synology decides to pull the plug on HEIC because of expensive royalties, I now need to find another format. I'm looking at JPEG XL (jxl) and see that Lightroom supports saving in that format. I only see two options though, 8 bit and 16 bit. First exported at lossless 16 bit and had files of 140+mb, then reducing to 90% quality 16 bit dropped it to 40mb. My HEIC files saved as 90% quality were 4-6mb.

How do I ensure so save in a similar quality to the HEIC format while saving on storage, or increasing quality while equaling storage requirement? A little bit weary of going back to 8 bit, so is it the 16-bit option that's making the file sizes really high? Can I drop the quality slider to 70-80% without introducing compression signs? If there was an option for export in 10 or 12 bit I guess that would be perfect, or use a plugin if anyone knows any?

Thanks,

Marc

[John R Ellis]

JPEG XL export is currently broken in LR -- the exported files are much too large:

https://community.adobe.com/t5/lightroom-classic-discussions/certain-raw-files-export-unusually-larg...

Adobe has yet to acknowledge this as a bug.

As my measurements in that post indicate, the issue isn't one of bit width -- 8-bit JXLs are just as oversized as 16-bit JXLs.

Alternatives to consider:

1. Use 10-bit AVIF (if your workflow supports AVIF).

2. Export to 16-bit TIFF, with an Export Post-Process Action that invokes the Imagemagick "magick" utility (free) to convert from TIFF to 16-bit JXL. Imagemagick uses the "libjxl" reference library so should produce reliably good quality results.  This will be slower due to the intermediate step of exporting large TIFFs.  

If you're not adept at command-line scripting, the Run Any Command plugin should make invocation of "magick" in a Post-Process Action easier.

[John R Ellis]

Moderators, @Rikk Flohr: Photography, please merge with this thread:

https://community.adobe.com/t5/lightroom-classic-discussions/certain-raw-files-export-unusually-larg...

[marc_7730]

Thanks for the reply! Ok I guess this would be a valid solution, to use an app/plugin that enables JXL export. Plus I assume that Imagemick utilizes the latest JXL open source version.

Could you point me towards a resource that will help me setup the command line on MacOS?

[John R Ellis]

The "magick" command line is:

magick -quality 90 filename.tif filename.jxl

The Run Any Command plugin has thorough documentation about how to invoke commands.

"I assume that Imagemick utilizes the latest JXL open source version"

It uses the libjxl reference library.

[marc_7730]

Just did all thesen steps, and- Fantastic!

Only issue when comparing tiffs to jxl files is that there is a slight color shift. When comparing in photoshop, the .jxl files seem to be read in Adobe Color profile when openened. While the edited photo is read with Canon Camera Neutral profile. What is going wrong when baking the files? I assume ImageMagick is not applying this profile when converting? As the exported Tiffs look fine.

Sizes are great, as Tiffs are ~255mb, HEIC are ~15mb and JXL are ~7mb for 45mp photos saved in 90% quality

[John R Ellis]

Hmm, when I do a LR export to a 16-bit Prophoto TIFF and then use "magick" to convert that to JXL:

magick file.tif file.jxl

the JXL is in SRGB, not Prophoto, even if I explicitly add the Prophoto profile:

magic file.tif -profile /Library/Application\ Support/Adobe/Color/Profiles/Recommended/ProPhoto.icm file.jxl

This seems like a "magick" bug to me, though googling doesn't reveal anyone else with that issue.

So instead, I exported from LR as 16-bit Prophoto PNG and used the "cjxl" command to convert from PNG to JXL, and it appears to preserve the colorspace correctly. 

"cjxl" is included in the "libjxl" package, which you can install on Mac using Macports.  "cjxl" doesn't appear to understand TIFF format.

[marc_7730]

Thanks for the help, really appreciate it! Actually weird that I am the only one having this issue... As the color shift is quite substantial. Indeed it saves at SRGB even though the export dialog for the Tiff is with AdobeRGB 98.

Is there not a color space insert I can add to the ImageMagick Command line?

Instead of using the lightroom plugin I made a script that runs automatically after export of the Tiffs, can you guide me on how to indicate the cjxl line, replacing the ImageMagick conversion?

See screenshot below:

Thanks, Marc

[John R Ellis]

After you install "cjxl" with Macports, you do the command "man cxl" in Terminal to see its command line options.  It's simplest usage:

cjxl file.png file.jxl

Note that "cjxl" apparently doesn't work on TIFFs (at least the TIFFs exported by LR) -- thus the need to export as 16-bit PNG instead.

Also, you might consider using Prophoto RGB instead of Adobe RGB -- it's a wider color space.

[peter_7626]

This might be fixed in the latest version of camera raw/lightroom.  A quick test gets me JXL files at a similar size to AVIF when at high quality (very high is still a very large file size).

[marc_7730]

Thanks for the heads up. Do you know if they added 10-12 bit output for the JXL? So what you are saying is that a 16 bit JXL at 90 quality will still produce an image as large as a 12 bit Avif in 90% quality?

[Rikk Flohr: Photography]

Setting status

[johnf17598660]

Mine is doing this as well, JPEG is 12 MB but the JXL is ~60 MB, definitely something going on here. I'll be waiting for Adobe to release a statement or patch. Files were from both Canon and Nikon RAW files, not converted to DNG at any point, lossless 16-bit export. 

[drewdawg]

I have this same issue on MacOS. If I open a .NEF (Nikon Raw File) it opens fine with Camera Raw. I can save the image with no issues from Camera Raw into DNG or JPG formats. However, with JPG XL, I have many issues:

1) The shows the same size as the .NEF file, so it has no compression at all.

2) The 'Very High' setting is 2 times LARGER that the .NEF file and has image discoloration and block artifacts.

3) The 'High' setting is about 1/2 the size of the .NEF file and has artifacts and color banding.

4) The 'Medium' setting has edge banding.

And so on.