Enhanced DNG - WHY so large?

[Camera Raw Fan]

I keep getting this question, and I have read through Adobe's literature, but I cannot seem to get a clear answer on this question:

Why are Adobe Enhanced DNG raw files so large in comparison to the native raw file, or a standard DNG raw file?

I am thinking that the Adobe Enhanced raw file is packaged also with the enhanced raw data as well. And when the image is actually rasterized, then it is applied? And even before that, it is applied to the preview that we see in Lightroom when we are making adjustments.

Thank you for as much detail as possible so I can help others understand.

[SeanMcCormack]

If you're checking Super Resolution, you're increasing your fine pixel dimensions by a factor of two in each direction. If you're just using Raw Details, it's potentially that your Raw is compressed and the new file has more detail, simply making it it larger due to that. 

[Camera Raw Fan]

Sorry about not being more clear. I am only talking about "Raw Details" or what we used to call "Enhanced Details" as I never use "Super Resolution." 

[DdeGannes]

See the screen capture when you use the "Enhance" feature in LrC with a RAW image file. As you can see from the info in the dialog the is going to be enhanced to produce more detail during this process the number pixels in the hight and width will be doubled, if Super Resolution is chosen, which means the total pixels will be quodrupled. If the raw file is 24 mega pixels the enhanced DNG will have 96 mega pixels.

[Camera Raw Fan]

I am only talking about "Raw Details" or what we used to call "Enhanced Details" as I never use "Super Resolution." 

[Jim Wilde]

Even without using the Super Resolution option, the Enhance function generates a Linear DNG, which is no longer a raw file and contains demosaiced data. As such it's akin to a Tiff file, with at least 3 times the amount of data of the original raw file. If you then add Super Resolution into the mix, you'll see another 3-4 times increase in the file size. On my system, a 36MB raw file increased to 146MB when using just Enhance, and then to 436MB when I used the Super Resolution option.

All as expected.

[Camera Raw Fan]

Sorry for not being more clear, but I do not use "super resolution." Your answer here is getting to the root of my question.

So my next question is: is a "raw details" image still raw?

Is it still "lossless" (parametric edits vs pixel edits) in the way that we consider raw files lossless?   

It does not at all seem to be rasterized.  Further adjustments on it in Lightroom/Camera Raw do not react like a typical rasterized/pixel editing Tiff or other rasterized file format would merely using the Camera Raw Algorithms, or the Camera Raw Filter on a pixel image...

This leads me to think that the demosaiced information in a "raw details" DNG ("Enhanced") is stored along with the original raw data, and that it remains lossless/raw information until the image actually gets flattened (or rasterized). 

Side Note: I have read the Digital Negative (DNG) Specification (PDF) about it.  It might just be me, but simply speaking, it seems a little ambiguous whether "raw details" is still raw?

Thank you ahead of time for taking the time to reply!  👍🏼

[Ian Lyons]

> This leads me to think that the demosaiced information in a "raw details" DNG ("Enhanced") is stored along with the original raw data, and that it remains lossless/raw information until the image actually gets flattened (or rasterized). 

The demosaiced information (i.e. the original raw file) does not appear to embedded in the 'Enhanced-DNG' file. If it is, then it would be easy to extract same using the DNG Converter 'Extract' option, and having just tried it, I get the following.

[JohanElzenga]

Like Jim already explained, the Enhanced image is not raw anymore. It is linear RGB. This has no influence on how images are edited in Lightroom, however. Lightroom is a parametric editor. Every image, not just raw images but even a jpeg image, is editted in metadata.

[Camera Raw Fan]

---

@JohanElzenga wrote:

Like Jim already explained, the Enhanced image is not raw anymore. It is linear RGB. This has no influence on how images are edited in Lightroom, however. Lightroom is a parametric editor. Every image, not just raw images but even a jpeg image, is editted in metadata.

---

thank you for your feedback here.   In my understanding this does not equate to lossless editing. The meta-data is a set of instructions to apply separate lossy rasterized adjustments to the jpeg or image.

That is unless the file is raw, and is being adjusted in Camera Raw. 

[Jao vdL]

The whole point of enhance details is to better demosaic the image so yeah an enhanced dng is no longer mosaiced. It is however, still linear response data that still has to be rendered through the profile and such so it is still very much "raw" data but it is no longer data direct from the camera sensor as it has been demosaiced by the AI algorithm in enhance raw details. It is however still in scene referred or camera color space - i.e. not rendered into a display color space etc. which is what a tiff, jpeg, etc. is. So you could still call this a raw file but it really depends on how much of a stickler to definitions you are.

Of course if do this, there is no way that the file is still as small. i.e. if you start with a 50 MP raw image in 12 bits raw, you have about 75MB of data (50*12/8 to store all the bits). Compressed using the cameras compression (which is not as good as dng's compression) you get about 40 to 50MB. Now demosaic this and you have 50*3*12/8= 225 MB of data (I am using base 10 numbers here for simplicity - note that computers generally use base 2 for filesize display but the difference is small enough to not really matter). It is still in 12 bits which is why you have the 12/8 factor, but you now have full 12 bit r,g,b triplets at each pixel instead of just one of the three. Now do a resolution enhance and you get 225*4=900MB of data. Compressed with dng compression which does about a factor of 2 for lossless this yields 450MB expected for the new enhanced DNG. So you should expect around a factor of 10 increase in filesize for the enhance details operation. Nothing you can really do about this. It is a direct consequence of what you are asking the computer to do.

[Todd Shaner]

So you should expect around a factor of 10 increase in filesize for the enhance details operation. Nothing you can really do about this. It is a direct consequence of what you are asking the computer to do.

By @Jao vdL

Actually you do have an option that will reduce the Enhaced DNG file size and while retaining most of its benefits. Export the file to Lossy DNG file format and it will reduce the Enhanced DNG file size to about 1/10th its size. I use Lossy DNG file format for large panorama and HDR DNG files and 99% of them show no visual difference when compared at 100% Zoom view. As long as you keep the original raw file you are future proofed if you later find an issue with the Lossy DNG rendering or the Enhanced Details tool is updated. It's a bit of a technical hypocrisy, but if the visual rendering is the same you can save a heck of a lot of disk space!

Here's one test example.

http://www.visualbakery.com/Tools/DNG_Lossy_Vs_Lossless.aspx

[Jao vdL]

Yeah the lossy compression available in dng is really good. You really have to be pushing development hard to see it break down. You have to go through a few extra steps but it is a really good option indeed while still retaining almost the same editing lattitude and final image quality.

[Camera Raw Fan]

Kind of a bummer because I and others really push images hard these days.  

This must be the same case with the DXO software that Adobe works with. Where we can allow the DXO PURE RAW Ai software to apply there a special AI noise reduction, but then allow us to open up the DNG again in Camera Raw and it appears as what seems like a RAW file.

Now I'm thinking that it's not.

[TheDigitalDog]

Linear DNG isn't really (fully) raw but it's a bit more 'raw' than a fully rendered image in a few very useful ways:

http://www.barrypearson.co.uk/articles/dng/linear.htm

A linear DNG is still scene-referred:

http://www.color.org/ICC_white_paper_20_Digital_photography_color_management_basics.pdf

[Camera Raw Fan]

Because "enhance – raw details "renders a demosaiced rasterized image (here I think Adobe is a little bit deceptive because most people think it's still truly raw ).

Is there any benefit to making all of the light room develop (Camera Raw) or Camera Raw parametric adjustments first before choosing Enhance -  Raw Details?

Hey sincere thank you again.

[TheDigitalDog]

I have no idea what most people think, the entire DNG methodology is fully documented even if many don't pay attention to this.

[Camera Raw Fan]

That is a great ;point. Thank you. They can dig and read about it. 

[Jao vdL]

>Is there any benefit to making all of the light room develop (Camera Raw) or Camera Raw parametric adjustments first before choosing Enhance -  Raw Details?

No it makes absolutely no difference. The enhance details algorithm works on the raw data and any existing edits are simply copied over in xmp metadata to the newly created dng but have no impact on the enhance details results. They simply set the sliders on the newly created dng that comes in. 

[Camera Raw Fan]

Thank you very much. I very much appreciate you having the patienc in taking time to help me understand!  🙏🏼 😊

Sorry to be a bit difficult but I would appreciate it very much if you could help to see if I am clear here.

In an above replies you said of the enhance - raw details option:

"It is still linear response data that still has to be rendered through the profile and such, so it is still very much "raw" data but it is no longer data direct from the camera sensor as it has been demosaiced by the AI algorithm in enhance raw details."

"The enhance details algorithm works on the raw data and any existing edits are simply copied over in xmp metadata to the newly created dng but have no impact on the enhance details results. They simply set the sliders on the newly created dng that comes in."

So, would it be correct to say it this way:

1. The democaicing, the profiling (such as choosing a linear profile made with the Adobe DNG Profile Editor, or even choosing an Adobe profile such as "Adobe Color"...) and the AI enhance Raw Details, is done based on linear response data direct from the camera sensor. Full access to that lossless data in our rendering choices. 

2. This is the part I know I am still a bit confused about:

Is it true that the LR Develop/Camera Raw edits/adjustments (on the enhanced raw details file) are not done based on the direct data from the camera sensor? Is there a loss of attachment there? The Demosaicing?

As an example, if I made a second version of the same enhanced raw details file to, lets say, increase the shadow luminance significantly, by using the black end of the point curve in LR/CR...  Am I still getting all the data for the shadows (drawing from the data that the sensor captured) in the same way as if I had not chosen "enhance - raw details"?

Lastly, if I instead used a standard Raw DNG, not enhanced -raw details (or even a non DNG proprietary company Raw File for that matter...) would I get all the same data for the shadow luminance adjustment (from the data that the sensor captured)?

Again, thank you for helping me to understand and I appologize if it takes me some time to process/understand. And thank you for your patience if I am misunderstanding still. 

[Jao vdL]

---

Is it true that the LR Develop/Camera Raw edits/adjustments (on the enhanced raw details file) are not done based on the direct data from the camera sensor? Is there a loss of attachment there? The Demosaicing?

For clarity I'll ignore the AI powered superscaling that is also a part of "enhance details" but that doesn't actually change any of the below

The only difference between the linear dng data and the original raw is the fact that the liner dng is demosaiced. Basically the raw data in the original file is still in a Bayer mosaic, which means that each pixel ONLY has a value for the green, blue, or red channel. These values are in a linear scale and basically counted the number of photons that arrived at the pixel location after passing through the pixel's individual color filter. What happens during demosaicing (and enhance detail is a AI powered version of demosaicing) is that instead of only one color value per pixel value, the algorithm interpolates the other color values at the pixel location. So if your original pixel only saw green light, the algorithm looks at the surrounding red and blue pixels and estimates what the correct red and blue values at the green pixel should be. The result is a r,g,b triplet of values that are all in linear space. 

For normal raw files, this process happens on the fly while you are editing using a fast (but not perfect) algorithm. All edits are applied after this process. For enhanced dng files, this process happens only once and now you have a new demosaiced source image that is still linear data but has three values at each pixel instead of just one. This circumvents the normal on-the-fly demosaicing but still presents a similar source to the editing engine.

As an example, if I made a second version of the same enhanced raw details file to, lets say, increase the shadow luminance significantly, by using the black end of the point curve in LR/CR...  Am I still getting all the data for the shadows (drawing from the data that the sensor captured) in the same way as if I had not chosen "enhance - raw details"?

---

Yes. It does not matter. You still get all the data from the sensor. It just skips one step it would do normally. This step was already done by you choosing "enhance details".

Lastly, if I instead used a standard Raw DNG, not enhanced -raw details (or even a non DNG proprietary company Raw File for that matter...) would I get all the same data for the shadow luminance adjustment (from the data that the sensor captured)?

Yes, all these things preserve the essence of the raw file. For a normal dng file, all that happens is that the raw sensor data is copied into the dng file. It is exactly the same data as in the original company proprietary file. All that happens is that the dng file uses a slightly better but still lossless compression algorithm. The data is identical! For an enhanced dng, all that happens is that the demosaic is done already, so the raw engine doesn't have to do this very first step anymore. So it fundamentally is the same thing. Just one small step into the chain.

[Yona24866456x8o1]

There's an easy way to apply the lossy DNG compression to images, which will greatly reduce the size of enhanced images as well as other uncompressed raws at the cost of a slight quality loss:

1. Select your enhanced images (I text filter for "enhanced"):
2. Libary | Convert Photos to DNG
3. Uncheck "Only convert Raw files"
4. Check "Delete originals after successful conversion"
5. Check "Use Lossy Compression"
6. Click OK

For me, files often go from 100-200MB to 20-30MB, almost a 10x size reduction, at no preceptible quality loss.