Lossy DNG shows strong visible differences but JPG not, is this a bug?

[biriem]

Hello,

I tried out the lossy DNG format with a lot of pictures.

For most pictures I can't find any visible differences.

But I found also some pictures with very noticible visible differences between lossless and lossy DNG.

It seems that the lossy DNG format has problems with dark pictures with small light sources for example

the stars in the dark sky or the red lights of an aeroplane in the dark sky.

I have an example where you can see red security lights of a great antenna on top of a mountain.

Surprisingly with this extreme example the JPG versions looks much better as the lossy DNG version of my source lossless DNG.

Here is the cropped JPG from the lossless DNG (file size 10877KB):

Between this JPG and the lossless DNG file is no visible difference.

And here a JPG from the lossy DNG (file size 13261KB):

Between this JPG and the source lossy DNG file is no visible difference.

The source of both DNG files was a NEF File from a Nikon D7100.

Also interessting is, that for this excample the file size of the lossy DNG is  greater than the lossless DNG. This seems crasy IMHO.

I was very surprised, that JPG result is is much better than the lossy DNG for this case.

Perhaps there is a hidden bug in the lossy DNG conversion process.

I hope Adobe can look in to this. If you need the original NEF file let me know.

[biriem]

Today I tried out Lightroom 5 beta.

I wanted to know, whether the above behaviour of the DNG lossy conversion is still there.

Unfortunately it is.

[areohbee]

biriem wrote:

Today I tried out Lightroom 5 beta.

I wanted to know, whether the above behaviour of the DNG lossy conversion is still there.

Unfortunately it is.

I thought you were referring to smart previews. Smart previews are just vanilla-flavored lossyDNGs so whatever happens with them would happen to those like them, and vice versa (theoretically).

Anyway, I did confirm some differences in exported lossy DNGs (by evaluating exported smart previews). Inside Lightroom's develop module they look almost exactly like the original raw, but when exported, things get a little wonked for some reason.

I haven't tried exporting the smart preview of your pic yet but I will.

Do standby...

Rob

[biriem]

I seems this thread becomes an one man show.

I don't know what is wrong with my article.

Perhaps this issue is not of any interest for the other members and also for Adobe.

I hoped that I can use lossy DNG, because I found for most of my pictures no visible differences between lossy or lossless DNG.

The results with lossy DNG were excellent except the problem I decribed here.

Unfortuately this is a showstopper for me and I will stop using lossy DNG, because I can't rely on it.

I must be sure, that for all kind of pictures the visible differences are not noticable within practicable range.

Unfortunately this is currently not the case.

very very sad

[JP Hess]

I thought about responding the other day, but decided my remarks wouldn't help you. And they still won't because I'm going to give you my personal opinion about lossy DNG. I don't think it is intended for your master "important" images. I watched a video on Adobe.tv about this, and the example that was given was, suppose you have taken pictures at a wedding. Perhaps you have 2000-3000 images, 500 of which are keepers that are really good. You don't want to get rid of the other bunch but want to save space. So those would be the images that you would convert to lossy DNG. I mean, let's be realistic about this. You can't throw away data and hope to still have full raw quality. That isn't the idea behind the format.

Personally, I think converting to DNG of any sort as a workflow routine is frivolous and unwise. I know a lot of people routinely download and convert, and then they don't have copies of their original raw images. Yes, I know, the DNG files contain all the original raw image data. But not all software will read those DNG files. For example, I use Nikon cameras. The cameras shipped with ViewNX 2. It's not a very useful program, but has a couple of features that I like to use once in a while. But it won't open the DNG files. And if I have converted and then discarded the original NEF files, I can't use the software. With my cameras there is only a 15-20% savings in space. The tutorial I watched indicated that the lossy DNG files would load in Lightroom up to 8 times faster. I haven't used a stopwatch, but if anything the DNG files take longer to load than the original NEF files.

The only benefit I see to converting to DNG is if I have a file from a camera for which I don't have native support because my software is older or not up-to-date. I know this hasn't helped you. But maybe it will stimulate some conversation and you'll get a more enlightening opinion from someone else.

[areohbee]

JimHess wrote:

The tutorial I watched indicated that the lossy DNG files would load in Lightroom up to 8 times faster. I haven't used a stopwatch, but if anything the DNG files take longer to load than the original NEF files.

Performance improvement (for images which are raw, or lossy-DNG being handled like raw) depends more on resolution reduction than compression scheme / file format.

[JP Hess]

I have been thinking about your answer, Rob. Please explain a little more in detail. How do you reduce the resolution? I'm not quite grasping your answer.

[areohbee]

Fast performance: full-rez rgb (e.g. jpeg, tiff, or dng-wrapped version of same)

Fastest performance: reduced-rez rgb.

Slowest performance: full-rez raw (whether dng or proprietary, whether lossless or lossily compressed)

Medium performance: reduced-rez "raw" (jpeg compression, but image data handled like raw - must be lossy DNG).

Smart previews are 2540 pixels long edge (lossy DNG), that's the main reason it's faster to edit them than their full-rez raw counterpart.

PreviewExporter can reduce rez further for increased performance.

You can reduce rez of any file by exporting it with smaller dimensions.

You can also reduce rez of  (supported) raw (or dng, whether raw or not) files and store as lossy DNG using Adobe DNG Converter (gui or command-line)

I use command line option "-side {long-edge-dimension}" to do it.

This option is not documented, but should be - and I pray one day it will be...

To be clear: DNG Converter can not dng-wrap an rgb file (in jpeg/tiff... format I mean, that any normal users know of), whether full-size or reduced-rez, although such feature has been requested - please include in your prayers... (granted if rgb file is already dng-wrapped, then it can be resized - yeah: it's a glaring omission for some of us...). Exporting via Lightroom is the only way I know of.

Did that make things clearer, or more confusing?

Rob

[JP Hess]

Yep, it sure did. Thanks for the explanation.

[areohbee]

Yeah, it's a common misconception that DNG is just another raw format (I'm not saying you had such a misconception), but like NEF, it can also wrap non-raw (what I call rgb) images. Many people don't know that about NEF either - i.e. NEF does not necessarily imply raw.

However, *unlike* NEF, the new lossy DNG format can store a highly compressed (lossy) version of "raw" image data (in any resolution) - this is my primary use for DNG, since I prefer original raws in, well, original format, and although there are some benefits (see note below) to wrapping (master) jpegs in dng (and I would like to do so), it doesn't fit well into my workflow yet,  because of the aforementioned hole (glaring omission) in the command-line DNG converter.

note: benefits of wrapping jpegs in dng:

* @Lr5, dng-wrapped jpegs also enjoy corruption detection via check-codes.

* It makes it clear that the photo is a master (unbaked) image, not fit for consumption without rendering / export...

drawbacks of wrapping jpegs in dng:

* They can no longer be viewed / edited in non-compatible software (without first unwrapping).

Cheers,

Rob

[biriem]

JimHess wrote:

I thought about responding the other day, but decided my remarks wouldn't help you. And they still won't because I'm going to give you my personal opinion about lossy DNG. I don't think it is intended for your master "important" images. I watched a video on Adobe.tv about this, and the example that was given was, suppose you have taken pictures at a wedding. Perhaps you have 2000-3000 images, 500 of which are keepers that are really good. You don't want to get rid of the other bunch but want to save space. So those would be the images that you would convert to lossy DNG. I mean, let's be realistic about this. You can't throw away data and hope to still have full raw quality. That isn't the idea behind the format.

Personally, I think converting to DNG of any sort as a workflow routine is frivolous and unwise. I know a lot of people routinely download and convert, and then they don't have copies of their original raw images. Yes, I know, the DNG files contain all the original raw image data. But not all software will read those DNG files. For example, I use Nikon cameras. The cameras shipped with ViewNX 2. It's not a very useful program, but has a couple of features that I like to use once in a while. But it won't open the DNG files. And if I have converted and then discarded the original NEF files, I can't use the software. With my cameras there is only a 15-20% savings in space. The tutorial I watched indicated that the lossy DNG files would load in Lightroom up to 8 times faster. I haven't used a stopwatch, but if anything the DNG files take longer to load than the original NEF files.

The only benefit I see to converting to DNG is if I have a file from a camera for which I don't have native support because my software is older or not up-to-date. I know this hasn't helped you. But maybe it will stimulate some conversation and you'll get a more enlightening opinion from someone else.

First of all I want to use lossy DNG because I want save space.

Second, I have no extra XMP-File for every image.

If I can't see any visible difference between a  lossy DNG and a lossless DNG, then I have no problems with losing some data.

[areohbee]

biriem wrote:

I seems this thread becomes an one man show.

I haven't noticed any unexpected / unreasonable (e.g. highlight) differences yet. If I do, I'll let y'all know...

I have noticed (expected and reasonable) differences in dark shadows when cranking exposure up to 5.

PS - Lr5's smart previews are just vanilla-flavored lossy DNG files (reduced rez), so there will be reports trickling in (assuming this is not an isolated phenomenon) as people gain experience when putting photos online that were edited offline based on smart preview.

~R.

[biriem]

Rob Cole wrote:

I haven't noticed any unexpected / unreasonable (e.g. highlight) differences yet. If I do, I'll let y'all know...

I had the same experience until I converted the image above.

For me the differences are very noticeable and that the file size of the lossy DNG is greater as the lossless DNG is unexpected.

Rob Cole wrote:

I have noticed (expected and reasonable) differences in dark shadows when cranking exposure up to 5.

Yes, same here and I thought that thes extreme exposure boosts have no practicable advantage for me.

So I thought that if lossy DNG would show no visible differences for exposure increments up to +2 this is enough for me and lossy DNG could be a great method to save space.

But in my example image above the exposure is untouched.

[areohbee]

biriem wrote:

For me the differences are very noticeable and that the file size of the lossy DNG is greater as the lossless DNG is unexpected.

If lossy file-size is greater than lossless, something is definitely wrong - perhaps this is a clue to your woes (are you sure you are converting raw files to lossy DNG?).

Consider reviewing details of what you are converting and how...

Also consider posting sample file(s), so others can confirm or deny symptoms.

Really, highlights have been virtually spot-on, in every case I've tested, and file size of lossy-DNG is *always* smaller than it's (lossless) raw counterpart.

Rob

[biriem]

Rob Cole wrote:

If lossy file-size is greater than lossless, something is definitely wrong - perhaps this is a clue to your woes (are you sure you are converting raw files to lossy DNG?).

Consider reviewing details of what you are converting and how...

Also consider posting sample file(s), so others can confirm or deny symptoms.

Here is the original NEF-File: https://dl.dropboxusercontent.com/u/18702754/S71_0985.NEF

You can try conversion for yourself. I have used the current version of DNG-converter V7.4.

Bill

[areohbee]

The original raw and the smart preview look "exactly" the same to me - I cannot confirm the difference you are seeing.

Related thread: http://forums.adobe.com/message/5354405#5354405

[biriem]

Rob Cole wrote:

The original raw and the smart preview look "exactly" the same to me - I cannot confirm the difference you are seeing.

Related thread: http://forums.adobe.com/message/5354405#5354405

Rob,

I think you have written your answer in the wrong thread. Is this possible?

"smart prewiew look" was not my theme.

Bill

[biriem]

Hello, may be I am wrong, but this issue with lossy DNG,s seems a realy great problem for Adobe.

I have no other explanation for this silence here.

I had also send a private message to one of the developers and I got only silence.

Not even the smallest response at least out of courtesy.

[areohbee]

biriem wrote:

I had also send a private message to one of the developers and I got only silence.

If that one was Eric Chan, I'm pretty sure he does not look at nor respond to PM's - I think that is what he said.

[Daniel_Tejada]

Let me share my simple workflow in order to get the best of both worlds. First I import the raw files as lossless DNG. Then I develop them as usual and export them to high quality jpg so that I can print, share, save, etc. After a couple of months have passed, and I am pretty sure I will not have to modify the lossless DNG files again, I convert them to lossy DNG using lightroom. Chances are I will never require using those DNG files again, but just in case, I am keeping a raw copy of them that will also take much less disk space than the original raw files. My jpg files are obtained from the original lossless DNG files, so I assure I don't have any quality degradation there due to the lossy DNG format. The great thing is that lightroom keeps the original develop settings from the lossless DNG files, so in lightroom I can still see my developed files but now they are taking much less disk space.

Hope this can help someone.

[Duarte_Bruno]

IIRC, lossy DNG is 8bit. Adaptable 8bit but still 8bit.
Because it's adaptable, it can work great 99% of the times in highlight zones and in shadow zones, but probably not in zones where there are both highlights and shadows.
It's also quite possible that I've just read the Wikipedia page for DNG and I'm just talking out of my arse.

Cheers!

[Henrik Helmers]

Duarte Bruno has it right. When converting to lossy DNG the image gets squeezed to fit within 8 bits. In most cases this is unproblematic. The image in this example has very deep blacks and bright highlights, and the conversion process does not handle it well.

It might be that this can be tweaked in the converter—or maybe this would require a new format (i.e. lossy DNG v2).

I am extremely satisfied with the lossy DNG format (and DNG in general), and hope Adobe will continue improving on it.

[fennworld]

It is truly discouraging that an issue like this remains ignored after years of waiting.

[JP Hess]

I don't know that it has been ignored. If you want to work with lossy DNG thing you have to be willing to work within its limitations. If you don't like the limitations then don't use the lossy option. It's pretty straightforward, in my opinion. After all, it is LOSSY.

[fennworld]

The problem is the limitations aren't definable because there is inconsistency that hasn't been addressed. No one would argue that you compress to the level you're comfortable with, and everyone has the right to determine their own size/quality threshold, but anomalies like this make it impossible to embrace a compression that may in fact be a better fit for most photographers.