P: AI Denoise output to TIFF option vs DNG

[mschlotz]

Most have discovered that the resulting DNG file size from AI Denoise is approximately 4 times that of the original raw. ex) Sony A1 50mb photo run thru AI Denoise resturns a ~200b DNG.  This maybe fine for those with only a few photos run thru Denoise but for many it quickly starts to consume a massive amount of drive space.  I tested taking the DNG into Photoshop then saving as TIFF with no compression and the resulting files size was back to 57mb. 

There needs to be an option in AI Denoise to have it internally create its output as a TIFF.

[Rikk Flohr: Photography]

Realize by doing this, you are throwing away data. 

The Enhanced DNG contains your original raw data also so that, if in the future, you wish to run Denoise again with a more advanced model, you can without having to resort to starting over from scratch with your non-enhanced DNG.  You are also 'baking-in' profile, White Balance and a few other settings by doing this. 

[m cs16279208]

Yes, definitely realize these limitations. However, in many cases the resulting tiff is perfectly usable. Regarding future advanced model, I still have the original raw so that's a non-issue (for me).  While some may might balk here, my profile and white balance are set prior to using Denoise. I know this bucks against those down in the weeds for details, but for many, getting the noise reduced on a high ISO shot is all that's necessary. Having a tiff output option makes it more usable for them.

Personally, I shoot a large amount of high ISO (+12,800) photos (sports) and simply cannot afford the space it takes with the DNG's. Having to painfully bring each of them into PS to create a tiff version, followed by importing them into LR and then having to go back and delete all the DNGs is a large waste of time. Adding a tiff option in the LR details section of the develope module would eliminate all of that.

[Jao vdL]

The size increase is inherent to it being a demosaiced image. A tiff is going to be similar size except if you lower the bit depth (which you don't want to do). Your Sony A1 for example is a 50 MP camera. Demosaiced to a 16 bit tiff, that 50 MP image balloons to 300 Mbytes of data (50 times 3 color channels times 2 for the 16 bit depth), which compressed goes to 150 to 200 Mbytes of data just to do a tiff from it. Raw files are so efficient because they only store one color channel per pixel and typically only store 12 bits of data which demosaiced formats can't do. Anytime you demosaic the data, your file size will balloon therefore. The AI Denoise essentially does a demosaic of the raw data and therefore the ~200 Mbyte filesize (especially since it includes the original raw which is weird since you still have that in your catalog but hey) is exactly what you would get if you went to tiff but you retain much better editability.

For you if you want to save some space, you can convert the resulting dng files to lossy compressed dng. This can gain you another factor of 2 with minimal loss of quality.

[mschlotz]

Jao, I do understand the how's and why's of the process Adobe is using. The bottom line is the second paragraph in my last reply above. The space the DNG takes up is too much. The time and effort to get around the issue by manually creating the tiff, having to import it back to LR and then delete the dng is too consuming. An optional process is needed to take the dng and do what save-as tiff does in PS, then send it back to LR. 

[GoldingD]

Another question is if we can strip the RAW out of the resultant DNG, then what size would the DNG be.

And as for future proof, or future mods. No, many of us are perfectly happy to start from scratch. After all, is not the process for most to create the AI Denoise DNG from the unedited RAw, then work on the DNG, not work on the RAW, then create the AI Denoise DNG.

[D Fosse]

@mschlotz  - I still don't understand why you think TIFF is any better? A Denoised DNG and a TIFF will be roughly the same size, simply because they are RGB files. It's going from a single channel at 14 bit depth, to three channels at 16 bit depth, that causes the size increase.

In other words, no advantages, plenty of  disadvantages. A linear DNG still gives you a lot more dynamic range to work with.

And you're not going to like what I say next. Here goes: if you worry about file sizes, you're in the wrong business. There. Image files will eat disk space for breakfast. You have to have expansion plans for what to do when (not if!) your disks fill up. I recently bought two 18 TB disks just to keep up. On the plus side, I don't get disk failure - I replace them before they have time to fail.

[Jao vdL]

@D Fosse @Is exactly right. A tiff is going to be about the same size (typically around 150 MB) and you make some really extreme compromises on quality for basically no gain in file size. Even worse if you want smaller size your only option is to go to 8bit which really drastically lowers the quality and editability.
you really only should be doing this to one or two images out of a shoot. The gain in quality on a 50MP image is only noticeable in prints that are 6 foot wide. This is not something you should do on all your images. Only on the real winners that you are going to print humongously large. There just is no point if what you do is share images online with clients. You're far better off using the standard sharpening and denoising tools. Also as said, storage is trivially cheap nowadays but really you're wasting your time if you run all your images through AI denoise.

[m cs16279208]

@D Fosse I never mention anything about TIFF being better.  Maybe you should re-read my posts.  Also run with a 26TB NAS but still dislike having 200mb files. They may be ok for others but not for me.

[m cs16279208]

@Jao vdL and others: If you take a 50mb RAW Sony file, run it thru AI Denoise, the resulting DNG is over 200mb.  Take that 200mb file into PS and then save as TIFF and the file is reduced to 57mb.  I've done this numerous times with the same results. So the resulting TIFF is definitely NOT "about the same size" as the DNG and it also retains the applied denoise without the bulk. 

I shoot sports professionally and have for many years. The overwhelming bulk of my work is produced under lighting that requies ISO 12,800 and higher. Typical output per match for the client is 150 files. I'm well versed in what my client's expectations are regarding quality. The standard (manual sliders) LR denoise capabilities are IMO marginal at best. There are third party applications that perform significantly better which I have been using.  With the introduction of AI Denoise, LR's denoise quality has greatly improved but at the same time it introduced compromises. The biggest is file size & right behind it is processing time. I won't even get started on the file renaming which Adobe doesn't provide a way to modify, sheesh - really Adobe? This thread was started to point out the need for an option in AI Denoise to output a file that doesn't retain the 3 color channels and eliminates the accompanied RAW. The process I just related does this but it requires multiple steps which can all be done internally if Adobe adds the programing necessary for the option in the AI Denoise panel.

[D Fosse]

@m cs16279208 

I don't know where you get those numbers. Here's a file from an a7r V:

Note that the enhanced DNG is almost exactly 4x the file size - that's R + G + B + original mosaic.

[m cs16279208]

No explanation of how the TIFF was created. 

Take the Enhanced DNG, into Photoshop. Hit File/Save as...  Select TIFF and no compression, the result will be back close to the original raw.  I'm on a Mac using lates PS & LR. 

[m cs16279208]

Ok you started with a DNG.  I'm starting with a RAW file.  xxxx.ARW right out of the Sony camera.

[m cs16279208]

Ok you started with a DNG.  I'm starting with a RAW file.  xxxx.ARW right out of the Sony camera. As far as I know Sony & Canon bodies do not automatically create DNG files. 

[D Fosse]

That's exactly how the TIFF was created:

"Take the Enhanced DNG, into Photoshop. Hit File/Save as... Select TIFF and no compression"

Starting with an .arw doesn't change the relationship between these numbers. Your camera is 50 megapixels, so you start with a 50MB original raw and get a 200MB Enhanced DNG. That's 4x.

My camera is 60 megapixels, so I start with a 72MB original raw and get a 289MB Enhanced DNG. That's 4x.

[m cs16279208]

...and take the 289mb DNG into PS. File/Save as... TIFF, hit ok. Result should be a smaller tif close to the original 72mb raw.  If you don't get that result then something is a miss on your end.   

[D Fosse]

No, the TIFF should not be smaller. These are the native and correct file sizes. Anyone else can confirm that.

Note that the TIFF is identical in size to the PSD: That is the native uncompressed 16 bit size.

In case you're concerned with DNG vs ARW starting file, I repeated with a (different) ARW straight from the camera and got exactly the same numbers - except that the Enhanced DNG is now slightly smaller (but not so much that I would consider it significant (last three entries):

[m cs16279208]

Don't have a clue why this doesn't work for you.  Send me a link to the raw file and I'll try it here.  Wonder if there is something different with the larger Sony raws that stops PS from reducing them after AI Denoise has been applied. 

[D Fosse]

There's nothing that "doesn't work" for me. Eveything behaves just like it should behave. Everything shown above is predictable and fully explained by how these files are built and what they contain.

[m cs16279208]

Ok, glad everything is working for you as you would like it too.  

Have a good one.

[Jao vdL]

@Jao vdL and others: If you take a 50mb RAW Sony file, run it thru AI Denoise, the resulting DNG is over 200mb.  Take that 200mb file into PS and then save as TIFF and the file is reduced to 57mb.  I've done this numerous times with the same results. So the resulting TIFF is definitely NOT "about the same size" as the DNG and it also retains the applied denoise without the bulk. 

You must be lowering the bitdepth to 8 bit and losing tremendous quality when you do this. Perhaps you have set your "edit in Photoshop" preferences to only give you an 8 bits file, otherwise this is impossible. A 50 MP default tif from Photoshop will be about 300 MB without compression and 150 MB with LZW or ZIP compression. There is no way around this but massively cutting the quality down. Even if you were to use 16 bit tif/psd files you would be much better off staying with inear dng as that retains your ability to actually edit the raw for white balance, exposure, etc. 

I shoot sports professionally and have for many years. The overwhelming bulk of my work is produced under lighting that requies ISO 12,800 and higher. Typical output per match for the client is 150 files. I'm well versed in what my client's expectations are regarding quality. The standard (manual sliders) LR denoise capabilities are IMO marginal at best. There are third party applications that perform significantly better which I have been using.  With the introduction of AI Denoise, LR's denoise quality has greatly improved but at the same time it introduced compromises. The biggest is file size & right behind it is processing time. I won't even get started on the file renaming which Adobe doesn't provide a way to modify, sheesh - really Adobe? This thread was started to point out the need for an option in AI Denoise to output a file that doesn't retain the 3 color channels and eliminates the accompanied RAW. The process I just related does this but it requires multiple steps which can all be done internally if Adobe adds the programing necessary for the option in the AI Denoise panel.

Sports with very low light conditions and needed high shutter speed is probably the only situation where I agree that running most of your images through this makes sense and where you might notice the difference even in lower resolution images at web sizes. With this type of photography, you generally will send in a large number of shots indeed and it might be efficient to just run them all through. That is really unique to sports though and is found nowhere else but yeah I feel your pain there. That said, I do think this type of workflow is not really what Lightroom is designed for and you might be better off using Bridge/camera raw and scripting it to do the enhance step, converting to 8-bit tiff and deleting the denoised dng automatically.

I completely agree that we need an option to not include the original raw. Including it is completely useless right now as there is no way to use the embedded raw in dngs anyway so it is just dead weight. I really don't understand the reasoning that it is included so you "can rerun it later if better algorithms are there" while no technical route to actually do this has been provided. That said, even after you remove the embedded raw, you will still end up at 3x the filesize. It is impossible to not include all three channels in every pixel as the demosaic is an integral part of doing the denoise so you will always end up with a 3x increase in file size. That part is unavoidable and exactly what other tools such as Topaz's tools do when working on raw. The denoised dngs that you get from Topaz's PhotoAI/denoiseAI for example are exactly 3x as large as the original raws - because they have been demosaiced!

[mschlotz]

Actually LR settings for edit in PS are set for 16bit. I also have brought up PS directly and brought in the dng. Either way PS shows 16 bit. 

[Jao vdL]

Something is wrong then. A 50 MP 16-bit tiff will clock in at 300 MB uncompressed. More if you include the needed previews. Lossless LZW or ZIP compression can only do about a factor of 2 so ~150 MB is what you expect. No real way around this. If you lower bit resolution to 8 bits then you expect about 75 MB. Here is a quick example for a 45 MP raw file I had laying around on the computer I am on right now to show you what you expect from the basic math:

 You see the 14-bit raw file is about 53.8 MB exactly like you expect for a losslessly compressed mosaiced raw file (uncompressed you expect 45*14/8= 79 MB - lossless compression brings this down to 54MB). A 16-bit tif file saved directly from this with LZW compression is 301.9 MB which is actually bigger than the expected 270 MB which often happens with LZW compression when there is not much to compress and there are previews included in the tiff. This is just opened straight in Photoshop from the raw file and saved as tiff straight from there. No layers, etc. With ZIP compression the same image clocks in at 238.1 MB. Zip compression is usually quite a bit more efficient than LZW on real images but you don't get a factor of two on this particular image which is nothing special.  When you now go to the enhanced dng, it clocks in at 158.5 MB. Smaller than the tiffs created directly from the raw file! dng compression is more efficient than either lzw or zip compression plus the fact that it is denoised makes that the image is now more compressable. Saved as a tiff with zip compression the denoised image is 233.7 MB. Still bigger than the enhanced dng! This is very much expected because dng compression is more efficient than zip. If you downsample this image to 8 bits and save as zip compressed tif, suddenly the filesize drops to 34 MB. This happens because it throws away a ton of info to do this. So really you can only get a smaller file by going to 8 bits.

This is very typical for what I am seeing with most images. The denoised dng files are very efficiently stored even while including the original raw data. Filesizes are smaller than tiffs from the same. Only when you throw away most of the data and go to 8 bits will you gain much space. 

[D Fosse]

Just a small insertion here - LZW was made for 8 bit data, and doesn't work well on 16 bit data, where it will often increase file size.  This isn't from personal experience, just what I've been reading.

[Jao vdL]

Just a small insertion here - LZW was made for 8 bit data, and doesn't work well on 16 bit data, where it will often increase file size.  This isn't from personal experience, just what I've been reading.

That's correct indeed and the reason why I only use zip on my tiffs. It's slower to save and compress out of Photoshop but far more efficient. Just had it here because these are the two options out of Lightroom.

[John R Ellis]

"Sony A1 50mb photo run thru AI Denoise returns a ~200b DNG ... taking the DNG into Photoshop then saving as TIFF with no compression and the resulting files size was back to 57mb."

@mschlotz, to build on the other replies, what you're saying about TIFFs doesn't add up.

1. The A1 has 50 megapixels.  It's impossible for a 50-megapixel image to be saved as an uncompressed 16-bit TIFF taking only 57 MB. As explained by @Jao vdL, the TIFF has three channels (RGB), each channel using 16 bits (2 bytes) per pixel, for a total of 50e6 * 3 * 2 = 300 MB.

2. I ran your experiment on three Sony raws, from an a1, a7 II, and a7 III:

https://www.dropbox.com/s/isb12c6pijb5e4m/denoise-tiff.2023.07.06.zip?dl=0

For each, I ran Denoise producing a DNG, then edited the DNG in Photoshop and saved as a 16-bit TIFF with zip compression:

In all three examples, the TIFF was somewhat larger than the DNG (1.1x to 1.5x), even though the TIFF has three 16-bit channels and the DNG four (RGB plus mosaic data).  I think this is because the compression used in DNGs is superior to TIFF zip compression, but I can't find an authoritative reference for that right now.

3. Three very knowledgeable people all agree that saving the output from Denoise as a TIFF won't save any disk space, but you're insisting otherwise. To make progress on understanding what you're observing, please upload a sample ARW to Dropbox, Google Drive, or similar and post the sharing link here.