Why are exported TIF files so large?

[jj_345]

I'm using lightroom to make some simple adjustments to images from my Sony A7 RIII.  File sizes of raw files are 43MB.

When I export the full size pictures as ZIP compressed TIF files, they end up being about 200MB...  Any idea why that is and how to avoid such large files?  The tif files are just one layer...

[D Fosse]

The raw file is grayscale - one channel - at 14 bit depth. A TIFF is RGB - three channels - at 16 bit depth.

The TIFF isn't "large", it's what a 7952 x 5304 pixel RGB image weighs in at. That's the native uncompressed size.

The jpeg format goes the other way. It aggressively compresses the file by chunking up data and throwing away a lot of information in the process. The net result can be 2-5% of the TIFF file size - but the original quality is irretrievably lost and for this reason jpeg should never be used as a working format. The file deteriorates with every resave and eventually disintegrates completely.

[jj_345]

Thanks!  I guess I wish there was a single format that stored the RAW files AND the lightroom edits (not in catalogue) to easily share while collaborating. 

[D Fosse]

There is: DNG.

Press ctrl+S to save the Lightroom edits to the file header (as well as the catalog).

[Just Shoot Me]

You can also enable the option to Automatically write changes into XMP and then when sending files to whomever you are collaborating with send the XMP along with the actual file.

But for any of the edits to be "VIEWED" with the RAW file those collaborators must be using either LR or ACR of nearly or the same version of LR/ACR or ACR stand alone that made the edits.

The same goes for using DNG. Only an Adobe image editor can read the edits made by LR or ACR.

No other image viewer or editor can read the edits applied by LR or ACR.

Not so with TIFF. All the edits are Burnt into the TIFF file.

One other reason the TIFF files are so big is because there is actually 2 copies of the image data in the TIFF file. One is for edits made in a program like PS, Layers and such, and the other is a Flattened version that can be Viewed by most any Image Viewing program. So there is always 2 copies of the image data in a TIFF file regardless if any edits have been done in programs like PS.

[jj_345]

Thanks!

A few questions:

1. How do I automatically save the XMP?

2. If I understood you correctly, I can send the XMP and original raw files?

3. I'd prefer to send one file vs two (easier not to forget!)Ican I simply export the DNG or do I need to manually save the XMP too?

4.  What options should I use in file settings (size/fast load data/compression/Embed original file).  I'd prefer not to loose any quality...

[Just Shoot Me]

1. That is an option toggle in the LR Catalog Settings.

2. Yes the XMP is a FILE that gets created in the same folder as the original RAW and has the same name as the original RAW (and or the same name as the RAW file when the XMP was created. Like if you rename on import the XMP sidecar file would have that same name and a .xmp extension.

3. With DNG, TIFF, JPG and PSD (and possibly PNG) files if you have the option to Auto write changes into XMP turned on then the edits you do get saved to a special section of those files.

But I am unsure if you Export as DNG whether LR actually write the edits into the exported DNG file. It may just Convert the original RAW to a DNG and NOT include the edits you have made. Do a test to see if they are included.

I just did one and it seems the edits are included but as stated before those edits can only be read by either LR/ACR or Stand alone ACR. No other image editor or viewer will see them.

4. If you are asking about the options for exporting a DNG, Fast load and nothing else if you want to keep the file size lower. No compression as it is LOSSY (you loose data).

[D Fosse]

I don't see any particular reason to export to DNG. Just convert it to DNG, ctrl+S, done.

Ctrl+S is the shortcut for the menu item Metadata > Save Metadata to File. This saves all edits in the file header for DNG, or as a separate sidecar file for proprietary raw files.

All that said, I'd just send TIFF. With a file transfer service such as wetransfer.com file size is not a concern. This will in any case be much too large for mail, whether raw or RGB.

[DdeGannes]

Quote “Thanks!  I guess I wish there was a single format that stored the RAW files AND the lightroom edits (not in catalogue) to easily share while collaborating”

Adobe  has two applications that process raw file data from digital cameras

a. Adobe Camera Raw, which is a plug-in which requires Adobe Photoshop to utilize the plug-in as the host application. Camera raw will process the raw data and provides many editing features without making any changes to the raw data and stores the edit info in a Sidecar file in the same folder with the same name as the raw file with a .xmp suffix. There is also an option to convert your raw file to a dng format provideo by Adobe which allows the raw data and the Xmp data in a single file with the .xmp suffix.

b. Lightroom a.k.a. Adobe Photoshop Lightroom which also provides the same editing features as Adobe Camera Raw and a number of additional management, sharing, printing etc. Lightroom by default unlike Adobe Camera Raw does not store the data about the editing etc in a sidecar file. There is also the option to have Lightroom store info to a sidecar file like ACR which allows Photoshop / ACR to apply the edits done in LR. This is the basic difference between the two applications which you need to be aware off Is you work with both applications. Also Lightroom is a completely stand alone application and does not require that you have any other Adobe application installed on your computer for it to function.

Having LR write info to .xmp files does not stop LR fro utilizing the Catalog to store all the info. When you are utilizing LR the info is in the Catalog. When you are working with PS / ACR the info is in the file and it cannot read the info in the Catalog.

[elie_dinur]

>> "I guess I wish there was a single format that stored the RAW files AND the lightroom edits (not in catalogue) to easily share while collaborating."

Depends what you mean by "collaborating.". If the recipient will be revising or adding to your edits using Adobe software, then DNG. Otherwise, Tiff. And if your version is the final version and in Adobe RGB or sRGB space, it can be 8 bit/channel (half the file size).

[Michael Prais]

I was asking this same question recently when I noticed that a 27MB image (from an Nikon D610 6016x4016) ballooned to 106MB when I chose to Edit in Photoshop from Lightroom. The replies here are the closest that I came to an explanation, but they really don't answer the above question. (I also have the same problem with my Sony A7 RIII.)

I noticed the problem when I started to run out of storage on my SSD. I am compositing images, and one of my .tif images grew with Layers to over 1GB. I expect that Exporting to .tif produces the same problem.

Looking into the image file tags with ExifTool and MATLAB and comparing them with the Adobe TIFF 6.0 and DNG 1.5 specifications and the ISO TIFF/EP specifications, my .tif files match the TIFF 6.0 spec from 1992: There is a short header, 106MB of pixel data, and a series of TIFF tags in a small Image File Directory. The 106MB of data appears to be a compressed version of 5824 cols x 3888 rows x 3 colors/pixel x 2 bytes/color = 135MB. The ZIP compression of runs of RGBRGBRGB data is not that efficient (135MB --> 106MB) because while a run of RRR or GGG or BBB are usually similar in values, the R, G, and B values in most any pixel are not. Camera RAW and DNG files separate the R, the two G, and the B values measured by the Color Filter Arrays and ZIP compress (Deflate) the "channels" separately and more effectively and efficiently. Camera RAW and DNG files use the more recent TIFF/EP specification (from 2013 and 2019) with its support for CFA image data. (To be precise, the .tif files created by Lr and ACR are created under the Profile 1 of TIFF/EP, which is compatible with TIFF 6.0 but includes ZIP compression. Camera RAW and .dng files are created under the more advanced Profile 2 of TIFF/EP.)

Lr and ACR (and GIMP) read and write Camera RAW and DNG image files. They can "adjust" the individual pixels, but they cannot replace the pixels the way that Ps and other image editors can. PS does not save back to Camera RAW or DNG formats, but you could try saving as a PSD file for effective compression. (I have not tried this.) Ps or other image editors, viewers, and printers, however, require images formated in other ways. A TIFF 6.0 (.tif) file is perhaps the most common pixel (raster) image format.

So, we're stuck creating large TIFF files from small Camera RAW or DNG image files when we want to use them in unspecified image editors and applications. You can Flatten your images to remove the layers (each Layer being the same size as the Background Layer (image)). You can also try to stay within Lr or ACR saving as DNG (as I am) and encourage your destination to use these Adobe products (or GIMP). Finally, you could reduce the size and perhaps the apparent quality of your images by saving to JPEG with Quality at 100 allowing for any size. (Each Save throws away some of the pixel information.)

BTW, TIFF/EP (Camera RAW and DNG) has a place in the file structure for XMP (editing commands). TIFF 6.0 can be extended to hold XMP as a Tag. Cmd-/Crtl-S updates the XMP information in the open image file. I expect that Ps/ACR and Lr read either the included or the accompanying XMP data when they open either .tif or .dng files (as long as you have used Smart Objects in Ps for non-destructive edits, otherwise, it makes no sense to track the changes that have permanently changed the original file).

[D Fosse]

There is no "problem" here. The difference is three channels at 16 bit depth, vs one single channel at 14 bit depth.

A raw file from a Sony a7riii is around 42 MB.

A 16 bit uncompressed PSD or TIFF from that raw file is around 247 MB.

That's just how this works and it has nothing to do with Photoshop specifically. If you're going to process files from a 42MP camera, this is what you get.

[Michael Prais]

The "problem" is the unexpected, large increase in the file size.

Your reply iis the closest that I have come to an explanation, but it is a bit terse and the part about a raw file being a one-channel grayscale is not the story.

First, two disclaimers:

I don't have a 42MB Sony A7RIII--I have a 24MB Sony A7III. The 5824 col x 3888 row cropped image that I discuss comes from that camera.

I was making an educated guess based on options in the TIFF/EP standard to explain why ARW files so small compared to TIFF files of the same image. ARW files do not use a Planar Configuration of the color samples.

Thanks for calling my hand, D, but the color samples in an ARW file are also not a 14-bit, one-channel, "grayscale" image.

I examined the metadata in a readily available ARW file from a Sony A7II that I have since replace. The file contains of two JPEGs, a TIFF, and possibly two thumbnails. While I can't find a ARW spec on the web, it is obviously a derivative of the TIFF/EP standard (ISO 12234-2)--noteably since the TIFF tag Format = 'tif'.

Filename: '/Users/michael/Desktop/sunset_cliffs_20180803-33.arw'

FileModDate: '03-Aug-2018 15:48:24'

FileSize: 25133056

Format: 'tif'

FormatVersion: [ ]

Width: 6048

Height: 4024

BitDepth: 14

ColorType: 'CFA'

FormatSignature: [73,73,42,0]

ByteOrder: 'little-endian'

NewSubFileType: 0

BitsPerSample: 14

Compression: 'unknown'

PhotometricInterpretation: 'CFA'

StripOffsets: 793088 \* StripOffset(s) + StripByteCount = 25120230 ~ FileSize

SamplesPerPixel: 1

RowsPerStrip: 4024

StripByteCounts: 24337152 \* = 4024 RowsPerStrip * 6048 PixelsPerRow * 1 SamplePerPixel

XResolution: 350

YResolution: 350

ResolutionUnit: 'Inch'

Colormap: [ ]

PlanarConfiguration: 'Chunky'

...

The 24.3MB = 4024 rows x 6048 columns allows a single byte (eight bits) for the 4x14 bits per RGGB pixel of the Color Filter Array (CFA) sensor. Each byte is not a 14-bit grayscale value describing the image at that pixel: It is 56 bits of data compressed into eight bits. Sony does not identify the compression algorithm ('unknown'), but it does identify the sequence of color samples as "in series" ('Chunky'). The Compression appendices in TIFF/EP describe JPEG compression as separating the four color samples and compressing them separately to get better results as I noted in my earlier post. Based on these items, it is likely each byte in the data stream carries only two bits of compressed-coded information for each of the four samples for each pixel.

The explanation for the file expansion that occurs when converting between a RAW file (an ARW file in particular) and a TIFF file is that the 4x14 bits per RGGB pixel of the camera sensor is compressed into eight bits per pixel in a RAW file that is expanded into 3x16 bits per RGB pixel in a TIFF file. That is a factor of 6 = 3x16/8 expansion. (ZIP compression of the data in a TIFF file has a limited effect because it does not separate the color samples.)

[D Fosse]

"4x14 bits per RGGB pixel of the camera sensor is compressed into eight bits per pixel in a RAW file"

No, you don't understand. There is no 4 x 14 bits. There is 1 x 14 bits per pixel. The raw file is a true grayscale file, there is no color information in the sensor. The color information comes from the Bayer filter in front of the sensor, and it is added to the data in the demosaicing process, in the raw converter. The color information is interpolated from the known pattern of the Bayer filter array.

This is is when single channel grayscale at 14 bit depth is encoded into three channel data at 16 bit depth, and that fully accounts for the increase in file size. There is nothing unexpected about it.

[Michael Prais]

PSD files have the ability to save color data in "planes" or separate "channels" when the TIFF tag PlanarConfiguration is set to "planar",, that is, in separate RRR..., GGG..., and BBB... grayscale images, but it appears that Ps only sets PlanarConfiguration to  "chunky", that is, RGBRGBRGB.... (One might think of the 'planar' configuration as a parallel arrangement of the color samples and the 'chunky" configuration as a serial arrangments of the color samples.)