DNG and XMP

[hamadahiro]

I just want to understand this — I have enabled the option "Automatically write changes into XMP" in Catalogue Settings, but there are no sidecars, because, as I understand metadata is embeded into DNGs? So, then what this option does in my case? What is the difference if it's checked or unchecked when there are no XMP files anyway?

[Conrad_C]

The metadata is written to external XMP files when the format is either read-only (like camera raw) or doesn’t support it.

When the format does allows writing metadata to the file, then the command updates the file. So if you use that command when a DNG (or JPEG, TIFF, etc.) file is selected, the metadata portion of the file is updated, which doesn’t affect the image data.

[hamadahiro]

Thanks, so in my case, I guess DNG files allows writing metadata and they would be updated with the new data. Since there are no sidecar files, it must me within the DNGs.

[johnrellis]

"I guess DNG files allows writing metadata and they would be updated with the new data. Since there are no sidecar files, it must me within the DNGs."

Correct.

The wording of that option is misleading: "Automatically write changes into XMP". When the option is enabled:

- For file formats defined by industry standards (DNG, JPEG, TIFF, PSD, PNG), the metadata is written into three metadata sections in the file: EXIF, IPTC, and XMP (each section having its own data format).

- For proprietary raw formats, the metadata is written in XMP format into a .xmp sidecar.

[richardplondon]

When it's all happening in the same file, your operating system will keep reporting that whole thing as newly changed, just because you moved a slider or two. Something to consider.

I haven't seen any complaints of file corruption to DNGs from repeatedly writing XMP into those - the file format is designed for that - but my personal judgement is that a separation of completely unchanging proprietary Raw vs changing LrC metadata file makes sense.

XMP sidecars are certainly the more efficient to re-backup regularly. And if simply discarded, you are then left with an untouched Raw that's still exactly as the camera made it. The only exception I know of, is: if correcting capture date and time, that change can get written out into the proprietary Raw as well as into XMP sidecar, if you check a setting in preferences to permit LrC to do so.

[C.Cella]

All DNGs can be validated withing LrC.

If a user is concerned about DNG integrity he/she can run the validation and verify if corruption happened.

Library > Validate DNG Files.

What I recommend to do is to:

1. Import or create the DNGs

2. View it in develop module not Library

3. If free of corruption then create the fingerprint/checksum by doing the following:

Metadata> Update DNG Preview and Metadata

.

[hamadahiro]

But how do I choose to write in a separate XMP file? I already have that option selected, yet, I don't have any sidecar files next to my DNGs.

[johnrellis]

"But how do I choose to write in a separate XMP file?"

You can't choose. For industry standard formats (DNG, TIFF, PSD/PSB, JPEG, PNG), LR writes metadata into the file itself. For proprietary raw formats, LR writes metadata into a .xmp sidecar. For videos, LR never writes metadata.

Lots of people have long requested the ability to use sidecars for all file types:

https://community.adobe.com/t5/lightroom-classic-ideas/p-store-the-xmp-metadata-outside-dng-jpeg-etc...

Camera Raw gives that option for DNGs, but not LR.

[Conrad_C]

But how do I choose to write in a separate XMP file? I already have that option selected, yet, I don't have any sidecar files next to my DNGs.

By @hamadahiro

It’s not written as clearly as it might be, but a big clue is that the word “sidecar” never appears in that Metadata settings window. It says XMP and it’s writing XMP, so it does what it says…but for formats that allow writing into the file itself, the XMP metadata goes into the file.

Not saying there’s anything wrong to want a sidecar even for formats that can write internally, it’s a perfectly legitimate request. But for now, that option isn’t provided, and adding that capability is apparently not high enough on the list of feature requests Adobe decides to work on next.

[C.Cella]

 Saving into XMPs unfortunately is incredibly slow in LrC (no matter the Hardware).

One priority should be to improve that aspect as well.

I am also an advocate for the separation of edits and metadata in sidecars.

[hamadahiro]

I understand, but in my mind, XMP is a separate file, as if .xmp. But now, I understand how this works. It would be good to have an option for the xmp sidecar because it doesn't change a massive DNG file everytime I make any change.

I use both CC sync, and Backblaze that backups my whole hard drive, and once I added a single keyword to over 40,000 images and it was backing up (uploading) for a whole week. At that time I didn't understand what's going on, but now I get it.

[C.Cella]

@hamadahiro 

With XMPs as they are now you would face a similar situation.

40k files to backup completely.

LrC overwrites the entire XMP even for a single keyword, even for a single .

So maybe not one week but for sure it would take long to do the backup.

[hamadahiro]

Yeah, but .xmp files are what,  50-60kb? My DNGs are between 40-350mbs. Huge difference 🙂

[C.Cella]

@hamadahiro 

Metadata is trivial in size but for edited images XMPs is more likely in the MB if you use Masking + Snapshots add to that.

Yes generally still less then the entire file...but not always.

[richardplondon]

I think we can say that some kinds of masking will make a bigger difference to metadata than others. For example: the XMP representing lots of individual mask brushstrokes or spot heals, as opposed to that representing a parametric-defined linear grad. I believe the bitmap resulting from an AI masking selection is kept in a cache folder alongside the Catalog, and so is not included in the editing XMP that might be written into the imported image file or else into a sidecar file.

Separation of external written metadata from the image file itself, has IMO logical as well as practical arguments for - and, against.

[Ian Lyons]

The mask bitmap, etc needs to be in the XMP sidecar or DNG otherwise it's not easily transferable to another computer.

[richardplondon]

The bitmap I referred to is the result of an AI selection, but specification for that is included in the XMP. So I'd guess, on another computer this AI selection would be evident as requiring recalculation, and then it would be this other computer which then caches the bitmap which results. I suppose just the same as, how a brush mask does not itself consist of a bitmap alone, but rather of a series of parametric defined brush "dabs" which interpreted and taken together, can be used to generate an overall result, which might similarly get cached in bitmap form to facilitate the image preview and processing.

[C.Cella]

@richardplondon 

Definitely brush (as usual) is the one tool that will add most to XMPs size.

The crs:Table associated to healing also adds to the size significantly.

I believe several healing can easily amount to MB as well.

Further separation of metadata and edits would be ideal...but that would require a new type of XMPs, one with two sub XMPs inside (edits and metadata) each writable independently.

I don't know if that is doable or it's a pipe dream.

.

[johnrellis]

"I understand, but in my mind, XMP is a separate file, as in .xmp."

To build on Conrad's reply, XMP (all caps) refers to an industry-standard metadata sepcification, eXtensible Metadata Platform.  XMP-format metadata can be stored inside nearly all photo and video formats, as well as in .xmp sidecars.

The wording of the option "Automatically Write Changes into XMP" is not only ambiguous but wrong -- when enabled, it not only writes changes into XMP metadata, but also into the EXIF- and IPTC-format metadata sections of non-raws.

[C.Cella]

@johnrellis 

Could be worth filing a Bug Report and ask for a correct, clear description.

I would definitely vote for it.

[DdeGannes]

Additional info also available at the link below.

https://helpx.adobe.com/lightroom-classic/help/metadata-basics-actions.html

[dj_paige]

I have enabled the option "Automatically write changes into XMP" ... What is the difference if it's checked or unchecked when there are no XMP files anyway?

If you check the option, your metadata and edits are written into the DNG file. I you don't check the option, then nothing is written into the DNG file.