P: LrC reads metadata for a DNG from an associated sidecar

[John R Ellis]

When LR imports a DNG, if there is a .xmp sidecar of the same file name, LR will incorrectly read metadata from the sidecar, but only if the date-modified of the .xmp file is newer than that of the .dng.  Similar behavior occurs with Metadata > Read Metadata From File. 

LR should never read metadata for a DNG from a .xmp sidecar (just as it doesn't for JPEG, TIFF, and other non-raw formats). And note that for sidecars of raw files, the sidecar is always read regardless of whether its date-modified is newer or older than the associated raw file.

See this thread for the original report of the problem.

Tested on LR 11.5 / Mac OS 12.4.  Steps to reproduce:

1. Disable the option Catalog Settings > Metadata > Automatically Write Changes Into XMP.

2. Download  a.xmp and a.dng into the same folder.

https://www.dropbox.com/s/n9kpqeddqtff83p/dng-xmp.2022.09.16.zip?dl=0

3. In Mac Terminal, do the command "touch a.dng"; on Windows, do "copy a.dng + nul a.dng".  These commands will set the date-modified of a.dng to "now", newer than a.xmp.

4. Import a.dng with the Add option.

5. Select a.dng and observe that the Metadata panel's Sidecar Files field is blank.

6. Observe that a.dng is black & white with no label or rating (correct).

7. Remove a.dng from the catalog.

8. In Mac Terminal, do the command "touch a.xmp"; on Windows, do "copy a.xmp + nul a.xmp".  These commands will set the date-modified of a.xmp to "now", newer than a.dng.

9. Import a.dng with the Add option.

10. Select a.dng and observe that the Metadata panel's Sidecar Files field is blank.

11. Observe that a.dng is color, not black & white, and it has a red label and rating = 2, indicating that LR has read metadata from a.xmp (incorrect).

[FercStar]

I often edit my photos on mulitple computers, so I have the setting "Automatically Write Changes to XMP" turned on. With proprietary raw formats this works great because it saves the edit info to an XMP sidecar, but with DNG, TIFFs, etc., it saves the edits to the file.

This creates a problem when the file has an XMP sidecar as well. When Lightroom reads the metadata from the file, it prefers the XMP over the DNG. This means, that it will ignore edits made to the DNG and reset all of the edits. From what I can tell, there is no option to change this.

I understand the Lightroom doesn't write metadata to an XMP for DNGs, but if it won't write to the XMP, why does it read from the XMP instead of the DNG? This essentially breaks mutliple device workflows using DNGs, or TIFFs. Lightroom should either allow writing to XMP sidecars for DNGs, prefer the metadata in the DNG over the XMP, give the user a choice which file to use, or merge the metadata between the two.

[JohanElzenga]

Why do your DNG files have a sidecar file in the first place? Where does it come from?

[FercStar]

They are created by another app (specifically Mylio which synchronizes my library to multiple devices).

[DdeGannes]

Just a couple of thoughts.

Only raw camera images will read info from xmp sidecar files at initial import, if there is one with the same file name and a .xmp extension.

Raw camera images will not read xmp data that is stored in a file with the same name with a .DNG, tiff, or jpeg extension.

LrC will "automatically write metadata to xmp" if the option has been selected.
LrC will read xmp at import if it's available in the file or sidecar, thereafter LrC will only read metadata from the file when prompted manually.

When you are working in LrC as far as the application is concerned the information is in the Catalog file.

[FercStar]

DNGs will also read from the xmp sidecar which is the problem I am having. If DNGs ignored xmp sidecars, then there wouldn't be an issue. Since they favor xmp sidecars over the xmp that is embedded in the DNG, when you transfer the files with their sidecars from one computer to another, you can not read the edits from the DNG because Lightroom ignores it in favor of the xmp.

[DdeGannes]

Maybe you could make a change in the thread header the Mylio xmp sidecars.

It may attract attention to someone is fimilar with the application.

[FercStar]

Mylio isn't the problem. It just happens to be the application creating the XMPs. Any XMP sidecar next to a DNG is the problem.

Your can recreate the issue easily.

1. Ensure that Automatically write changes into XMP is checked
2. Take any DNG and apply edits in Lightroom
3. Create an empty text file with the same name as the DNG and the xmp extension
4. Read Metadata from File
5. Your edits will be reset

[John R Ellis]

[This post contains formatting and embedded images that don't appear in email. View the post in your Web browser.]

"Your can recreate the issue easily.

1. Ensure that Automatically write changes into XMP is checked
2. Take any DNG and apply edits in Lightroom
3. Create an empty text file with the same name as the DNG and the xmp extension
4. Read Metadata from File
5. Your edits will be reset"

This degenerate case is a bug in LR -- the empty file is an invalid XMP file and LR should ignore it entirely.  Note that in this example, if you create the empty .xmp file first, then do the edits (which will modify the .dng), then do Read Metadata From File, the bad behavior doesn't occur.  Apparently LR is comparing the modification dates of the two files, which it normally doesn't do with sidecars.

In general, LR has always ignored .xmp sidecars for all non-raw formats (to the consternation of many users over the years who have wished otherwise).  For example, I edited a raw a.cr2 and LR created its sidecar a.xmp.  This shows up in the Metadata panel:

Then I converted a.cr2 to DNG with Library > Convert Photo To DNG, which left a.dng, a.cr2, and a.xmp all in the same folder, and a.cr2 has been removed from the catalog.  But LR doesn't consider a.xmp to be a sidecar of a.dng, as you can see in the Metadata panel:

The sidecar has references to a.cr2 in it, but you can change those to a.dng or delete the lines entirely, and LR still won't read metadata from a.xmp into a.dng.  

So there's something particular going on with your DNGs and the specific content of those .xmps that is triggering another bug in LR.  I recommend that you upload a sample DNG and its .xmp sidecar generated by Mylio to Dropbox, Google, or similar and post the sharing links here.  We can then put the files under the microscope and get an actionable bug report filed that Adobe is more likely to pay attention to.

[FercStar]

In Lightroom, it doens't show a sidecar file in the metadata section, but it still reads from it. I confirmed that it reads from whichever was edited most recently.

I uploaded a DNG with a sidcar XMP here: https://1drv.ms/u/s!ApoitvEGEtVbguZajT5DMvKg_suCcg?e=7Wl5fN

If you make some edits in Lightroom and then open the XMP and change a value (e.g. setting rating to 3) then read the metadata again in Lightroom, it will reset the edits.

[DdeGannes]

Ok, thanks for sharing the files.

I have a question was the DNG file created at import to LrC and was the Mylio xmp sidecar present in the folder at the time?

Reason I am asking, the image was created by a Canon camera was the original Raw? 
What Develop Profile applied?

[FercStar]

The DNG was created from a Canon CR2 on import into Lightroom. The xmp was created later.

Lens + CA is the only development setting I apply on import.

[DdeGannes]

Ok thanks for that clarification, so MyLio creates xmp sidecars from the DNG's?

[FercStar]

Yes, which has some significant advantages many of which have been discussed on this forum. The chief among them is the synchronization/backup of files. If you take 3,000 raw images at 30 MB each at an event you have to backup 90 GB of files. If you come back in the next day and add keywords in Lightroom, you have another 90 GB of files. Every time you make even a small edit to those files in Lightroom, they need to be backed up again. When you write to XMP, you only need to back up a tiny text file that is several orders of magnitude smaller than the raw image.

[FercStar]

It won't let me change the title, but it should be reverted back to what it was. Changing the title to include Mylio misrepresents the issue. Any xmp added next to a DNG will cause the issue. It doesn't matter what the source is.

[DdeGannes]

At FercStar, I have reread the thread and did some checks with the DNG and XMP sidecar that you supplied. My thought are as follows.

You use Mylio to manage your image files including original raw CR2's and make some metadata changes there and xmp sidecars are created.

Subsequently you import into LrC using Copy as DNG and LrC reads initial info from the XMP sidecar and create a DNG. So you now have two images a CR2 and a DNG.

Lightroom treats the files as two separate files, the DNG files do not read / or should not read from the xmp sidecar.

Is this how you are working, if not please explain.

Ok just read your last response.

[FercStar]

No, I import the files into Lightroom directly from the SD card using Copy as DNG. There is only one image file, the DNG, as the CR2 files are deleted during the import process.

Mylio doesn't see that the files exist until well after the photo has been converted to DNG and the CR2 is removed. The xmp is created for the DNG.

Lightroom treats the xmp as a sidecar for the DNG and will read from it instead of the DNG if the timestamp on the xmp is newer which is the core issue.

[DdeGannes]

Ok, I downloaded the DNG and the XMP Sidecar you shared again. Then proceeded to do the following, first I placed DNG into a folder on my desktop and imported the DNG using the "Add" option in the import dialog and expected to get what you had done when you first imported. See the screen capture.

Next, I added the XMP Sidecar you created with MyLio so I could see the edits you did, then proceeded to Read the Metadata from the file. See the screen captures.

The results show there has been some tone adjustments applied. But no movement of the sliders.

[FercStar]

There aren't any edits in the DNG or the XMP. To replicate the issue, you need to edit the DNG in Lighroom, then perform some action to update the modified date on the XMP (e.g. change the rating in a text editor), and finally read the metadata again.

This will result in the edits being lost.

[DdeGannes]

One final thought, when you have "Automatically write metadata to XMP" selected, make sure you do not have the option to "Save to XMP set to pause. See the screen capture.

[FercStar]

It is not paused. The issue is not in saving to xmp or dng. It is that Lightroom reads the XMP when it should be reading the DNG.

[John R Ellis]

I filed this bug report:

https://community.adobe.com/t5/lightroom-classic-bugs/lr-incorrectly-reads-metadata-for-a-dng-from-a...

The bad behavior also occurs with Automatically Write Changes Into XMP turned off.

[FercStar]

Thanks

[Thomas Knoll · Accepted solution]

This is expected behavor as far as Adobe is concerned.  

If Mylio does not want the sidecar file to be attached to the ".DNG" file, but instead a file with a different extension but the same name, is should be including the "photoshop:SidecarForExtension" tag inside the .XMP sidecar.  If this tag exists and does not specify "DNG", then Adobe software will then ignore the sidecar when reading DNG files.

[FercStar]

If this is the expected behavior, why doesn't it work this way with a TIF. If you recreate the scenario and use a TIF instead of a DNG, Lightroom does not read from the XMP. It only reads from the file. The read metadata from file behavior should be consistent across all file types where the edit info is embedded in the file and not in a sidecar file.