• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
2

P: LrC reads metadata for a DNG from an associated sidecar

LEGEND ,
Sep 16, 2022 Sep 16, 2022

Copy link to clipboard

Copied

When LR imports a DNG, if there is a .xmp sidecar of the same file name, LR will incorrectly read metadata from the sidecar, but only if the date-modified of the .xmp file is newer than that of the .dng.  Similar behavior occurs with Metadata > Read Metadata From File. 

 

LR should never read metadata for a DNG from a .xmp sidecar (just as it doesn't for JPEG, TIFF, and other non-raw formats). And note that for sidecars of raw files, the sidecar is always read regardless of whether its date-modified is newer or older than the associated raw file.

 

See this thread for the original report of the problem.


Tested on LR 11.5 / Mac OS 12.4.  Steps to reproduce:

 

1. Disable the option Catalog Settings > Metadata > Automatically Write Changes Into XMP.

 

2. Download  a.xmp and a.dng into the same folder.

https://www.dropbox.com/s/n9kpqeddqtff83p/dng-xmp.2022.09.16.zip?dl=0

 

3. In Mac Terminal, do the command "touch a.dng"; on Windows, do "copy a.dng + nul a.dng".  These commands will set the date-modified of a.dng to "now", newer than a.xmp.

 

4. Import a.dng with the Add option.

 

5. Select a.dng and observe that the Metadata panel's Sidecar Files field is blank.

 

6. Observe that a.dng is black & white with no label or rating (correct).

 

7. Remove a.dng from the catalog.

 

8. In Mac Terminal, do the command "touch a.xmp"; on Windows, do "copy a.xmp + nul a.xmp".  These commands will set the date-modified of a.xmp to "now", newer than a.dng.

 

9. Import a.dng with the Add option.

 

10. Select a.dng and observe that the Metadata panel's Sidecar Files field is blank.

 

11. Observe that a.dng is color, not black & white, and it has a red label and rating = 2, indicating that LR has read metadata from a.xmp (incorrect).

 

TOPICS
macOS , Windows

Views

1.7K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct answer

Adobe Employee , Sep 20, 2022 Sep 20, 2022

This is expected behavor as far as Adobe is concerned.  

If Mylio does not want the sidecar file to be attached to the ".DNG" file, but instead a file with a different extension but the same name, is should be including the "photoshop:SidecarForExtension" tag inside the .XMP sidecar.  If this tag exists and does not specify "DNG", then Adobe software will then ignore the sidecar when reading DNG files.

Votes

Translate

Translate
Adobe Employee ,
Sep 22, 2022 Sep 22, 2022

Copy link to clipboard

Copied

Because TIFF format was invented before both DNG format and XMP sidecar files were invented (by me, BTW), so it was not possible to change all the existing TIFF readers in the world to read metadata from XMP sidecar files, only the embedded XMP inside the TIFF.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 21, 2022 Sep 21, 2022

Copy link to clipboard

Copied

From what I can see, there are three separate behaviors when reading metadata from XMP.

 

1 - TIF, PNG, JPG - only read from the file not from a sidecar

2 - RAW (cr2, arw, etc) only read from xmp not from the file

3 - DNG read from whichever has the most recent modified date

 

Behaviors 1 and 2 make sense, but behavior 3 does not.

 

DNGs should receieve the same treatment as either proprietary raw formats and get edits written to a sidecar and read from a sidecar, or they should get the same treatment as open format files and have their edits written and read from the file ignoring sidecars.

 

The inconsistent behavior of the DNG is a problem and it is hard for me to see how having a special behavior for DNG is the "expected behavior."

 

If the behavior for a DNG is correct then the behavior for TIF, PNG, and JPG is incorrect. If the behavior for TIF, PNG, and JPG is correct, then the behavior for DNG is incorrect.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Sep 22, 2022 Sep 22, 2022

Copy link to clipboard

Copied

Since I invented both DNG format and XMP sidecar files for raw formats, I think I get to define what is correct.

Sidecar XMP files for DNG format are prefered by many users because it works more effiently with their file backup and multiple machine file syncronization workflows.  Camera Raw offers a user preference for how metadata should be handled for DNG format when saving metadata (prefer to embed, or prefer to use sidecar).   Before this preference existed, users would force the use of XMP sidecar for DNG format by just marking the DNG files as read only at the file system level.

So there are a very large number of DNG files with XMP sidecar files already in existence for many users.  Your proposal to always ignore sidecar XMP files for DNG format would be all these users would lose their Camera Raw settings for this DNG files.  Sorry, but I'm not trash all these users metadata just to work around a bug in some non-Adobe software.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 23, 2022 Sep 23, 2022

Copy link to clipboard

Copied

Adobe Camera Raw is exactly how it should work. It gives users the choice. Lightroom does not. In Adobe Camera Raw, you can pick to save the edits to the DNG, or you can choose to only use sidecars. In Lightroom, you don't have a choice. It picks for you, and it picks inconsistently. If the timestamp on the XMP is newer than the timestamp on the DNG, it reads from it, and, as you put it, trashes the user's metadata from the DNG.

 

You can recreate the problem I am having using only Adobe software using the below steps.

 

  1. Set Camera Raw to ony use sidecars
  2. Set Lightroom to save edits to xmp
  3. Edit photo in Adobe Camera Raw
  4. Close the photo and reopen in Adobe Camera Raw (this confirms that is saved and read your edits from the XMP)
  5. Rate photo in Lightroom (this results in the modified date of the DNG being updated)
  6. Reopen photo in Camera Raw
  7. See that you just lost your Camera Raw edits

 

If Lightroom had the same ability for the user to choose to use sidecars or to save to DNG as Camera Raw, there would be no issue, but as you can see Lightroom's inablitly to allow the user to chose causes the user to lose edit information whether or not they are editing the XMP with a third part app.

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 23, 2022 Sep 23, 2022

Copy link to clipboard

Copied

I think the difference in the results between these to workflows best displays the issue.

 

Adobe Camera Raw/Mylio

  1. Edit photo in Adobe Camera Raw saving edits to XMP
  2. Close photo
  3. Rate photo in Mylio
  4. Open photo in Camera Raw
  5. Edits are retained and rating is retained

 

Lightroom/Mylio

  1. Edit Photo in Lightroom saving edits to file
  2. Rate photo in Mylio
  3. Read metadata from photo in Lightroom
  4. Edits are lost rating is retained

 

There should be a way to have Lightroom function in the same way as Camera Raw for this scenario.

 

That could be accomplished by giving the user the option to chose to write their edits in the sidecar file, or by making the behavior consistent with either RAW files or with TIF/JPG.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Sep 23, 2022 Sep 23, 2022

Copy link to clipboard

Copied

As a Lightroom user from inception in Feb 2007 the default procedure of Lightroom is, read and write to the Catalog file. As far as Lightroom is concerned the info is in the Catalog.

Many users of Lightroom follow the default options, there is no necessity to write metadata to xmp, this is an option that allows Lightroom users to share xmp data with other applications mainly Bridge / Adobe Camera Raw.

Adobe Photography plan provides two options for working with raw images files from your digital cameras, first there is Bridge (a file browser) / Camera Raw that stores metatadata and edits to the file (utilizing xmp sidecars to store information for raw images, but not 100%), then there is Lightroom which has a Library Module that utilizes a Catalog data base file to store all your metadata and edits.

 

The best option for you is to use Camera Raw since you have chosen MyLio to manage your Library and have no need for Lightroom or Bridge.

I am a Lightroom user (now Lightroom Classic) and have no need for xmp sidecar or DNG files, just my personal choice.

There is no bug in LrC that needs to be fixed.

There is the option in the forums to post a idea for a new feature.

 

P.S., In your original post you also indicated you converted your RAW images to DNG prior to working with them in MyLio. Have you checked how things would work if you did not choose to convert to DNG?

 

Regards, Denis: iMac 27” mid-2015, macOS 11.7.10 Big Sur; 2TB SSD, 24 GB Ram, GPU 2 GB; LrC 12.5, Lr 6.5, PS 24.7,; ACR 15.5,; Camera OM-D E-M1

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 23, 2022 Sep 23, 2022

Copy link to clipboard

Copied

I have strong reasons for using some of the catalog features if Lightroom, and Camera Raw comes no where close to meeting those needs. Switching from Lightroom to Camera Raw would substantially impair my workflow.

 

I write my edits to xmp because I edit my photos on multiple computers and want the edits to travel with the files. The only way to do that without writing to xmp is the extremely cumbersome process of exporting, transferring, and importing catalogs. The other option is to use prohibitively expensive Adobe cloud storage.

 

The way Lightroom handles xmps for dngs is inconsistent with how it handles other file types and inconsistent with how Adobe Camera Raw handles dngs.

 

It makes the incorrect assumption that no other app will write to an xmp sidecar for a dng. It was mentioned previously in this thread that xmp sidecars should not be used for dngs. This is clearly incorrect from Adobe's point of view as Camera Raw creates sidecars for xmps.

 

The fact that Adobe Camera Raw does not have this issue is additional evidence for the fact that there is a flaw in Lightroom's implementation of handling dng sidecar files.

 

I have lost hundreds of edits to this "not a bug." Since I found the issue I stopped converting my photos to dng, but that doesn't fix the the tens of thousands of photos I trusted to the dng format.

 

This is a bug because it causes the unintentional loss of user information, and makes the false assumption that the file with the most recent modified date is also the file with the most recent edit data.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Sep 23, 2022 Sep 23, 2022

Copy link to clipboard

Copied

You stated,

1 - TIF, PNG, JPG - only read from the file not from a sidecar

2 - RAW (cr2, arw, etc) only read from xmp not from the file

3 - DNG read from whichever has the most recent modified date

My expectation.

1, There should be no XMP sidecars files for these file types, the data is stored in the file.

 2. Agreed. XMP data is/should not be stored in RAW files, xmp sidecar files are created to store the data.

3. XMP sidecar files should never be created for DNG files, that's how the DNG file type is designed.

Regards, Denis: iMac 27” mid-2015, macOS 11.7.10 Big Sur; 2TB SSD, 24 GB Ram, GPU 2 GB; LrC 12.5, Lr 6.5, PS 24.7,; ACR 15.5,; Camera OM-D E-M1

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 23, 2022 Sep 23, 2022

Copy link to clipboard

Copied

If it is true that XMPs should never be created for DNGs, then why does Lightroom read from them? If they shouldn't exist, then Lightroom should have no way to read from them.

 

Adobe Camera Raw creates XMPs for DNGs, so Adobe clearly believes that it is okay for a DNG to have an XMP. Additionally, apparently in the past you could force XMPs to be created by preventing editing of the DNGs in the file system.

 

Regarding TIF, JPG, etc. and proprietary raw formats, the current behavior doesn't bother me because it is consistent. For the purposes of backup and syncing, a case could be made for using XMPs with them as well.

 

Lightroom always does the same thing with these files. For a given file type, it always reads and writes to the same file. Treating the files consistently is more important than optimizing them for backup.

 

With DNGs it does not do this. Instead it always writes to one file, but it will read from either file depending which file was updated most recently. This inconsistency is the problem because it is based on the faulty assumption that the most recently modified file is the one with the most recent editing information

 

Fortunately there is a conceptually simple solution. Let the user choose where to save their edits for DNG files and then only read and write from that.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Sep 24, 2022 Sep 24, 2022

Copy link to clipboard

Copied

quote

Fortunately there is a conceptually simple solution. Let the user choose where to save their edits for DNG files and then only read and write from that.


By @FercStar

Make a feature request; that's about all you can now do:

https://community.adobe.com/t5/photoshop-ecosystem-ideas/how-do-i-write-a-feature-request/idi-p/1238...

Author “Color Management for Photographers" & "Photoshop CC Color Management/pluralsight"

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 24, 2022 Sep 24, 2022

Copy link to clipboard

Copied

Like this one from 8 years ago? Feature Request 

 

I am not wanting a new feature. I am wanting a design flaw that results in the loss of user data to be corrected. It could be corrected by adding that feature but that is not the only way it could be corrected.

 

It could also be corrected by reading both files and either detecting that only the dng has edit info or by prompting the user to pick which file to read from if it detects a sidecar with conflicting data.

 

There are a myriad of ways to correct the design flaw. Many of which I have undoubtedly not thought of.

 

If this is treated as a Feature Request and not the data deleting design flaw that it is, I have no confidence that it will be addressed. I offer as evidence the 8 year old feature request linked above.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Sep 24, 2022 Sep 24, 2022

Copy link to clipboard

Copied

quote

Like this one from 8 years ago? Feature Request 

By @FercStar

 

Yup, 8 years old and 36 votes. Now you know why the Adobe team isn't taking the request very seriously. 

Did you vote there? Did you get other like-minded users to vote? 

This isn't a design flaw; it was made by choice, which you and those 36 others don't agree with. 

Maybe it's time to find another software solution if this is a deal breaker for you. 

 

Author “Color Management for Photographers" & "Photoshop CC Color Management/pluralsight"

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Sep 24, 2022 Sep 24, 2022

Copy link to clipboard

Copied

As I stated above. I am not requesting that feature. The reason for the feature request is entirely different than this issue. It just happens to be a change that would correct the underlying design flaw. I also listed other possible solutions to the problem and not just that one.

 

I have provided detailed instructions on how to recreate the issue and reiterated numerous times why it is an issue. With the exception of the forum user who replicated the issue and created the initial bug report based on my first forum post, the issue has not been taken seriously by anyone. Instead it has been treated dismissively with comments such as:

 

quote

Yup, 8 years old and 36 votes. Now you know why the Adobe team isn't taking the request very seriously. 

...

Maybe it's time to find another software solution if this is a deal breaker for you. 

 

By @TheDigitalDog

 

or 

quoteYour proposal to always ignore sidecar XMP files for DNG format would be all these users would lose their Camera Raw settings for this DNG files.  Sorry, but I'm not trash all these users metadata just to work around a bug in some non-Adobe software.
 

 

It is true enough that the potential solution I provided would have an unacceptable downside, but the problem should not be dismissed because the user reporting it is not able to provide the correct solution. Neither should other software be blamed when the issue can be replicated with a text editor or entirely using Adobe software. I have suggested some potential solutions, not because I am commited to them being implemented in the exact way I state them, but in hopes that it would help lead to a resolution to the problem.

 

I do not wish to stop using Lightroom. I simply want the issue I have raised to be taken seriously. There are undoubtably other users who experienced the same data loss I have, and due to the nature of the issue, there are probably many who do not know that they have lost some of their edits.

As for me, I can work around the problem. I have already stopped using DNGs because of this, and I can write a script that updates the modified date on all the DNGs in a folder if the XMP was updated more recently, so I won't lose future DNG edits.

 

 

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Sep 24, 2022 Sep 24, 2022

Copy link to clipboard

Copied

quote

As I stated above. I am not requesting that feature.  The reason for the feature request is entirely different than this issue.

By @FercStar

 

The feature request from 8 years ago (and 36 votes) you provided us today? 

 

I do not wish to stop using Lightroom. I simply want the issue I have raised to be taken seriously.

 

Maybe start by actually posting the feature request you desire and stop arguing with the fellow who invented the technology that might provide you the feature you haven't correctly requested yet:


My core belief is that if you're complaining about something for more than three minutes, two minutes ago you should have done something about it.“ -Caitlin Moran

 

Make a feature request; that's about all you can now do:

https://community.adobe.com/t5/photoshop-ecosystem-ideas/how-do-i-write-a-feature-request/idi-p/1238...


By @TheDigitalDog

 

Author “Color Management for Photographers" & "Photoshop CC Color Management/pluralsight"

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Sep 27, 2022 Sep 27, 2022

Copy link to clipboard

Copied

LATEST

[This post contains formatting that doesn't appear in email. View the post in your Web browser.]

 

I think it's safe to say that this thread contradicts the prior received wisdom in this forum about LR and DNG sidecars. @Thomas Knoll clarified that Camera Raw will read metadata from DNG sidecars depending on preference settings, but he didn't provide an explanation or design rationale for some of Lightroom Classic's behaviors. So I'll try to document LR's behavior and show how it fits in with Camera Raw; as always, I'd love to have any mistakes corrected.

 

Camera Raw has three options for the preference DNG File Handling > Sidecars.  The Help is long out-of-date, but reverse engineering indicates the following meanings:

 

Embed XMP in DNG: If a sidecar exists, it is read; otherwise the metadata is read from the DNG.  But when the image is closed, the metadata is always written into the DNG and any sidecar is deleted.

 

Always use sidecar XMP files: If a sidecar exists, it is read; otherwise the metadata is read from the DNG.  But when the image is closed, the metadata is written to the sidecar and the DNG is left untouched.

 

Ignore sidecar XMP files: The metadata is never read from or written to sidecars if they exist.  Metadata is only read from and written to the DNG.

 

Lightroom Classic always uses the option Embed XMP in DNG, meaning it will read metadata from an existing sidecar, but unlike Camera Raw, it doesn't delete an existing sidecar when the user does Metadata > Save Metadata To File.  To the chagrin of some users, LR doesn't provide the other two Camera Raw options.

 

Since LR will read an existing DNG sidecar but not write it, it kinda-sorta makes sense that the Metadata panel's Sidecars field will always be blank regardless of the presence of a sidecar. In general, the Sidecars field will indicate a sidecar only if it will write it. But it sure is confusing for LR to read metadata from the sidecar but not display the sidecar in the Sidecars field.

 

What about the fact that LR will sometimes only read a sidecar if its newer than a DNG?  It's complicated.

 

When there's a sidecar present, and one of the options Embed XMP in DNG or Always use sidecar XMP files is selected, then the sidecar's metadata is read only if Camera Raw determines that metadata is newer than any XMP metadata in the DNG.  In particular, it will read the sidecar if:

 

- The DNG doesn't contain any XMP metadata; or

 

- The sidecar contains the undocumented Adobe-proprietary field photoshop:EmbeddedXMPDigest and the contents of that field (an MD5 hash) matches the hash of the current XMP metadata in the DNG; or

 

- The sidecar doesn't contain the field photoshop:EmbeddedXMPDigest and the sidecar's last-modified date is newer than that of the DNG.

 

This algorithm ensures that if another app modifies the metadata in the DNG, any existing sidecar will then be recognized as "stale" and won't be read. (It's similar to the algorithm defined by the Metadata Working Group standard for determining whether to read fields from XMP or IPTC-IIM.)

 

This is why sidecars produced by the Mylio app would only be read if they were newer than the DNG -- they didn't contain photoshop:EmbeddedXMPDigest.

 

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines