• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
1

Is it possible to use XMP files with DNG?

Participant ,
Dec 05, 2019 Dec 05, 2019

Copy link to clipboard

Copied

I have used XMP files since the very beginning of Lightroom. It's a handy way of having an "instant backup" of changes just in case something ever happens to my Lightroom Catalog.  I love the safety net that XMP files give me.

 

A year and a half ago (after 15 years with Canon) I switched to the Sony a7iii.  The biggest downside to that is that Sony's ARW files are 47mb each.  But if I use the Adobe DNG Converter, I can losslessly compress them down to just 27mb each.  As I shoot thousands upon thousands of images each month, that 20mb per image savings is huge.  Especially when you take into account backups.  

However, this is one big thing I hate about using DNG files.  And that is that every time I make a change in Lightroom (change a star rating, adjust exposure, crop, etc) that 27mb DNG file needs to be backed up.  AGAIN.  Because unlike my Canon raw files, with XMP sidecar files, the DNG files are written back to whenever I make any changes.  

Is there any way to change Lightroom to write to a tiny XMP file every time I make a change, rather than modifying my 27mb DNG files?  I would much rather backup and rebackup tiny little XMP files.  It makes no sense to have to backup an entire 27mb file just because a few bytes inside of it have changed.  And it costs me money (I have to pay extra to my Internet provider as I go over their imposed limits). 

So please ... is there any way to use XMP files with DNG?  And if not, can you please add this to the wishlist?  My hard drives and my internet bills will thank you.

Views

6.8K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Dec 06, 2019 Dec 06, 2019

Copy link to clipboard

Copied

No, DNG files do not need XMP sidecar files, because their embedded metadata can hold all the info. And so sidecar files are not supported with DNG by design.

 

This is a user-to-user forum. For feature requests, go to https://feedback.photoshop.com. Search for your feature request first. If it is already requested by someone else, then don't create a new thread but add your vote to the existing request.

-- Johan W. Elzenga

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Dec 06, 2019 Dec 06, 2019

Copy link to clipboard

Copied

Yes, I realize they designed it that way.  But now I've provided a big reason (one that is important to me, and could be to others) as to why that "design" is suboptimal. 

I would much rather have the external XMP file.  In all of these years (since LR 1.0 came out) having an XMP file for me has never been an inconvenience, so having one with a DNG wouldn't inconvenience me now. But I very much like the idea that a raw file is a pristine file - that you can trust that nobody ever touches it. That you can back it up and store it forever as an untouchable entity.  But with DNG it gets altered every time I change the star rating or do anything else to the image ... it's a raw file that isn't pristine (the raw data is, but the file itself is not).  And so it has to be backed up every time it gets touched.  And I don't just mean that I feel compelled to back it up, I mean that my backup software which automatically backs up any file that changes on my system will automatically sense that the file has changed and send another 27mb over the internet to the backup server.  So, for example...  let's say I shoot 64GB worth of files for a wedding.  It backs up 64GB worth of files after I get home and import the files into my system.  It'll be a few weeks before I get to the wedding, but when I do, I'm going to rate all of the images.  Now they get backed up again.  I probably won't finish all of my edits in one day, so the next day as I continue editing... any remaining files that get changed will be backed up again.  And again.  And again, for as long as I keep making changes inside of Lightroom.  See how this is suboptimal?  Certainly it's not a lot of money, but it all adds up and is costing me extra for my Internet usage because it is metered.  Whereas if it was using XMP files, then only tiny (what, 10K?) files change as I make my LR edits and that is all that has to be backed up, with XMP files the much larger raw files remain unchanged.

So for me ... DNG storing internally any changes is expensive.  I would love to have an XMP option if it were available.  Thank you, I'll take this to the feature request link you mentioned.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Dec 06, 2019 Dec 06, 2019

Copy link to clipboard

Copied

The practice, not the design, is clearly suboptimal because you are repeatedly, unnecessarily backing up large amounts of data.

 

Of course, you're not the first to ask for sidecars for DNGs, but it's lazy thinking. Sidecars are just a workaround for file formats which are not publicly documented - not for formats like DNG, JPEG, TIF etc which are designed to carry their own XMP metadata. XMP is also designed primarily for communication between applications, not primarily as a backup, so it doesn't contain lots of Lightroom work. Unnecessary sidecars will also introduce potential conflicts reading metadata - eg should LR or other app prefer embedded or sidecar metadata? So sidecars for DNGs are unnecessary, don't include all your work, and will cause problems later.

 

Once you adopt DNG, you should fine tune your backup strategy, just like you would do after any other change to one's workflow. Back up DNGs once upon their creation, meaning all your image data is secured and your space usage is not bloated. Second, back up your work by backing up the catalogue. That gives you full backup coverage, and sensible use of space.

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Dec 06, 2019 Dec 06, 2019

Copy link to clipboard

Copied

Except ... the backup process is automatic - as it should be.  It detects any changes in any file and automatically backs it up.  It doesn't know that it's a file that has already been backed up that has just a few bytes updated in it that don't need to be backed up.  Backups just don't work that way.  

Now if backups could perform byte-wise differential backups that would be cool.  But they don't. Back in modem days I heard of such things, but I think fast Internet has basically done away with the idea... who needs differential backups when you can squirt the whole file up in seconds.  The trouble is the metered connection that charges extra for all of those bytes.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jul 01, 2023 Jul 01, 2023

Copy link to clipboard

Copied

The problem with what you are saying is that backup tools are designed to be robust, automated and fail-safe. There are no separate "photo backup tools" but backup tools that work at either file or block device level. These tools do not have any awareness of the file content other than its binary data. Therefore the combination of DNG's embedded XMP/metadata and LR's abckground metadata updates cause bloated backups and inability to monitor photo library for data corruption using standard file based checksum tools. One's ability to "fine tune the backup strategy" is therefore limited to the available backup tool options. I am now talking about proper automated & versioned backups, not some manual file copy schemes which always break due to human error.   

 

I went to DNG 15 years ago and with 1TB photo library I have regretted my decision ever since. Luckily was smart enough to keep the old RAW+XMP files too so I have now removed DNG from my workflow and will eventually replace the DNG files with RAW+XMP versions. Well, someday...

 

Adobe, please implement XMP sidecar option for DNG files...

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Dec 06, 2019 Dec 06, 2019

Copy link to clipboard

Copied

"However, this is one big thing I hate about using DNG files. And that is that every time I make a change in Lightroom (change a star rating, adjust exposure, crop, etc) that 27mb DNG file needs to be backed up. AGAIN. Because unlike my Canon raw files, with XMP sidecar files, the DNG files are written back to whenever I make any changes. "

 

Obviously that's a suboptimal backup plan - both because of space demans and because XMP doesn't contain all the work you've done on the images.

Instead, back up the DNG once when it's new, and routinely back up the catalogue.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Dec 07, 2019 Dec 07, 2019

Copy link to clipboard

Copied

Maybe I'm missing something here - but why do you need xmp as backup when backing up the catalog is so much more efficient? If you later need that metadata to accompany the file, it's just a ctrl+S away. If you don't, the DNG remains pristine.

 

The point I'm trying to make - that others here have also tried to make -  is that it's very hard to argue any practical advantage to this. In any standard workflow it just complicates everything. Feature requests with no demonstrable advantage are never going to get any traction.

 

I have a backup regime that automatically copies all changed and new files, triggered with a single click. I removed the Lightroom catalog from this routine at some point, because the synchronizing software treats it as a single file, and so it is fully rewritten every time. That makes it vulnerable to corruption that might propagate. Instead I rely on the built-in scheduled backup routine. The point is that the DNGs aren't touched, and so they remain pristine.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Dec 07, 2019 Dec 07, 2019

Copy link to clipboard

Copied

Back in LR 1.0 we had to delete our catalogs about every 3-4000 images due to performance issues.  Ever since LR 2.0 came out, that has no longer been necessary. But I started the XMP practice back with v1.0 and have kept it going just in case my LR catalog ever gets corrupted.  I can literally, at any time, start over with a new catalog and lose nothing (of my settings) simply by reimporting the images.

Just a few days ago a friend of mine accidentally removed 130 images from his catalog right after he finished editing and before he exported them.  OOPS!  I asked if he had XMP turned on ... nope.  Darn.  If he had, all he would have had to do was reimport the images and he'd have all of his edits back.  Instead he had to re-edit all 130 images.  So XMP files can save you from even small disasters.  

Anyway, at this point (seeing that there doesn't sound like there would be much interest in doing what I am asking for), I might just decide to turn of the XMP writes to avoid writing back to my DNG files.  I means the loss of one level of backup ... I dunno.  I'll consider it.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Dec 08, 2019 Dec 08, 2019

Copy link to clipboard

Copied

A catalog backup can also save you from small disasters, if you know how. You friend should have opened a catalog backup that still contained those images, exported the images as a new catalog, and then imported that catalog into his main catalog.

-- Johan W. Elzenga

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Dec 08, 2019 Dec 08, 2019

Copy link to clipboard

Copied

As mentioned ... he had just finished his edits.  

In other words, he had not closed Lightroom or had time to perform any sort of backup.  Therefore no backup was available.  XMP files are written immediately - as you do the work.  It's a much smarter backup in that case. Plus XMPs follow the raw files.  If your catalog is sitting on a drive that fails, you can lose your catalog, but if as long as the XMP files are sitting wherever the raw files are, it means as long as the raw files themselves are readable so is your XMP backup.  

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Dec 08, 2019 Dec 08, 2019

Copy link to clipboard

Copied

I think XMP is a "good" EXTRA backup strategy.  I do not have mine set to write continuously but have a couple smart collections of "Recently Modified" and select the photos and do an XMP save once in awhile.   The XMP saves your Master Edits (the hardest thing to recreate) and Keywords which are the most important 2 things to me.  If the XMP would save Virtual Copy edits like it does Snapshots than is would be a way better backup but I just "Export as Original" with a standard rename of my final virtual copies (not test ones) to solve that issue.  There is at least one thread on the forum where Adobe acknowledged one release of Lightroom was messing up the Backkups.  That being said, I think the catalog is fairly reliable but you can't have too many backups.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

Another vote from me for the facility to use sidecars with DNG files. I've just opened my main catalog and had a noodle around with a few images for a few hours. Now the backup is re-backing up the changed dngs, over 200Gb by the looks (probably more than one editting session there by the looks). Clearly the sidecars would be nowhere near 200Gb.....

 

And honestly, I can't see any good reason why the option would not be included in the software to allow the customer to make an informed decision which fits their backup workflow.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

You need to tell Adobe, not us other Lightroom Classic users in this forum, because we can't change the software. The proper link to do that was given above.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

@dj_paige - Yep, appreciate that.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

If your DNG files have been changed by your latest 'noodling' in Lightroom, then I recommend that you check whether 'Automatically write chnages to xmp' is enabled in Catalog Settings. When this option is disabled nothing gets written to you DNG files unless you intentionally save the changes to files (individually or in bulk).

 

auto-write-xmp-1.png

 

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

@Ian_Lyons - Yes, I understand that and want the changes to be recorded, it is the manner in which this is performed that I'm questioning. 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

Unless your camera shoots DNG than why are you converting to DNG if you want XMP.  

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

@Bob_Somrak - Way back when it wasn't clear that raw formats would ever have any longevity of support, converting to dng was one way of making the potential problem go away. The dng files I have in my catalog are probably over 10 years old for the most part. I haven't converted to dng other than round trips to other raw editors for a number of years. 

 

However, I occasionally revisit edits and look at my old images to see where thry can be improved. These are the sorts of images I've been playing with and obviously when you shut LR down and it says "not finished writing to xmp" and you ignore it, a few times in a row, rthis is what you get!

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

And sidecars for JPEGs and TIFs and PSDs and any other publicly-documented formats that have no need of sidecars? And how are LR and other apps supposed to decide whether to read metadata that is stored inside the file or metadata stored in a sidecar that the file format doesn't need? What is the backup value of storing only some of your LR work in such a sidecar versus all of it being backed up when the catalogue is backed up? Etc. In other words, if you use DNG, you fine tune your backup procedures.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

Hi @John_Beardsworth,

I've seen this argument on other posts in here. Assuming "write to xmp" is ticked, it would be perfectly simple to build logic into LR to write an xmp file and use a new switch, "use xmp for dng" to direct LR behaviour. This way the user can decide if an xmp or embedded data are used. 

 

My catalogues are all backed up too, and I appreciate it is easy to move images from one catalogue to another but cannot see why there seems to be such opposition to this simple change in behaviour if it has the potential to make users lives easier. While we're at it we could use the same switch to create xmp options for the other formats listed. Same backup problem occurs.

How would you suggest I circumvent the issue of no xmp files for dngs with regards changing my backup process, interested to see how this would work. Just backup the original dng and allow changes to be kept in the catalogue?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

No, because of those issues I listed it would not be perfectly simple, and the advantage of DNG and other publicly-documented formats is they can be safely written. Sidecars are not the norm but are an exception for file formats that aren't documented. A classic DNG workflow is to routinely back up the catalogue and the DNGs once, and the latter can be accomplished by not writing xmp automatically or by adjusting which drives/folders your backup software targets. For example, it might target drive #1 which is where you always import files, and once they're backed up you would move them to another drive which isn't continually targetted by the software.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

Cheers John, while I appreciate the dng format is publicly documented, I think you'd agree that not all dngs are created equal! And I'm still not convinced that this change to LR would be difficult or a bad idea, I guess we will have to agree to disagree here. 

Backup can be adjusted to just backup the dngs once but, honestly, given that I don't mess with old images that often, I'd prefer to know that everything on my server was the same as the two copies I keep at home, less confusing and I know where I stand with restores. 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

ALL DNG's like TIFF (or any other documented file format) should be created equally. 

IF they are not, then those not creating the DNG to spec is at fault, it's a bug and up to that company to follow the spec. 

Case in point is Epson's Print Layout. It doesn't correctly support TIFF with compression. It should. That's Epson's bug. I can find nearly 2 dozen applications that support this kind of TIFF but this utility can't. And I've reported it as a bug. It isn't Adobe's fault. It's Eposn's. 

Author “Color Management for Photographers" & "Photoshop CC Color Management/pluralsight"

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Aug 18, 2020 Aug 18, 2020

Copy link to clipboard

Copied

Agreed, this is not an Adobe issue but it does highlight how, even with a publicly published format, corners are cut!

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines