P: Store the xmp metadata outside DNG, jpeg etc file to be backup efficient

"If you adopt DNG, you also need to review your backup procedure."

I don't think it is a raw developers prerogative to tell the user to change his backup strategy. I have a very robust backup strategy in place, and what you're suggesting would mean switching to a much less robust model. Most importantly, if a file that has already been backed up is changed, it should be backed up again, since otherwise there is no backup of the changes to this file!

Then again, if you like to do your backups differently, then you should be able to do so. This decision should not be forced upon you by the developer of some piece of software you use.

Regarding locked DNG files and XMP sidecars, I apologize, I indeed misremembered. Photoshop/ACR will write changes to XMP if the DNG is locked; Lightroom will not. See https://diglloyd.com/blog/2018/20181226_1738-locking-DNG-files.html for example. Although this still doesn't change the fact that the capability to use XMP files in addition to the catalog is already there in Lightroom, as it is used with all proprietary raw files when the "Automatically write to XMP" option is enabled for the catalog. And again, anyone who dislikes the effects of a hypothetical "Force XMP for all file types" option could leave it disabled and remain entirely happy.

“See my comment above. This whole discussion presumes you have "Automatically write changes to XMP" enabled for your catalog. Otherwise locking DNG files will of course not cause LR to produce XMP files since all changes will be written to the catalog”

Changes are always written to the catalog, even if you have this option turned on. Writing changes to XMP is additional, it does not replace the catalog entry. That may explain why this does not work in Lightroom, while apparently it does work in Bridge.

No, even with automatic writing enabled, LR does not create sidecars for locked DNGs. You must be thinking about Bridge or some other app.

xmp sidecars are intended only for file formats which aren't publicly documented. They aren't the norm and wouldn't exist unless they were necessary for one exception - proprietary raw files.

If you adopt DNG, you also need to review your backup procedure. Clearly it's not efficient to keep backing up the DNGs when some keywords were added to the files. In any case, this backup omits other work you may do in LR. Instead, consider a backup model that covers new files when they are created, and the catalogue on a daily or better routine.

@188115
See my comment above. This whole discussion presumes you have "Automatically write changes to XMP" enabled for your catalog. Otherwise locking DNG files will of course not cause LR to produce XMP files since all changes will be written to the catalog. Although you can still prevent the backup problem that way since LR (or ACR, for that matter) won't have write access to your DNGs anymore.

As I said, this should be an option (and off by default, I might add). And anyone who thinks their workflow might be impacted by it should leave it off.

For me and many others, this is about how Lightroom (or ACR) handles raw files. All non-DNG raw files get XMP sidecar files and are never touched by LR/ACR. (Modification dates on the raw files remain unchanged, so if your file is from 2005, its modification date will likely still be something from 2005.) In Lightroom, any operations on those raw files are recorded either inside the catalog file or in XMP sidecar files (depending on whether you have the "Automatically write changes into XMP" option enabled or not). I don't use Photoshop, but I presume in ACR the same applies without any such option since there is no catalog.

Treating proprietary raw files this way means they are preserved in their original state. And this is what many want for DNG raw files, too, because otherwise we get a huge backup burden for tiny, metadata-only changes. (Nowadays we have individual raw files that can be larger than 100 MB. If those are DNG, add one keyword and you have to re-backup those > 100 MB as a result.)

To be clear, this does not concern any files exported from Lightroom/Photoshop. Those will always have their metadata properly embedded. And if you use Lightroom and you're working with JPEGs or TIFFs, you'll want to use the export function for distribution anyway since the original files will not contain your actual edits; so I don't see how that would be a problem. (But if so, as I said, you'd still be good by just leaving the hypothetical force-XMP option disabled.)

"if you lock a DNG file in Finder, Lightroom immediately starts writing metadata changes to an XMP file for that DNG"

Oh no it doesn't!

So if the sidecar and the DNG/TIFF/JPEG have different data than the file, which is used? Lots of programs can edit common file formats, and none of them would know about the sidecars. This would be a HUGE change that would involve not just DNG/RAW files. How does Facebook handle a JPEG with sidecar, for example?

Yes, Adobe, please provide an option to force creation of XMP sidecar files for ALL file types, including DNG. I just have to change or add one single keyword for a larger number of photos (an operation which may not take more than a split second), and because they are DNG, instead of a few megabytes (at most) my next backup is hit with dozens or even hundreds of gigabytes of data which is getting needlessly re-copied.

The capability is already there (at least on Macs): if you lock a DNG file in Finder, Lightroom immediately starts writing metadata changes to an XMP file for that DNG, just like it would with a proprietary raw file. Unfortunately, locking DNGs manually is not remotely practical for normal usage; but it shows it must be nearly trivial to offer such an option. So, please, please do so.

XMP has a tag that tells what filename is linked to that sidecar.