P: Store the xmp metadata outside DNG, jpeg etc file to be backup efficient

[Yalis0]

Now a days we can it's very easy to backup datas on clouds , amazon, synology etc.But with lightroom and dnd, jpeg, etc the xmp metadata are stored inside the file and not outside like c2r, pef,nef raw file.So each time we modify a small things the whole big file is modified and need to be uploaded insteed of a small xml kilobyte file that are backup friendly. Upload terabyte on internet or local network contains errors and it's took a lot of time to checks backup.The actual solution is a monolitic outdated and ineffecient purpose.Please add this feature into our image favorite software.A simple workaround can be to place the xmp outside when the file is in write only to be fully non destructive.

[GoldingD]

Do you have a recommendation on a xmp capable app to view the JPEG with as a test?

[Jao vdL]

Photoshop works. It will open it through camera raw if there is xmp meatadata.

[TheDigitalDog]

---

@debertek wrote:

Hi @dj_paige, those sites were locked yesterday, and now this forum is the official place for everything.

---

Indeed, you can make requests now this way using the modules outlined below:

From Adobe (community forums):
Currently, the community on community.adobe.com is used as a destination for customers to ask questions
and engage in peer-to-peer conversations. IOW, this is a user to user support forum.

Beginning August 31st, 2021, migrated content will be available on community.adobe.com and the Adobe
Community team will introduce two new modules - IdeasandBugs- for the Photoshop ecosystem, Lightroom ecosystem, Lightroom Classic and Camera Raw and DNG. This will allow customers to report bugs and/or submit feature requests directly to the product teams.

[GoldingD]

How  about a non Adobe App, that when it opens a JPEG file that has edits in LrC, not exported, shows the JPEG with LrC edits. Yes PS does that, but what App, that say a non photographer would have, does that?

Currently running a test as to learn?understand  this metadata saved to jpeg issue.

Took new Photos, camera set to RAW plus JPEG. Copied both files onto seperate folders (on purpose as to keep RAW and JPEG apart). Imported both via ADD into KrC, no presets, and standard Adobe Develop  RAW setting in preferences. And Save metadata to xmp selected. After Import, verified the two files existed and had no xmp companion at this point (i.e. nothing weird). Made a change to B&W, no exports. Exited LrC, verified that no xmp file was created for the JPEG, and one did exist for the RAW. 

Then, I looked at the JPEG using various Apps, Yes PS shows B$W, interestingly DxO does not. Landscape Pro does not. fairly much everything I have that does not say Adobe, does not.

PS yes

CorelmDraw-PaintX5, not

DxOmPhoto Lab, not

LandscapeStudio Pro, not

Paint (Windows) not, no suprise

Photos (Windows), not, no suprise

Paint 3D, not (just what the heck is that one)

Snip & Sketch (Wwindows), not. no ssuprise

Portrate Apro Studio, not

[Jao vdL]

No non-Adobe app will do that. This is completely Adobe specific. I think this will only work for dng files if you have updated the embedded preview. For jpeg, tiff, etc. everything non-Adobe will ignore the xmp instructions since they are camera raw specific. 

[GoldingD]

Ok, I am learning something new, and clearly needed to.

And, I now understand the issue a lot better.

Only workaround I can see is to use virtual copy's, leaving original raster images alone.

[TheDigitalDog]

The important XMP (edits) are all proprietary. Something like "Saturation +10" is utterly meaninless outside Adobe's software and vise versa. 

Frankly, I don't see the point of all this. Parametric edits (proprietary) are useful for raw processing. For rendered images, why not simply process and bake the edits? Now you pass this non baked edit elsewhere or even Adobe without proper prefs, the edits are never seen. 

[Jao vdL]

The issue is not sharing files with others, it is that if you have automatic xmp writing turned on, every time you make a change in Classic, the original file is rewritten and marked as updated. So your backup solution will copy the entire file over instead of a small xmp sidecar. This increases backup time (and size!) enormously. All this could be avoided if Classic could write xmp in sidecars instead of in the file's metadata area for jpeg, tiff, dng, etc. making backups much quicker if you use xmp automatic writing. That's all the original question is about I think.

[TheDigitalDog]

Yes, I understand they trigger for a backup, I'm fine with that. All my backups take place automatically, usually in the middle of the night. Smaller size of my data isn't an issue for me (and in all cases, I'm speaking personally for myself). HDs are cheap. Cloud storage which I use too, isn't expensive either. 

Old saying: You have to pay me now or you have to pay me later. At some point, the edits need to be burned into the JPEG or TIFF etc, and hopefully, you'll back up that data. 

Sidecars from Adobe were designed for one reason only: Adobe will not write them to proprietary raws. And that isn't an issue with DNG. In fact, DNG allows many useful attrubites like a big, pretty high quality JPEG of the rendering inside it, so  if worse comes to worse, you have that too. 

Saving out XMP just to save backup, well I suppose for some, that be useful but I highly doubt Adobe will ever do this based on their design of XMP sidecars from day one. 

[Jao vdL]

If you're non-destructive, do not touch my files! 😉 I never use automatic xmp writes for this reason. I really dislike that it touches my originals and rewrites them.

Anyway, camera raw/Bridge has been able to do this for ages:

[DdeGannes]

I agree with Jao vdL’s workflow I do not use XMP. Just do not consider it necessary to include in my workflow. I use Lightroom since 2007 so have no need for Adobe Camera Raw.

[TheDigitalDog]

---

@Jao vdL wrote:

Anyway, camera raw/Bridge has been able to do this for ages:

---

For TIFFs and JPEGs? 

If so, the dialog you show and what Adobe describes is rather unclear:

https://helpx.adobe.com/camera-raw/using/camera-raw-settings.html

Specify where Camera Raw settings are stored

Choose a preference to specify where the settings are stored. The XMP files are useful if you plan to move or store the image files and want to retain the camera raw settings. You can use the Export Settings command to copy the settings in the Camera Raw database to sidecar XMP files or embed the settings in Digital Negative (DNG) files.

When a camera raw image file is processed with Camera Raw, the image settings are stored in one of two places: the Camera Raw database file or a sidecar XMP file. When a DNG file is processed in Camera Raw, the settings are stored in the DNG file itself, but they can be stored in a sidecar XMP file instead. Settings for TIFF and JPEG files are always stored in the file itself.

[Jao vdL]

That is correct. The setting only works for dng but you can't even do that in Classic. This is another reason I don't use dng except to exchange working copies and for raw files coming from cellphones and of course for merged HDR and pano images that Lightroom creates itself. The fact that Lightroom has no qualms writing into dng originals is one reason for me not to use dng files for my originals if I can avoid it which is a pity as dng is the better format. I would be happy if classic just supported sidecars for dng in the metadata it writes in automatic or command-S.  It just should never touch the orginals if that can be avoided and xmp sidecar files are easy ways to avoid rewriting the file. DNG files from panos can get to be 100's of megabytes to even a gigabyte and I don't want that getting rewritten and copied to a backup all the time. For tiff, they run into gigabytes with just a few layers surprisingly quickly. Those should not get rewritten and recopied over to a backup every time I change a keyword (I backup to a NAS, attached storage, and online backup all basically live or every hour). Not being able to use automatic xmp writing is not a big loss for me as I don't rely on the sidecars and do regular backups of the catalog files too but lots of people rely on the xmp writing to backup edits to online backups. The xmp sidecars could be really useful lightweight last resort backups of edits and metadata if they would just make them possible for dng, psd, and tiff.

[TheDigitalDog]

---

@Jao vdL wrote:

That is correct. The setting only works for dng but you can't even do that in Classic. 

---

Thanks, that's what I figured. So it doesn't really help the OP. 

LR doesn't have qualms writing to anything but proprietary raws, hence Side-cars. 

Big files are big files; gotta back them up if you don't want to lose them. Been this way for at least a few decades before LR and ACR existed. 

If you really want to edit a rendered image, you have to touch it. And it gets backed up if you allow that. 

Frankly, I draw a line in the sand between parametric edits of raw, then anything that gets rendered (and then edits burned, layers or otherwise) in the document. But again, that's just me. 

[johnrellis]

This thread should be merged with this longstanding Idea (feature request):

https://community.adobe.com/t5/camera-raw-ideas/lightroom-camera-raw-store-the-xmp-metadata-outside-... 

[Erik Bloodaxe]

If I set my Camera Raw preferences for DNG file handling as in this image

then all my edits are atored in XMP sidecars and backing up take just seconds as the DNG files are not overwritten when any further edits are made. 

[ujay68]

Camera Raw has an option always to write XMP sidecar files, also for DNG files. Lightroom at the moment can only either not write XMP at all, or, if it does write XMP, embed it in DNG, modifiying the DNG. Embedding XMP in DNG is painful both in terms of backups and for workflows that rely on sidecar XMP data. See also this thread: https://community.adobe.com/t5/lightroom-classic-discussions/prevent-lightroom-from-writing-to-dng-f...

[Todd Shaner]

Camera Raw has a Preferences setting to create XMP sidecar files for DNG files. It shouldn't be too difficult to add this capability as a LrC Preferences option. This would speed DNG file backup updates after editing sessions since only the small XMP file has been changed.

[DdeGannes]

I always thought one of the major reasons for creating the DNG file format was to store xmp data in the file header, now to allow the option to create a sidecar to store the data.

Lightroom was designed to utilize a Catalog File to store the data as an alternative to Camera Raw and Bridge which require the info in the file.

While you are working in LrC it reads and writes to the Catalog File, having the info in the file does nothing to improve the functioning of LrC.

When I decided to adopt Lightroom as my application to work with my raw camera files, it allowed me to forget having to manage sidecar files. The DNG concept was not adopted by me despite trying for a few months on two separate ocassions.

Just my preference.

[ujay68]

This is one of the reasons why I suggested this as a new option so everyone can pick their preference. For me and probably many others, additional sidecars files that Lr manages automatically would much better than having additional gigabytes to backup after each edit round of a set of DNG files. 

[johnrellis]

Moderators, @Rikk Flohr: Photography, please merge with the existing Idea:

https://community.adobe.com/t5/camera-raw-ideas/p-store-the-xmp-metadata-outside-dng-jpeg-etc-file-t...

[Todd Shaner]

Merging this post to the older exisiting post with the same request.

[Penalaak]

Hi!

I mainly take photos in raw-format (.nef), but some photos are also JPGs. 

I want to use setting "save changes to xmp" to have original file unchanged.

Lightroon Classic cc do not create xmp-file when the processed file is jpg-file but stores changes directly to the original file. That means after the change I don't have original file. I store my files (nef and jpg) to monthly folder. 

If have been thinking how to keep also original jpg-files unchanged. I got idea to this workaround. If I set jpg-file to state "read only" LR can not store changes to original file but still changes are stored to my LR catalog. I use powercript to rename thotos and it is easy to set jpg to "read only". LR warns me that changes are not store with exclamation mark, but I can accept it with jpg-files. Using LR I can see still see cahanges I have made to the photo and when I export the photo changes and keyworts etc are storet to the new file.    

I use "write to xmp" for the reason if my catalog file is destryed and I can not restore it I have still my raw-file changes also in original + xmp. In my solution this works only in raw-files, not in JPGs, but I can accept  it to have still original jpg-file.

Do you see drawback in this workaround I'm describing above?

Do you have other solutions to keep original jpg unchanged except back up it to somewhere before you touch it with LR?

I try to keep raw and jpg handling as similar as possible in back up and storing.

BR

penalaak

PS: Do you know the idea not to have also xmp-file solution with jpg-photo?

[richardplondon]

You may want to turn off the option to write your Develop parameters to JPG files, while continuing to write out keywording etc. That seems, to me, to give the best of both worlds and reduce how frequently JPGs (or, more significantly, larger bitmap files) will get modified on disk and therefore need to be re-backed up. XMP files for Raw are small, and quick to write to disk as well as to back up - so writing Develop changes out to Raw as you work, is a good idea with little practical consequence IMO. 

The irreplaceable original JPG picture content is not changed when LrC metadata is written, of whatever type. If you never write your Develop adjustments to the JPG file, then when imported to some different Catalog it would arrive showing no remembered edits. But even if it did, it is easy to Reset all editing.

It is routine for images to have their internal metadata changed. Even when you just rename a file its Modification Date alters - and that is a real change to the file's contents (albeit, only to its metadata). A Raw file on disk is not invulnerable just because you aren't writing XMP into that, you still need to backup data regardless. So the above suggestion is really focused on pragmatic risk and efficiency.

[Penalaak]

Hi!

Thanks richardplondon!

The feature you suggested was new to me.

It affects too to psd, tiff which I use too but not very much.

I'll test how it works and solves jpg-problem.

BR

penalaak