P: Store the xmp metadata outside DNG, jpeg etc file to be backup efficient

36 Votes
Community Beginner ,
Oct 08, 2014 Oct 08, 2014

Copy link to clipboard

Copied

Now a days we can it's very easy to backup datas on clouds , amazon, synology etc.But with lightroom and dnd, jpeg, etc the xmp metadata are stored inside the file and not outside like c2r, pef,nef raw file.So each time we modify a small things the whole big file is modified and need to be uploaded insteed of a small xml kilobyte file that are backup friendly. Upload terabyte on internet or local network contains errors and it's took a lot of time to checks backup.The actual solution is a monolitic outdated and ineffecient purpose.Please add this feature into our image favorite software.A simple workaround can be to place the xmp outside when the file is in write only to be fully non destructive.

Idea No status
TOPICS
macOS , Windows

Views

821

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
73 Comments
LEGEND ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

Do you have a recommendation on a xmp capable app to view the JPEG with as a test?

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

Photoshop works. It will open it through camera raw if there is xmp meatadata.

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied


@debertek wrote:

Hi @dj_paige, those sites were locked yesterday, and now this forum is the official place for everything.


Indeed, you can make requests now this way using the modules outlined below:

From Adobe (community forums):
Currently, the community on community.adobe.com is used as a destination for customers to ask questions
and engage in peer-to-peer conversations. IOW, this is a user to user support forum.

Beginning August 31st, 2021, migrated content will be available on community.adobe.com and the Adobe
Community team will introduce two new modules - IdeasandBugs- for the Photoshop ecosystem, Lightroom ecosystem, Lightroom Classic and Camera Raw and DNG. This will allow customers to report bugs and/or submit feature requests directly to the product teams.

Author “Color Management for Photographers" & "Photoshop CC Color Management/pluralsight"

Votes

Translate

Translate

Report

Report
LEGEND ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

How  about a non Adobe App, that when it opens a JPEG file that has edits in LrC, not exported, shows the JPEG with LrC edits. Yes PS does that, but what App, that say a non photographer would have, does that?

 

Currently running a test as to learn?understand  this metadata saved to jpeg issue.

 

Took new Photos, camera set to RAW plus JPEG. Copied both files onto seperate folders (on purpose as to keep RAW and JPEG apart). Imported both via ADD into KrC, no presets, and standard Adobe Develop  RAW setting in preferences. And Save metadata to xmp selected. After Import, verified the two files existed and had no xmp companion at this point (i.e. nothing weird). Made a change to B&W, no exports. Exited LrC, verified that no xmp file was created for the JPEG, and one did exist for the RAW. 

 

Then, I looked at the JPEG using various Apps, Yes PS shows B$W, interestingly DxO does not. Landscape Pro does not. fairly much everything I have that does not say Adobe, does not.

 

PS yes

CorelmDraw-PaintX5, not

DxOmPhoto Lab, not

LandscapeStudio Pro, not

Paint (Windows) not, no suprise

Photos (Windows), not, no suprise

Paint 3D, not (just what the heck is that one)

Snip & Sketch (Wwindows), not. no ssuprise

Portrate Apro Studio, not

 

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

No non-Adobe app will do that. This is completely Adobe specific. I think this will only work for dng files if you have updated the embedded preview. For jpeg, tiff, etc. everything non-Adobe will ignore the xmp instructions since they are camera raw specific. 

Votes

Translate

Translate

Report

Report
LEGEND ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

Ok, I am learning something new, and clearly needed to.

And, I now understand the issue a lot better.

 

Only workaround I can see is to use virtual copy's, leaving original raster images alone.

 

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

The important XMP (edits) are all proprietary. Something like "Saturation +10" is utterly meaninless outside Adobe's software and vise versa. 

Frankly, I don't see the point of all this. Parametric edits (proprietary) are useful for raw processing. For rendered images, why not simply process and bake the edits? Now you pass this non baked edit elsewhere or even Adobe without proper prefs, the edits are never seen. 

Author “Color Management for Photographers" & "Photoshop CC Color Management/pluralsight"

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

The issue is not sharing files with others, it is that if you have automatic xmp writing turned on, every time you make a change in Classic, the original file is rewritten and marked as updated. So your backup solution will copy the entire file over instead of a small xmp sidecar. This increases backup time (and size!) enormously. All this could be avoided if Classic could write xmp in sidecars instead of in the file's metadata area for jpeg, tiff, dng, etc. making backups much quicker if you use xmp automatic writing. That's all the original question is about I think.

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

Yes, I understand they trigger for a backup, I'm fine with that. All my backups take place automatically, usually in the middle of the night. Smaller size of my data isn't an issue for me (and in all cases, I'm speaking personally for myself). HDs are cheap. Cloud storage which I use too, isn't expensive either. 

Old saying: You have to pay me now or you have to pay me later. At some point, the edits need to be burned into the JPEG or TIFF etc, and hopefully, you'll back up that data. 

Sidecars from Adobe were designed for one reason only: Adobe will not write them to proprietary raws. And that isn't an issue with DNG. In fact, DNG allows many useful attrubites like a big, pretty high quality JPEG of the rendering inside it, so  if worse comes to worse, you have that too. 

Saving out XMP just to save backup, well I suppose for some, that be useful but I highly doubt Adobe will ever do this based on their design of XMP sidecars from day one. 

Author “Color Management for Photographers" & "Photoshop CC Color Management/pluralsight"

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

If you're non-destructive, do not touch my files! 😉 I never use automatic xmp writes for this reason. I really dislike that it touches my originals and rewrites them.

 

Anyway, camera raw/Bridge has been able to do this for ages:

Screen Shot 2021-08-25 at 12.06.41 PM.png

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

I agree with Jao vdL’s workflow I do not use XMP. Just do not consider it necessary to include in my workflow. I use Lightroom since 2007 so have no need for Adobe Camera Raw.

 

Regards, Denis: iMac mid-2015, 5K 27”, GPU 2GB, Ram 24GB, HDD 3TB, macOS 11.6.8,; LrC 11.5, Lr 5.5, Ps 23.5.1, ACR 14.5,; Camera OM-D E-M1.

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied


@Jao vdL wrote:

Anyway, camera raw/Bridge has been able to do this for ages:


For TIFFs and JPEGs? 

If so, the dialog you show and what Adobe describes is rather unclear:

https://helpx.adobe.com/camera-raw/using/camera-raw-settings.html

Specify where Camera Raw settings are stored

Choose a preference to specify where the settings are stored. The XMP files are useful if you plan to move or store the image files and want to retain the camera raw settings. You can use the Export Settings command to copy the settings in the Camera Raw database to sidecar XMP files or embed the settings in Digital Negative (DNG) files.

When a camera raw image file is processed with Camera Raw, the image settings are stored in one of two places: the Camera Raw database file or a sidecar XMP file. When a DNG file is processed in Camera Raw, the settings are stored in the DNG file itself, but they can be stored in a sidecar XMP file instead. Settings for TIFF and JPEG files are always stored in the file itself.

Author “Color Management for Photographers" & "Photoshop CC Color Management/pluralsight"

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

That is correct. The setting only works for dng but you can't even do that in Classic. This is another reason I don't use dng except to exchange working copies and for raw files coming from cellphones and of course for merged HDR and pano images that Lightroom creates itself. The fact that Lightroom has no qualms writing into dng originals is one reason for me not to use dng files for my originals if I can avoid it which is a pity as dng is the better format. I would be happy if classic just supported sidecars for dng in the metadata it writes in automatic or command-S.  It just should never touch the orginals if that can be avoided and xmp sidecar files are easy ways to avoid rewriting the file. DNG files from panos can get to be 100's of megabytes to even a gigabyte and I don't want that getting rewritten and copied to a backup all the time. For tiff, they run into gigabytes with just a few layers surprisingly quickly. Those should not get rewritten and recopied over to a backup every time I change a keyword (I backup to a NAS, attached storage, and online backup all basically live or every hour). Not being able to use automatic xmp writing is not a big loss for me as I don't rely on the sidecars and do regular backups of the catalog files too but lots of people rely on the xmp writing to backup edits to online backups. The xmp sidecars could be really useful lightweight last resort backups of edits and metadata if they would just make them possible for dng, psd, and tiff.

Votes

Translate

Translate

Report

Report
Community Expert ,
Aug 25, 2021 Aug 25, 2021

Copy link to clipboard

Copied

 


@Jao vdL wrote:

That is correct. The setting only works for dng but you can't even do that in Classic. 



Thanks, that's what I figured. So it doesn't really help the OP. 

LR doesn't have qualms writing to anything but proprietary raws, hence Side-cars. 

Big files are big files; gotta back them up if you don't want to lose them. Been this way for at least a few decades before LR and ACR existed. 

If you really want to edit a rendered image, you have to touch it. And it gets backed up if you allow that. 

Frankly, I draw a line in the sand between parametric edits of raw, then anything that gets rendered (and then edits burned, layers or otherwise) in the document. But again, that's just me. 

Author “Color Management for Photographers" & "Photoshop CC Color Management/pluralsight"

Votes

Translate

Translate

Report

Report
LEGEND ,
Aug 31, 2021 Aug 31, 2021

Copy link to clipboard

Copied

This thread should be merged with this longstanding Idea (feature request):

https://community.adobe.com/t5/camera-raw-ideas/lightroom-camera-raw-store-the-xmp-metadata-outside-... 

Votes

Translate

Translate

Report

Report
Enthusiast ,
Sep 02, 2021 Sep 02, 2021

Copy link to clipboard

Copied

If I set my Camera Raw preferences for DNG file handling as in this image

then all my edits are atored in XMP sidecars and backing up take just seconds as the DNG files are not overwritten when any further edits are made. Camera Raw Preferences - File Handling.jpg

Votes

Translate

Translate

Report

Report
Explorer ,
Jan 16, 2022 Jan 16, 2022

Copy link to clipboard

Copied

Camera Raw has an option always to write XMP sidecar files, also for DNG files. Lightroom at the moment can only either not write XMP at all, or, if it does write XMP, embed it in DNG, modifiying the DNG. Embedding XMP in DNG is painful both in terms of backups and for workflows that rely on sidecar XMP data. See also this thread: https://community.adobe.com/t5/lightroom-classic-discussions/prevent-lightroom-from-writing-to-dng-f...

Votes

Translate

Translate

Report

Report
Community Expert ,
Jan 16, 2022 Jan 16, 2022

Copy link to clipboard

Copied

Camera Raw has a Preferences setting to create XMP sidecar files for DNG files. It shouldn't be too difficult to add this capability as a LrC Preferences option. This would speed DNG file backup updates after editing sessions since only the small XMP file has been changed.

Votes

Translate

Translate

Report

Report
Community Expert ,
Jan 16, 2022 Jan 16, 2022

Copy link to clipboard

Copied

I always thought one of the major reasons for creating the DNG file format was to store xmp data in the file header, now to allow the option to create a sidecar to store the data.

Lightroom was designed to utilize a Catalog File to store the data as an alternative to Camera Raw and Bridge which require the info in the file.

While you are working in LrC it reads and writes to the Catalog File, having the info in the file does nothing to improve the functioning of LrC.

When I decided to adopt Lightroom as my application to work with my raw camera files, it allowed me to forget having to manage sidecar files. The DNG concept was not adopted by me despite trying for a few months on two separate ocassions.

Just my preference.

Regards, Denis: iMac mid-2015, 5K 27”, GPU 2GB, Ram 24GB, HDD 3TB, macOS 11.6.8,; LrC 11.5, Lr 5.5, Ps 23.5.1, ACR 14.5,; Camera OM-D E-M1.

Votes

Translate

Translate

Report

Report
Explorer ,
Jan 16, 2022 Jan 16, 2022

Copy link to clipboard

Copied

This is one of the reasons why I suggested this as a new option so everyone can pick their preference. For me and probably many others, additional sidecars files that Lr manages automatically would much better than having additional gigabytes to backup after each edit round of a set of DNG files. 

Votes

Translate

Translate

Report

Report
LEGEND ,
Jan 19, 2022 Jan 19, 2022

Copy link to clipboard

Copied

Votes

Translate

Translate

Report

Report
Community Expert ,
Jan 20, 2022 Jan 20, 2022

Copy link to clipboard

Copied

Merging this post to the older exisiting post with the same request.

Votes

Translate

Translate

Report

Report
Community Expert ,
May 27, 2022 May 27, 2022

Copy link to clipboard

Copied

LATEST

Given the fact that ACR has this feature, (for DNGs) shouldn't the post be moved to the LrC features request forum?

Votes

Translate

Translate

Report

Report