P: Store the xmp metadata outside DNG, jpeg etc file to be backup efficient

[Yalis0]

Now a days we can it's very easy to backup datas on clouds , amazon, synology etc.But with lightroom and dnd, jpeg, etc the xmp metadata are stored inside the file and not outside like c2r, pef,nef raw file.So each time we modify a small things the whole big file is modified and need to be uploaded insteed of a small xml kilobyte file that are backup friendly. Upload terabyte on internet or local network contains errors and it's took a lot of time to checks backup.The actual solution is a monolitic outdated and ineffecient purpose.Please add this feature into our image favorite software.A simple workaround can be to place the xmp outside when the file is in write only to be fully non destructive.

[R_L_Hess]

Just because you dislike it, doesn't mean everyone dislikes it. While I've had no problems with copying files recently, I did lose a copy of a WAV file that was copied because metadata was changed and it somehow became truncated. I fortunately found a copy on an older backup.

Let's look at the universe:
Proprietary camera raw: Has sidecar
Standard TIF and JPG files: no sidecar
DNG: no sidecar

What if TIF and JPG and DNG files had side cars? The Adobe products would see them and look there for the metadata.

One of my major concerns is actually about JPGs because some users/devices originate in JPGs so those are your "negatives." It was actually that which caused me to not propagate updates to the backup NAS.

[FeedbackCommunityMember]

"I am not suggesting a less robust model but one which covers all images and all the work."

You have no idea how my current backup solution looks like, what my workflow looks like, how I have Lightroom set up, what other backup needs and software I have, what kind of a system I'm even running etc. pp. You are hardly in a position to judge what would be a good backup solution for me. And neither for many other users out there. So please respect that while you may be perfectly content with how Lightroom interoperates with your preferred backup method, for others it is a constant headache—and one that has a very easy solution, if only Adobe chose to provide a very simple preference setting that would have absolutely no effect on users like you.

And yes, "leave it disabled" is a perfectly sensible argument. That's actually the whole point of options—they provide options! They are there so the user can choose based on his specific needs. I could choose to enable the option to accomodate my needs, you could choose not to enable it to accomodate yours. That would be a very satisfactory solution, unlike the current state that allows your needs to be met, but not mine.

[johnrellis]

Note that LR uses .xmp sidecars for HEIF files (what Apple calls HEIC), even though the format specifies where to store EXIF and XMP metadata inside the files.

[john beardsworth]

I couldn't care less what your current backup solution looks like, but it's clearly not optimized for DNG if you are repetitively backing up files. Flabby backup procedures aren't a reason for adding what isn't just a simple option but one which has other consequences, since LR and other apps will no longer know whether to prefer embedded or sidecar metadata. And while development resources may or may not be a zero-sum game, time spent on an unnecessary option will not be available to work on things one does appreciate. So no, no option thank you.

[R_L_Hess]

John, I don't know what environment you work in, but I would suspect many users here are lucky to be able to run simple NAS hardware and at least some are using arrays of HDDs. The simplest implementation involves using OS-based tools that are available for Windows and Macs that have a proven track record and do not create custom files.

For me, it is important that all the backups are a replica of the tree and accessible to the individual file level without special software.

That works for NAS units and local HDDs and is mimicked in Backblaze and Dropbox.

Most OS-level tools (unlike Backblaze and Dropbox) that I am aware of do not do block level compare/copy, but rather just copy the entire file.

But besides that, I am not pleased at risking my original archival "negative" files being written to on a regular basis.

[kimballisms]

Lightroom should behave nicely with common and ubiquitous consumer-level backup solutions.  Writing metadata changes & develop settings into a DNG that is 20-100 MB per image is not good design and does not play well with others.

Lightroom and the DNG spec do not exist in a vacuum nor in a perfect world.  It's time Adobe create an option for those who want XMP sidecars with DNGs that aren't re-written every time they're edited.

[Robert_Ripps]

DNG files aside, I still wish there was a way to make changes to a file’s rating or flag, without Lightroom changing the files CREATION DATE. I understand that it will write the changes in a rating to the jpeg file (although I am not happy about that) which changes modification date, but why does Lightroom change the creation date as well? That makes no sense! Interestingly, Lightroom does not change the creation date of an xmp file when I make a rating change, only the modification date.

Why can’t Lightroom give us better choices in settings to including not write changes in ratings, flags, etc. to jpegs, while still allowing changes made in Lightroom (or Camera Raw) to raw files to be recorded to the corresponding xmp files?

[johnrellis]

"I still wish there was a way to make changes to a file’s rating or flag, without Lightroom changing the files CREATION DATE. I understand that it will write the changes in a rating to the jpeg file (although I am not happy about that) which changes modification date, but why does Lightroom change the creation date as well?"

If you want LR to preserve the file date-created, add your constructive opinion to this long-standing feature request:
https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of...

To understand why Adobe doesn't find it important to preserve the file date-created, see two of my posts in that topic:

https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of...

https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of...

[john beardsworth]

"But besides that, I am not pleased at risking my original archival "negative" files being written to on a regular basis."

Sure, and in a DNG workflow, when you choose to write only some of your work to the files, the DNGs recorded in Lightroom gain a more cache-like status. In any case, their backup value is diminished because of xmp doesn't contain large areas of the work you've done on them in LR. The original archival "negative" files are the copies which backed up when the DNGs were first created.

"Lightroom should behave nicely with common and ubiquitous consumer-level backup solutions. "

And I think that's done better by other means. For example, incremental backup of the catalogue / transaction logs would not break Adobe and third party standard practices for metadata location.

[edmundgall]

Frankly speaking, I don't think John is the person you need to convince, folks - it's the guys at Adobe. So, if John or any non-Adobe staff doesn't understand your points, that's okay...

[edmundgall]

I think the guys at Adobe (and anyone who knows how incremental cloud backups work) understand what you're asking for, David Terry & Vinelander. DNGs operate like a mix of JPEG with raws - you get more pixel-related data than a JPEG (but less than a raw) and metadata is stored within the file (just like JPEGs). The apparent trade-of was always there: you're saving local disk space but if you use cloud backups, then bandwidth usage will increase as changed DNG data gets transferred after any modification (in addition to changed LR catalog data).

I want to make 3 points that haven't been mentioned.

First, if I understand David's and Vinelander's posts above (and Benoit Malrat's original post 5 years agoe), the concern is that if a simple metadata change (e.g. change in flag rating) is made, they think the entire DNG is re-uploaded to their cloud backup. Thus, they conclude that by transferring metadata from DNGs to sidecar XMPs, the amount of data that gets re-transferred to cloud backups drops significantly. I recommend that they confirm with their cloud backup service provide whether that's the case today (as Richard Hess hinted above both 3 years ago and yesterday).

For e.g., I use Crashplan who use a block-based method for their incremental backups. Basically, a file occupies a set of blocks on disk; blocks are a much smaller unit of size (e.g. a 1MB file could comprise 1000 blocks if each block is 1KB in size). When Crashplan finds a new file, it copies the entire file to cloud storage. After that, it uses incremental backups: Crashplan compares each block for that file on local disk with that already within the cloud backup, and only transfers the blocks that were changed – not the entire file. This is why, even after a long editing session, Crashplan does not need to re-transfer my entire multi-GB LR catalog file to cloud backup. See: https://support.code42.com/CrashPlan/6/Backup/How_backup_works

If you don't use Crashplan, see if your service provider's website has a similar explanation, or contact their support team. What this means is – provided LR does not recreate DNG files for the simple changes you're concerned about (and I think it doesn't) – your assumption that the entire DNG file is re-transferred to cloud backup may not be accurate. If your cloud backup does not use a similar bandwidth-efficient means for incremental backups, then perhaps you need a new service provider.

My 2nd point is to consider re-evaluating your use of DNG (rather than changing the way DNGs work). With storage prices dropping each year per unit, if you are in some type of crazy Internet service plan that still limits your bandwidth and charges for excess, then – as counterintuitive as it may sound – maybe your overall cost for storage, backup and Internet would reduce if you switch back to raw files (and/or get a better Internet plan).

My 3rd point is, if you've re-evaluated and it's still better for you to stick with DNG, then you can also check when you convert to DNG in your workflow. Some years back when I was consuming as much training about LR before adopting it, a highly-experienced photographer explained he used DNG at the end of his workflow as part of his long-term archiving strategy. In other words, he used proprietary raws during his initial stages of editing a photo. Once he was sure he won't need to do any more processing, then he'd convert the image to DNG, link the DNG to the editing metadata in LR (by swapping the raws with DNGs) and then move that folder to his archive storage (NAS, cloud or whatever). So, if you're concerned about the Internet/backup overhead from modifying DNGs, maybe you can re-examine whether you can convert to DNGs much later in your workflow.

Hope this helps...

[ As a side note, if Adobe enables the storage of metadata within XMPs instead of DNGs, they have to consider the wider consequences. For e.g., there would be additional administrative overhead when Adobe was aiming for simplification (even if implemented via an optional control in Preferences as John Ellis mooted above). The more 'moving parts' you introduce into the workflow (i.e. an image's data is now spread over the catalog, DNG and sidecar), the greater the risk of something going wrong. You may think that's fine for you because you're an experienced user with stronger technical nous than the average customer, but it can create work (read: trouble) for inexperienced users who just need a solution that reduces their local storage usage (relative to proprietary raw formats) and provides a backup (albeit imperfect one) to what's stored in the LR catalog in a single file. If I look back at the changes Adobe made over the last five years, they're targeting more non-professionals: so a lot of the complexity has to be hidden or removed. You read this forum for a year and you'd realise how many newbies get into trouble because they just jump straight into using LR without any training, because Adobe's marketing makes them think it's as simple as iPhotos. Any new feature we customers ask for has to be looked at through this prism. ]

[Robert_Ripps]

If you want LR to preserve the file date-created, add your constructive opinion to this long-standing feature request:
https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of...

I thought I was by posting to this forum, over and over...;).

To understand why Adobe doesn't find it important to preserve the file date-created, see two of my posts in that topic:

I think I understand what you are saying & why Lightroom does what it does, but I don’t like it. However, I am not an engineer, or even a software expert. I am a photographer who is relying on some basic software tools to be able to sort some of my cataloged images (jpegs) based on the dates the jpeg was created, not the date I changed a rating on that image. I need a simple way to preserve the date I created that jpeg (and not the earlier date an underlying image used in that master Photoshop file that this jpeg was saved from), and I need Lightroom to be able to sort by the date I created the jpeg from my PSD file- seems simple enough. When the modification date the OS presents to me, and to Lightroom is consistent with the date I created the psd, then saved as a jpeg, it doesn’t matter if the actual file is really an identical copy that underneath has a new creation date.

If Adobe doesn’t think preserving the creation date of the original file and writing in the new file so it can be read by the MacOS, but instead thinks that the most recent modification date should be displayed as the creation date is the way to go, even though the real creation date is saved somewhere in the file/OS, then why doesn’t Lightroom offer a way to see that date, and use it to sort by date without me having to modify things to change dates? Seems unfair- Lightroom can change the creation date when it wants, but then it forces you to use that date when sorting your files, even though it knows it is not the real creation date. Also, the capture date of a raw image is not necessarily the creation date of a jpeg from an image created in Photoshop- to me, the creation date is the date that I saved a jpeg from a finished psd file, not the dates of one of the images used to create that Photoshop file.

Why do changes in xmp files, which seems the same principal as changes to jpegs, not change the creation date as well? And why do Photoshop & Bridge not behave this way (from what I have read in this thread)?

And, if Lightroom offered a way to write changes to xmp files only, AND not write them to jpegs or dng files, this would not be an issue.

I will try you suggestion to set the captured time to the file creation time when I import the next batch of jpegs and see if that works for my issue.

[debertek]

Hi,

Please add an option in Lightroom Classic to save .xmp sidecar files for all file formats (including JPEG, TIFF, PNG, PSD, and DNG). I love the way the "Automatically write changes into XMP" option works with my ARW and CR2 files, leaving the original files intact, and I'd very much prefer leaving my scans (mostly in TIFF) and cellphone images (in JPEG) intact as well.

[dj_paige]

@debertek  Please tell this to Adobe at https://feedback.photoshop.com/photoshop_family/categories/photoshop_family_photoshop_lightroom

We are not Adobe in this forum, we are just other Lightroom Classic users who cannot modify the program.

UPDATE: looks like that link is no longer going to work, according to https://community.adobe.com/t5/lightroom-classic/bd-p/lightroom-classic?page=1&sort=latest_replies&f.... On August 31, there will be a new place for feature requests as explained at the link.

[debertek]

Hi @dj_paige, those sites were locked yesterday, and now this forum is the official place for everything.

[dj_paige]

No, @debertek , this forum is not the place for it. As stated above, there will be a new forum where feature requests should go, which will be available on August 31. 

This forum is for technical support, not feature requests or bug reports.

[JohanElzenga]

There will be a new forum within this community.

[GoldingD]

I love the way the "Automatically write changes into XMP" option works with my ARW and CR2 files, leaving the original files intact,

Automatically saving changes into XMP has nothing to do with non destructive editing. That option just allows you to save xmp files containing much of the edit steps. LrC uses a database, the catalog, to keep track of those edits, it does not need those xmp files. Those xmp files are to share with other people or other programs that cannot use the catalog. You would share your original photo with the xmp file, if the non Adobe software could handle the xmp file, then the person could see the photo with edits.

You can edit RAW, or Raster images in LrC nondestructive. When you Export, that us another issue entirely. Exporting creates a new photo with the edits baked in, no xmp file of value with that.

Most of us do not save the xmp file, it has little value, just takes a bit of space and slows down LrC a bit.

As for  your request for change , a few more days, and you will be able to post to the new community. Reccomend that you provide a lot more context into why you want it, otherwise it might get exactly one vote.

[Jao vdL]

We (i.e. people on this forum) have requested this for many years on the feedback forum and directly to Adobe. Bridge can do it but Lightroom can't. Clearly hasn't been a priority for Adobe.

[GoldingD]

and I'd very much prefer leaving my scans (mostly in TIFF) and cellphone images (in JPEG) intact as well.

The raster files should be left alone, LrC is nondestructive. Now, if when you export those raster files, you select to export the original, as opposed to a copy, then yes, you will change the original, that would be user error.

[debertek]

I'm aware of this; my point is to have all my edits (in an .xmp) next to the original photos. I don't care about the history or anything else that is saved in LrC's library files. I just want to keep my original files, untouched, and a sidecar file containing the changes. This is what LrC does with all file formats, except .jpg, .tiff, .png, .psd, and .dng. I'm aware that XMP data can be saved to these files non-destructively (which LrC does). Still, I don't want my original files to be changed, because it's screwing with my backup solution (I'd prefer syncing a few kilobytes of XMP data when I change something in LrC, not gigabytes of TIFFs).

As there is no real way to use the same LrC library/catalog on multiple computers (and it's even worse if you're using both macOS and Windows), writing XMPs is the only solution that I found viable. (Yes, I could use the cloud version of Lightroom, but I really don't want to pay for ~6 TB of cloud storage.)

[debertek]

The contents of the files still changes; that's my whole point! It's just the metadata, so the data itself stays the same (LrC is non-destructive, as you also noted), but the original file still changes, screwing with my backups. This is the case for JPEG, TIFF, PNG, PSD, and DNG files. All other file types are safe. 🙂

[GoldingD]

What exactly is changing? Were do you see the change outside of LrC?

Try an experiment.

• Take a photo, a JPEG photo, a color photo
• Close LrC if open
• Copy the photo to your hard drive
• Using your OS file manager, look at the photo
• Open LrC
• Bring up the Import screen. Select ADD as opposed to Copy as DNG, or Copy, or Move. Deselect any presets to apply during import. Select the new photo. Import
• In the Develop module edit the photo, make it black and white, or make a drastic change, something that is obviously different from the original
• Do not export.
• Close LrC
• Using your OS File manager, look at the image. Any changes?

Also

Bring your OS file manager back up, navigate to the folder you put the photo in, is the JPEG alone, or is it accompnied by a xmp file?

[debertek]

When "Automatically write changes into XMP" is enabled, all changes are saved in the metadata part of JFIF/JPEG files. If you open them with an app that supports XMP, it will show the photo with the changes applied. If you open them with an app that doesn't support XMP, it will show the "original" photo. JPEGs are never accompanied by XMP files when using LrC to edit them, as LrC writes the XMP data directly in the JPEG files. And that's what I want to avoid, because this way I have to sync huge files whenever I make any change to them in LrC. Using sidecar XMPs for these file formats (the same way as LrC works with most raw formats) would save me from this.

[GoldingD]

Ok, missunderstood the issue.

I see that I need to get off my rear end, crank up my workstation, and accomplish my own expriment. As opposed to remembering things, and looking up on the web. Tyically RAW plus JPEG shooter, but RAW edit only. LrC placing develop module edits into JPEG metadata new to me.