Participant

Open for Voting

P: Store the xmp metadata outside DNG, jpeg etc file to be backup efficient

Forum|Forum|11 years ago
October 8, 2014
52 replies
5620 views

Now a days we can it's very easy to backup datas on clouds , amazon, synology etc.But with lightroom and dnd, jpeg, etc the xmp metadata are stored inside the file and not outside like c2r, pef,nef raw file.So each time we modify a small things the whole big file is modified and need to be uploaded insteed of a small xml kilobyte file that are backup friendly. Upload terabyte on internet or local network contains errors and it's took a lot of time to checks backup.The actual solution is a monolitic outdated and ineffecient purpose.Please add this feature into our image favorite software.A simple workaround can be to place the xmp outside when the file is in write only to be fully non destructive.

Show previous replies

P

Penalaak

Participating Frequently

Hi!

I mainly take photos in raw-format (.nef), but some photos are also JPGs.

I want to use setting "save changes to xmp" to have original file unchanged.

Lightroon Classic cc do not create xmp-file when the processed file is jpg-file but stores changes directly to the original file. That means after the change I don't have original file. I store my files (nef and jpg) to monthly folder.

If have been thinking how to keep also original jpg-files unchanged. I got idea to this workaround. If I set jpg-file to state "read only" LR can not store changes to original file but still changes are stored to my LR catalog. I use powercript to rename thotos and it is easy to set jpg to "read only". LR warns me that changes are not store with exclamation mark, but I can accept it with jpg-files. Using LR I can see still see cahanges I have made to the photo and when I export the photo changes and keyworts etc are storet to the new file.

I use "write to xmp" for the reason if my catalog file is destryed and I can not restore it I have still my raw-file changes also in original + xmp. In my solution this works only in raw-files, not in JPGs, but I can accept it to have still original jpg-file.

Do you see drawback in this workaround I'm describing above?

Do you have other solutions to keep original jpg unchanged except back up it to somewhere before you touch it with LR?

I try to keep raw and jpg handling as similar as possible in back up and storing.

BR

penalaak

PS: Do you know the idea not to have also xmp-file solution with jpg-photo?

R

richardplondon

Community Expert

You may want to turn off the option to write your Develop parameters to JPG files, while continuing to write out keywording etc. That seems, to me, to give the best of both worlds and reduce how frequently JPGs (or, more significantly, larger bitmap files) will get modified on disk and therefore need to be re-backed up. XMP files for Raw are small, and quick to write to disk as well as to back up - so writing Develop changes out to Raw as you work, is a good idea with little practical consequence IMO.

The irreplaceable original JPG picture content is not changed when LrC metadata is written, of whatever type. If you never write your Develop adjustments to the JPG file, then when imported to some different Catalog it would arrive showing no remembered edits. But even if it did, it is easy to Reset all editing.

It is routine for images to have their internal metadata changed. Even when you just rename a file its Modification Date alters - and that is a real change to the file's contents (albeit, only to its metadata). A Raw file on disk is not invulnerable just because you aren't writing XMP into that, you still need to backup data regardless. So the above suggestion is really focused on pragmatic risk and efficiency.

P

Penalaak

Participating Frequently

Hi!

Thanks richardplondon!

The feature you suggested was new to me.

It affects too to psd, tiff which I use too but not very much.

I'll test how it works and solves jpg-problem.

BR

penalaak

Todd Shaner

Legend

Merging this post to the older exisiting post with the same request.

U

ujay68

Known Participant

Camera Raw has an option always to write XMP sidecar files, also for DNG files. Lightroom at the moment can only either not write XMP at all, or, if it does write XMP, embed it in DNG, modifiying the DNG. Embedding XMP in DNG is painful both in terms of backups and for workflows that rely on sidecar XMP data. See also this thread: https://community.adobe.com/t5/lightroom-classic-discussions/prevent-lightroom-from-writing-to-dng-files/m-p/12644677#M259671

Todd Shaner

Legend

Camera Raw has a Preferences setting to create XMP sidecar files for DNG files. It shouldn't be too difficult to add this capability as a LrC Preferences option. This would speed DNG file backup updates after editing sessions since only the small XMP file has been changed.

Erik Bloodaxe

Legend

If I set my Camera Raw preferences for DNG file handling as in this image

then all my edits are atored in XMP sidecars and backing up take just seconds as the DNG files are not overwritten when any further edits are made.

debertek

Participating Frequently

Hi,

Please add an option in Lightroom Classic to save .xmp sidecar files for all file formats (including JPEG, TIFF, PNG, PSD, and DNG). I love the way the "Automatically write changes into XMP" option works with my ARW and CR2 files, leaving the original files intact, and I'd very much prefer leaving my scans (mostly in TIFF) and cellphone images (in JPEG) intact as well.

dj_paige

Legend

@debertek Please tell this to Adobe at https://feedback.photoshop.com/photoshop_family/categories/photoshop_family_photoshop_lightroom

We are not Adobe in this forum, we are just other Lightroom Classic users who cannot modify the program.

UPDATE: looks like that link is no longer going to work, according to https://community.adobe.com/t5/lightroom-classic/bd-p/lightroom-classic?page=1&sort=latest_replies&filter=all. On August 31, there will be a new place for feature requests as explained at the link.

debertek

Participating Frequently

Hi @dj_paige, those sites were locked yesterday, and now this forum is the official place for everything.

R

Robert_Ripps

Known Participant

If you want LR to preserve the file date-created, add your constructive opinion to this long-standing feature request:
https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of_image_files

I thought I was by posting to this forum, over and over...;).

To understand why Adobe doesn't find it important to preserve the file date-created, see two of my posts in that topic:

I think I understand what you are saying & why Lightroom does what it does, but I don’t like it. However, I am not an engineer, or even a software expert. I am a photographer who is relying on some basic software tools to be able to sort some of my cataloged images (jpegs) based on the dates the jpeg was created, not the date I changed a rating on that image. I need a simple way to preserve the date I created that jpeg (and not the earlier date an underlying image used in that master Photoshop file that this jpeg was saved from), and I need Lightroom to be able to sort by the date I created the jpeg from my PSD file- seems simple enough. When the modification date the OS presents to me, and to Lightroom is consistent with the date I created the psd, then saved as a jpeg, it doesn’t matter if the actual file is really an identical copy that underneath has a new creation date.

If Adobe doesn’t think preserving the creation date of the original file and writing in the new file so it can be read by the MacOS, but instead thinks that the most recent modification date should be displayed as the creation date is the way to go, even though the real creation date is saved somewhere in the file/OS, then why doesn’t Lightroom offer a way to see that date, and use it to sort by date without me having to modify things to change dates? Seems unfair- Lightroom can change the creation date when it wants, but then it forces you to use that date when sorting your files, even though it knows it is not the real creation date. Also, the capture date of a raw image is not necessarily the creation date of a jpeg from an image created in Photoshop- to me, the creation date is the date that I saved a jpeg from a finished psd file, not the dates of one of the images used to create that Photoshop file.

Why do changes in xmp files, which seems the same principal as changes to jpegs, not change the creation date as well? And why do Photoshop & Bridge not behave this way (from what I have read in this thread)?

And, if Lightroom offered a way to write changes to xmp files only, AND not write them to jpegs or dng files, this would not be an issue.

I will try you suggestion to set the captured time to the file creation time when I import the next batch of jpegs and see if that works for my issue.

E

edmundgall

Inspiring

I think the guys at Adobe (and anyone who knows how incremental cloud backups work) understand what you're asking for, David Terry & Vinelander. DNGs operate like a mix of JPEG with raws - you get more pixel-related data than a JPEG (but less than a raw) and metadata is stored within the file (just like JPEGs). The apparent trade-of was always there: you're saving local disk space but if you use cloud backups, then bandwidth usage will increase as changed DNG data gets transferred after any modification (in addition to changed LR catalog data).

I want to make 3 points that haven't been mentioned.

First, if I understand David's and Vinelander's posts above (and Benoit Malrat's original post 5 years agoe), the concern is that if a simple metadata change (e.g. change in flag rating) is made, they think the entire DNG is re-uploaded to their cloud backup. Thus, they conclude that by transferring metadata from DNGs to sidecar XMPs, the amount of data that gets re-transferred to cloud backups drops significantly. I recommend that they confirm with their cloud backup service provide whether that's the case today (as Richard Hess hinted above both 3 years ago and yesterday).

For e.g., I use Crashplan who use a block-based method for their incremental backups. Basically, a file occupies a set of blocks on disk; blocks are a much smaller unit of size (e.g. a 1MB file could comprise 1000 blocks if each block is 1KB in size). When Crashplan finds a new file, it copies the entire file to cloud storage. After that, it uses incremental backups: Crashplan compares each block for that file on local disk with that already within the cloud backup, and only transfers the blocks that were changed – not the entire file. This is why, even after a long editing session, Crashplan does not need to re-transfer my entire multi-GB LR catalog file to cloud backup. See: https://support.code42.com/CrashPlan/6/Backup/How_backup_works

If you don't use Crashplan, see if your service provider's website has a similar explanation, or contact their support team. What this means is – provided LR does not recreate DNG files for the simple changes you're concerned about (and I think it doesn't) – your assumption that the entire DNG file is re-transferred to cloud backup may not be accurate. If your cloud backup does not use a similar bandwidth-efficient means for incremental backups, then perhaps you need a new service provider.

My 2nd point is to consider re-evaluating your use of DNG (rather than changing the way DNGs work). With storage prices dropping each year per unit, if you are in some type of crazy Internet service plan that still limits your bandwidth and charges for excess, then – as counterintuitive as it may sound – maybe your overall cost for storage, backup and Internet would reduce if you switch back to raw files (and/or get a better Internet plan).

My 3rd point is, if you've re-evaluated and it's still better for you to stick with DNG, then you can also check when you convert to DNG in your workflow. Some years back when I was consuming as much training about LR before adopting it, a highly-experienced photographer explained he used DNG at the end of his workflow as part of his long-term archiving strategy. In other words, he used proprietary raws during his initial stages of editing a photo. Once he was sure he won't need to do any more processing, then he'd convert the image to DNG, link the DNG to the editing metadata in LR (by swapping the raws with DNGs) and then move that folder to his archive storage (NAS, cloud or whatever). So, if you're concerned about the Internet/backup overhead from modifying DNGs, maybe you can re-examine whether you can convert to DNGs much later in your workflow.

Hope this helps...

[ As a side note, if Adobe enables the storage of metadata within XMPs instead of DNGs, they have to consider the wider consequences. For e.g., there would be additional administrative overhead when Adobe was aiming for simplification (even if implemented via an optional control in Preferences as John Ellis mooted above). The more 'moving parts' you introduce into the workflow (i.e. an image's data is now spread over the catalog, DNG and sidecar), the greater the risk of something going wrong. You may think that's fine for you because you're an experienced user with stronger technical nous than the average customer, but it can create work (read: trouble) for inexperienced users who just need a solution that reduces their local storage usage (relative to proprietary raw formats) and provides a backup (albeit imperfect one) to what's stored in the LR catalog in a single file. If I look back at the changes Adobe made over the last five years, they're targeting more non-professionals: so a lot of the complexity has to be hidden or removed. You read this forum for a year and you'd realise how many newbies get into trouble because they just jump straight into using LR without any training, because Adobe's marketing makes them think it's as simple as iPhotos. Any new feature we customers ask for has to be looked at through this prism. ]

E

edmundgall

Inspiring

Frankly speaking, I don't think John is the person you need to convince, folks - it's the guys at Adobe. So, if John or any non-Adobe staff doesn't understand your points, that's okay...

john beardsworth

Community Expert

"But besides that, I am not pleased at risking my original archival "negative" files being written to on a regular basis."

Sure, and in a DNG workflow, when you choose to write only some of your work to the files, the DNGs recorded in Lightroom gain a more cache-like status. In any case, their backup value is diminished because of xmp doesn't contain large areas of the work you've done on them in LR. The original archival "negative" files are the copies which backed up when the DNGs were first created.

"Lightroom should behave nicely with common and ubiquitous consumer-level backup solutions. "

And I think that's done better by other means. For example, incremental backup of the catalogue / transaction logs would not break Adobe and third party standard practices for metadata location.

johnrellis

Legend

"I still wish there was a way to make changes to a file’s rating or flag, without Lightroom changing the files CREATION DATE. I understand that it will write the changes in a rating to the jpeg file (although I am not happy about that) which changes modification date, but why does Lightroom change the creation date as well?"

If you want LR to preserve the file date-created, add your constructive opinion to this long-standing feature request:
https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of_image_files

To understand why Adobe doesn't find it important to preserve the file date-created, see two of my posts in that topic:

https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of...

https://feedback.photoshop.com/photoshop_family/topics/bug_report_lightroom_changes_creation_date_of...

Show more replies

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded