P: Prevent loss of Edit Histories when Reimporting Photos

[FeedbackCommunityMember]

When importing DNGs with stored edits (included XMP data) then the history of the photo just shows "Imported..." instead of the list of edits.

I have a corrupt catalogue. (I did nothing to cause the correction :()
The catalogue contains photos which are not associated to folders in the library module. When I choose "Got to folder in Library module" from the context menu for such photos, nothing happens. I imported them just like any other photos, but somehow the corresponding library folder wasn't created or lost.

I tried synchroning the parent folder but the missing subfolders are not created again.

That's why I decided the only way forward is to create a new catalogue. However, the new catalogue doesn't have any of the edit history. The rendering is OK and I can reset it to see the original version of the photos but I cannot see the edit history anymore.

Why is the edit history not recreated? The essence of it must be available because otherwise the correct final rendering could not be created.

I believe edit histories should be available for JPGs, RAW and DNG files. When I decided to use DNG files vs RAW files with sidecar (XMP) files, I didn't know that I'd lose the history with a fresh import of a DNG file. I suppose that if I had XMP files, I could copy these and still had my edit histories.

[Lee_Jay]

"And you don't even need to store the differences in the XMP - LR could already compute them from the adjustment values already in the XMP."

That's not true, John. Remember, due to the Camera Raw defaults and the application of presets, the starting point isn't always fixed.

[john beardsworth]

Application of presets? You mean upon import? I'd say they form part of the difference. But that's arguable. Some will want them in, some out, and as there's not a lot of value from having simplified history....

[Lee_Jay]

We have a simplified history already. It just doesn't look like it. The numbers are cumulative so they aren't quite the same as those in the PS history palette.

[john beardsworth]

Not as simple as Dan is suggesting though, and I wouldn't want Photoshop history to be replicated in Lightroom (no, don't anyone suggest a history brush!!!)

[dtull]

@John I think we're confusing two things here.

The minimize feature would definitely lose the complete path from start to finish as that isn't the intention. For me, I never (ever) care about the whole path. I care about a summary of the set of adjustments I made (that is, what path would I have taken if I'd known where I was going and made no mistakes). That's a distinct way to view history.

On the other hand for storing complete history in XMP, even if you store the deltas, you can reconstitute any step along the way (source control systems use this trick for efficient storage). It is undiluted, just very smartly compressed.

As an example of the reason for using deltas, I once was sent a corrupted catalog that was 7 GB. The size was dominated by the data for a small handful of photos. The history for the largest of them was was 450 MB (it had been heavily retouched with localized corrections). As an experiment, I blasted that data (history step by history step) into a Mercurial (source control) repository and committed each step. The repo at completion compressed to just under 700KB (and was only a couple of MB total, only about half again or maybe double the size of the XMP file containing only the final settings which were about 1.6MB).

All that is to say, while there are reasons not to store history in XMP, size is not one of them unless the implementation is naive.

[FeedbackCommunityMember]

Dan, I like your idea of an "essential history". It could be just a view on an underlying full history but maybe the user could also be given the option to "compress" the history to retain only the essential steps.

History size in XMP files or DNG metadata could be best controlled by using data compression (c.f. .zip files). I think in compressed form it would be feasible to store complete histories. Those who prioritise storage space could opt to store essential histories only.

Current XMP contents must be close to "essential histories", otherwise LR wouldn't be able to produce the final rendering from the original.

Lee, you wrote: "Remember, due to the Camera Raw defaults and the application of presets, the starting point isn't always fixed." I don't understand what you mean. When LR only got DNG or RAW+XMP files to work with (i.e., not catalogue data) it must be able to recreate the final rendering as if the catalogue data were available. Hence an entire essential set of instructions (which is close to but not the same as a essential history) must be contained in the DNG or XMP. I don't understand your comment about already having a simplified history.

[FeedbackCommunityMember]

"I care about a summary of the set of adjustments I made": Yes, I do too. I'd call that "essential history". It would be nice if one could compress/minimise one's histories into essential histories.

Regarding your Mercurial experiment: Certainly version control systems can store the latest version of a document efficiently as a delta against a base. However, given that you need the base as well, I reckon that most of the compression you saw (the couple of MB total vs the original 450MB) came from a compressed (à la .zip files) storage.

[FeedbackCommunityMember]

As you know this started as a bug report. If I could edit my original post, I'd update the text to read as a feature request. Maybe a staff member could replace the text of the first post with the following?

=== Feature request ===
LR catalogues store full edit histories. However, when changes are saved to image files (DNG, RAW+XMP, JPG) only a set of instructions to recreate the final rendering is stored, all of the edit history is lost.

When importing such image files one only gets a "Imported..." entry, instead of the full list of edits.

I suggest that users have the following options regarding data stored in image files:
a) no history stored (current situation)
b) essential history stored
c) full history stored

An "essential history" only contains those steps that are necessary to create the final rendering. For instance, all "back and forth" settings to a single parameter are replaced by a single parameter change. It would actually be nice if users were given the option to reduce the in-catalogue histories to "essential histories" as well.

The motivation for having histories outside catalogues is twofold:
1. It would simplify backup strategies in that one would only need to back up image folders to retain final renderings with their edit histories.

2. It would create a safety net against catalogue corruption. If a catalogue develops a problem one may not immediately notice and thus create multiple backups with the same problem. Instead of being required to rebuild a clean catalogue from (potentially very old) clean backups, one should have the option to just create a new catalogue from the image files.

The size of histories stored outside of catalogues could be addressed by efficient storage schemes (e.g,. binary compression).

[FeedbackCommunityMember]

Just for giggles, I compressed a LR catalogue with WinZip. A 56.3 MB catalogue was reduced to 5.86 MB. That's just a fraction over 10% of the original size.

I though it was worth a new feature request to suggest that catalogues are stored in compressed form.

[dtull]

Surprisingly, no. Zip compression of the 450 MB develop history only reduced it to a third of its size or so (IIRC), which is beaten by a factor of nearly 100 by deltas. Local corrections develop history compresses really badly with the zip algorithm. A RAR based compressor did much better (making the whole 7 GB catalog only 130 MB), but was MUCH slower to perform.

The 10:1 compression ratio comes (I think) from the metadata table's XMP content, which has its own efficiency issues.

[dtull]

See my remark lower in the thread. Zip compression works remarkably poorly on extremely large develop histories. They do better on large numbers of nearly identical develop settings, though even that is better accomplished by not storing identical settings many times. Most catalogs contain large numbers of photos with the same settings applied (either defaults or preset applied settings).

[FeedbackCommunityMember]

Thanks a lot for your comments, Dan. It is very good to see that you have looked into compression options already at some point.

It seems that there is little contention about compressing (older) backups. I'll do that manually for now, but it would be nice if some automation would be available in the future. The latter could include moving backups from a (quick working) drive to another (archival) drive once backups have reached a certain age.

[Victoria Bampton LR Queen]

If the purpose is just for backup/transfer/security, perhaps I could throw another alternative into the net - how would it work for people if there was a LR-specific sidecar option, which could contain absolutely everything including History, flags, etc. Something like XMP sidecars, but without interfering with the XMP spec and without having to worry about it breaking other programs that use XMP.

[john beardsworth]

Not another format, another type of sidecar!

TBH storing extra fields would not "interfere with the XMP spec" or trouble other programs. It's the X in XMP - extensible by you, me or Adobe. Adding extra fields is what it's all about, and other programs simply pass over fields which they aren't set up to read. OTOH a new format or sidecar would make it more complicated to update other programs as they'd have to have logic for a new file type and data format, rather than just being told to read a new XMP field.

[areohbee]

Would it really be necessary to invent a new file format, just to add a few blocks to the xmp?

[Victoria Bampton LR Queen]

But John, you just said you didn't want that extra data in XMP....

Putting it in the normal XMP files would certainly be my first choice.

[john beardsworth]

I've said I don't see much value in Dan's summarised history, but I did say "I do think there should be an option... to save history to XMP, but it [the option] should include other Lightroom work that isn't currently saved out. So I'd like it to be my choice whether to save stacking info, assignment to collections, virtual copies etc.". I'd probably switch off the history option though, and think off should be the default.

[dtull]

If we could ensure the storage design was efficient enough to not be an undue burden (bloated size) and XMP was extended to be more complete with respect to the catalog, I would say the default behavior should be to preserve all data and let people optionally choose to exclude the data they don't mind losing rather than making presumptions about what is important and what is not.

[Lee_Jay]

I agree 100%, Dan.

[john beardsworth]

It doesn't really matter whether the default is on or off - but what is important is that if it's added, it should only be an option.

Also missing is metadata for custom fields defined through the SDK. There's a fine XMP reading and writing engine that's just waiting to be used!

[FeedbackCommunityMember]

Dan, I also agree a 100% with your comment.

[FeedbackCommunityMember]

I think I waited a bit too long to create a new thread. There are so many contributions in here that it would be suboptimal to start afresh. However, maybe my first post could be edited to feature the rewritten feature request I posted here as a comment.

[Ian Lyons]

Dan,

The default should be Off, just as it is with Photoshop, which has had the ability to save history to XMP and/or a text file for years. Actually, take a look at how it done in Photoshop.

Also, be warned, set default to On and there will be h.ll to play!

[areohbee]

The default only matters if you don't know you have options, or where to go to change. Consider a warning dialog - first time save: warn about the fact that not everything is in, or it includes the kitchen sink..., then the user knows what to expect, can decide to do it differently, knows where to find the setting to change mind in future...

[dtull]

I'd have to look at the specific trade-offs involved in the Photoshop design. Lightroom and PS are different products that may justify different choices.