P: Prevent loss of Edit Histories when Reimporting Photos

See my remark lower in the thread. Zip compression works remarkably poorly on extremely large develop histories. They do better on large numbers of nearly identical develop settings, though even that is better accomplished by not storing identical settings many times. Most catalogs contain large numbers of photos with the same settings applied (either defaults or preset applied settings).

Surprisingly, no. Zip compression of the 450 MB develop history only reduced it to a third of its size or so (IIRC), which is beaten by a factor of nearly 100 by deltas. Local corrections develop history compresses really badly with the zip algorithm. A RAR based compressor did much better (making the whole 7 GB catalog only 130 MB), but was MUCH slower to perform.

The 10:1 compression ratio comes (I think) from the metadata table's XMP content, which has its own efficiency issues.

Just for giggles, I compressed a LR catalogue with WinZip. A 56.3 MB catalogue was reduced to 5.86 MB. That's just a fraction over 10% of the original size.

I though it was worth a new feature request to suggest that catalogues are stored in compressed form.

As you know this started as a bug report. If I could edit my original post, I'd update the text to read as a feature request. Maybe a staff member could replace the text of the first post with the following?

=== Feature request ===
LR catalogues store full edit histories. However, when changes are saved to image files (DNG, RAW+XMP, JPG) only a set of instructions to recreate the final rendering is stored, all of the edit history is lost.

When importing such image files one only gets a "Imported..." entry, instead of the full list of edits.

I suggest that users have the following options regarding data stored in image files:
a) no history stored (current situation)
b) essential history stored
c) full history stored

An "essential history" only contains those steps that are necessary to create the final rendering. For instance, all "back and forth" settings to a single parameter are replaced by a single parameter change. It would actually be nice if users were given the option to reduce the in-catalogue histories to "essential histories" as well.

The motivation for having histories outside catalogues is twofold:
1. It would simplify backup strategies in that one would only need to back up image folders to retain final renderings with their edit histories.

2. It would create a safety net against catalogue corruption. If a catalogue develops a problem one may not immediately notice and thus create multiple backups with the same problem. Instead of being required to rebuild a clean catalogue from (potentially very old) clean backups, one should have the option to just create a new catalogue from the image files.

The size of histories stored outside of catalogues could be addressed by efficient storage schemes (e.g,. binary compression).

"I care about a summary of the set of adjustments I made": Yes, I do too. I'd call that "essential history". It would be nice if one could compress/minimise one's histories into essential histories.

Regarding your Mercurial experiment: Certainly version control systems can store the latest version of a document efficiently as a delta against a base. However, given that you need the base as well, I reckon that most of the compression you saw (the couple of MB total vs the original 450MB) came from a compressed (à la .zip files) storage.

Dan, I like your idea of an "essential history". It could be just a view on an underlying full history but maybe the user could also be given the option to "compress" the history to retain only the essential steps.

History size in XMP files or DNG metadata could be best controlled by using data compression (c.f. .zip files). I think in compressed form it would be feasible to store complete histories. Those who prioritise storage space could opt to store essential histories only.

Current XMP contents must be close to "essential histories", otherwise LR wouldn't be able to produce the final rendering from the original.

Lee, you wrote: "Remember, due to the Camera Raw defaults and the application of presets, the starting point isn't always fixed." I don't understand what you mean. When LR only got DNG or RAW+XMP files to work with (i.e., not catalogue data) it must be able to recreate the final rendering as if the catalogue data were available. Hence an entire essential set of instructions (which is close to but not the same as a essential history) must be contained in the DNG or XMP. I don't understand your comment about already having a simplified history.

@John I think we're confusing two things here.

The minimize feature would definitely lose the complete path from start to finish as that isn't the intention. For me, I never (ever) care about the whole path. I care about a summary of the set of adjustments I made (that is, what path would I have taken if I'd known where I was going and made no mistakes). That's a distinct way to view history.

On the other hand for storing complete history in XMP, even if you store the deltas, you can reconstitute any step along the way (source control systems use this trick for efficient storage). It is undiluted, just very smartly compressed.

As an example of the reason for using deltas, I once was sent a corrupted catalog that was 7 GB. The size was dominated by the data for a small handful of photos. The history for the largest of them was was 450 MB (it had been heavily retouched with localized corrections). As an experiment, I blasted that data (history step by history step) into a Mercurial (source control) repository and committed each step. The repo at completion compressed to just under 700KB (and was only a couple of MB total, only about half again or maybe double the size of the XMP file containing only the final settings which were about 1.6MB).

All that is to say, while there are reasons not to store history in XMP, size is not one of them unless the implementation is naive.

Not as simple as Dan is suggesting though, and I wouldn't want Photoshop history to be replicated in Lightroom (no, don't anyone suggest a history brush!!!)

We have a simplified history already. It just doesn't look like it. The numbers are cumulative so they aren't quite the same as those in the PS history palette.

Application of presets? You mean upon import? I'd say they form part of the difference. But that's arguable. Some will want them in, some out, and as there's not a lot of value from having simplified history....