P: Prevent loss of Edit Histories when Reimporting Photos

Report · Apr 06, 2011

When importing DNGs with stored edits (included XMP data) then the history of the photo just shows "Imported..." instead of the list of edits.

I have a corrupt catalogue. (I did nothing to cause the correction :()
The catalogue contains photos which are not associated to folders in the library module. When I choose "Got to folder in Library module" from the context menu for such photos, nothing happens. I imported them just like any other photos, but somehow the corresponding library folder wasn't created or lost.

I tried synchroning the parent folder but the missing subfolders are not created again.

That's why I decided the only way forward is to create a new catalogue. However, the new catalogue doesn't have any of the edit history. The rendering is OK and I can reset it to see the original version of the photos but I cannot see the edit history anymore.

Why is the edit history not recreated? The essence of it must be available because otherwise the correct final rendering could not be created.

I believe edit histories should be available for JPGs, RAW and DNG files. When I decided to use DNG files vs RAW files with sidecar (XMP) files, I didn't know that I'd lose the history with a fresh import of a DNG file. I suppose that if I had XMP files, I could copy these and still had my edit histories.

Report · Apr 06, 2011

Edit history isn't in the XMP because plenty of people - me included - simply wouldn't want it there.

If catalogue corruption is the real problem, it's that problem that needs to be resolved. eg by the user using a backup (which contains edit history), or by Lightroom's backup being smaller (ie a transaction log allowing rollback/forward)

John

Report · Apr 07, 2011

I see your point and concede that if the catalogue corruption would not have occurred, I would not have reported the lack of edit histories. However, now with a corrupt catalogue, I'm left between a rock and a hard place. Either I use the corrupt catalogue and cannot access some of the images in it properly, or lose all edit histories.

Note that some people may prefer to have all information that is needed to recreate a LR catalogue in DNG or RAW+XMP files. That would alleviate the problem of having to backup catalogues separately.

Also note that I didn't notice the corruption a long time. I went back to multiple catalogue backups and they didn't help. I guess LR never added the library folders properly (just added the photos to the catalogue as orphans) and hence there is no "good" version. The only option would be to go back to an intact but incomplete catalogue and try to build it up again. I'd rather prefer a synchronise or repair operation to succeed.

Report · Apr 07, 2011

TK said: "Also note that I didn't notice the corruption a long time. I went back to multiple catalogue backups and they didn't help."

Whoa. Yeah, people act as if backups are a panacea, but that assumes your backups aren't wonky.

I've known that edit history is only in the catalog for a long time, so it wouldn't have been a surprise, but it was a surprise the first time... - I feel for ya.

So, I take it the catalog integrity check that I assume you were smart enough to perform before the backups (if not, I've heard confession is good for the soul...) was passing, despite the problem? I hope you've sent the funky catalog to Adobe for inspection(?)

Anyway, if the catalog is mostly OK, it may be possible to reconstruct the missing pieces using an SQL client. Or at least someone at Adobe maybe could, I'm not sure I know enough, or you...

Report · Apr 07, 2011

One idea is to create a new catalogue, and then use File > Import as Catalog to import the corrupt catalogue. As someone who has almost deliberately corrupted catalogues (eg by running SQL) I've found that importing into a clean catalogue can purge awkward problems. A nice bloke at Adobe sometimes helps fix serious cases of catalogue corruption too. Have you posted elsewhere about the corruption?

Overall, I am not blindly against saving history steps to XMP - though XMP does have the adjustment settings so you're only losing the "route". I do think there should be an option (haven't we had this conversation before?) to save history to XMP, but it should include other Lightroom work that isn't currently saved out. So I'd like it to be my choice whether to save stacking info, assignment to collections, virtual copies etc.

On the other hand, I wouldn't want Adobe to see this as anything more than a very secondary level of backup.

John

Report · Apr 07, 2011

TK,

John's suggestion may just fix you right up. If not, if you send me both catalogs + a list of missing info, I'll transfer what I can, which may be nothing...

R

Report · Apr 07, 2011

Unfortunately, the corrupt catalogue passed and passes the integrity test. If it is just an SQLite integrity check then it wouldn't mind about orphan photos (photos who lost their direct folder association), would it?

I'd be happy to send the catalogue to Adobe but won't do it unsolicited. I seem to remember some bug like this being discussed in the U2U forums.
IIRC, such corruption can occur when you move folders within LR. Seems that moving them with the OS and then finding them again in LR is safer. But the latter issue could be a separate one to what I was experiencing because some of the missing folders were never moved; just imported "into a subfolder" as I always do.

Report · Apr 07, 2011

"One idea is to create a new catalogue, and then use File > Import as Catalog to import the corrupt catalogue.": Thanks a lot for this suggestion. I thought about doing that but wasn't sure what end result to expect from it. I'll try it. Thanks again.

The only other place I reported about the corruption is when I suggest that a folder synchronise should bring back / restore missing subfolders.

I agree that having options regarding what is saved in XMP files / DNG metadata would be desirable.

Report · Apr 07, 2011

Rob, thanks a lot for the offer! I'll see what I can do myself first. I hope I won't have to bother you with this.

Report · Apr 07, 2011

I dont know what all the integrity check does, but I now know one thing it does not do. Do you know any SQL, TK?

Report · Apr 07, 2011

OK. The Lr database is simple and straight-forward in some respects, and a bit complicated and intimidating in other respects. Some things may be readily transferable, other things - not so much...

Report · Apr 07, 2011

Good news:
I discovered that the corrupted catalogue had only one corrupt entry. It was an image whose "root folder" was "nil".

I exported all photos in the corrupted catalogue -- except for the one problematic one -- to a new catalogue and, surprise, surprise -- the new one shows all the missing subfolders again!

So LR stumbled across one corrupt database row, causing it to drop a number of library folders from the display. I also noticed that LR became very slow with the corrupted catalogue in some operations (displaying "All Photographs").

Maybe (wild speculation) corrupted catalogues can sometimes be the source of some "performance" problems?

Report · Apr 07, 2011

That is good news!

One time, simply optimizing my catalog improved performance by A LOT, meaning it went from buggy slow to normal.

Report · Apr 07, 2011

Thanks for the update. Glad to hear things are working again!

Report · Apr 07, 2011

Question (to the staff): Should I post this as a feature request? It seems it is not really a bug because the loss of the edit histories is "as designed". But it seems that there might be cases when saving editing histories in or with photo files would be desirable.

It seems that I cannot convert this bug report into a feature request anymore, so should I create a new one?

Report · Apr 07, 2011

It should be showing as an "idea" now.

Report · Apr 07, 2011

Yes, it does. Thanks Chad! It doesn't really read like a feature request, though. If it is clear what I'm after then that's OK with me. I'd be happy to rephrase the original bug report as a feature request, if this would be useful.

Report · Apr 07, 2011

Personally, I'd like anything that can be stored in XMP to be stored there. I don't trust the catalogs or backups of the catalogs as I've been burned by them going bad on me several times (Dan saved me once). I'm not willing to lose the work I do between catalog backups which means I really want a reliable backup after each image edit, which is totally impractical using the catalog backup technique. As of now, I've given up entirely on backing up the catalogs and only backup the XMP data, which is image-by-image, basically instant and has saved me a couple of times when a catalog went bad inexplicably (I would have lost perhaps ten hours of work each time going back to a catalog backup). Basically, the catalogs seems fragile to me and the XMP data is not so I rely on the far more robust XMP backup strategy instead of the fragile catalog backup strategy. The bad news is that I can't use many of LR's features including collections, stacks, VCs and flags because they aren't stored in the XMP data. Personally, losing the Develop history is the least important to me as I rarely use it for anything anyway.

Report · Apr 07, 2011

Wish I could work with one hand tied behind my back too....

In any case, LR catalogues are impressively robust and we have very few reports of corruption. That's why they are worth backing up.

Report · Apr 07, 2011

Lee, I perhaps wouldn't call the catalogues "fragile" but I agree with your overall sentiment. I think you should press the "like" button for the feature request. 🙂
Still not sure whether the former bug reports works as a feature request. Shall I create a new one?

Report · Apr 07, 2011

John, I had quite a number of backups of the catalogue but they didn't help one bit in this instance because the error was a silent one and I didn't discover it before several generations of corrupt backups. Going back multiple generations to one that is still clean means that a lot of work has to be redone. This is not optimal.

Report · Apr 07, 2011

Feel free to rephrase this as a feature request in a new thread if you'd like to make it a bit more concise.

Report · Apr 07, 2011

"File > Import as Catalog" isn't panacea either. I've done this before to try to fix a catalog and only realized much later that I had lost many of my collections and web galleries. That one still stings...

Report · Apr 08, 2011

Two thoughts on this: I've generally been of the opinion that significant metadata should (eventually) find its way into XMP since XMP can be used as a kind of distributed catalog backup. The main reason I could see not to have history in the XMP is that it'd be too big, but if it were stored smartly as a series of deltas against the final settings, it's should be fairly small in most cases.

Which reminds me of another mini-feature I've wanted for the history panel: a "minimize" option that elides multiple (even non-contiguous) adjustments of the same slider into a single step. It would turn a messy history list into a simplified summary of all the actions taken to get from original to final.

The reason that's possibly relevant here is that minimize could probably be implemented in such a way that it could compare the default settings with the current settings which would mean an image imported without history would get a concise summary of the changes. Not as complete as the original, but it provides at least some of the utility.

Report · Apr 08, 2011

Yes, Dan, I'll take both of those with a side of fries, please. 😉

Report · Apr 08, 2011

If one's to have an *option* to write the edit history in the XMP, Dan, I'd rather have it undiluted. History's about how one got to the final result, and restoring only the differences would mean people wouldn't be able to go back to a certain stage of work or use it for Before/After comparison. How could one get that value from the differences from default?

And you don't even need to store the differences in the XMP - LR could already compute them from the adjustment values already in the XMP. So again, where's the value?