You may not know your Classic CC catalog is corrupt!!

[johnshore]

If you regularly tick "test integrity before backing up" when backing up your catalog upon LR exit, you may be in for a nasty surprise. 

Short version: If the integrity test fails, LR backs up the catalog anyway, but it doesn't inform you that your backup is corrupted.   At least, that's what's happening to me.

For the last few days, I've been struggling to recover from Lightroom declaring that my catalog is corrupt.  I did the usual - replaced the catalog with a recent backup in order to start from there.   That's always worked for me, but not this time!  That backup was also corrupted, as were all of my recent backups.  All LR attempts at repair fail. 

There are two ways to test catalog integrity:   (A) If you have LR set to prompt you for a catalog when starting (Preferences->General), you can tick "Test integrity of this catalog" at the lower right of the catalog-selection pop-up.  (B) The other way is to tick "test integrity before backing up" in the LR exit dialog that results when LR is set to back up every time LR exits (Catalog Settings->General). 

Here's the rub:  If test integrity fails on startup (method A), a pop-up tells you so, LR exits, and it remembers that the catalog is corrupt.  But if the integrity test fails on backup (method B), you never know it - no pop-up, no email, no nothing.   What you do have is a backed-up catalog that won't pass the integrity test when restored!  This problem might might be a local hardware/software issue, but I do know that others have encountered it. (See Lightroom Classic CC Backup Catalog Corrupted - I'm in that thread, but when I realized the underlying problem, I thought it best to start a new thread). 

The obvious question is how could you be backing up a corrupt catalog if LR was working properly on it before exit?   Unfortunately, just because LR is running on a catalog doesn't mean that the catalog is clean.  Apparently, there are forms of corruption that are benign until certain operations trip up LR.   I know this from experience.  With some of my backups, LR will start OK with them (when restored), but sooner-or-later it crashes with words to the effect "LR can't read something in the catalog and has to exit."   It doesn't seem to be predictable when or how the crash happens - just like some software bugs.  

Hmm, maybe this is a Classic CC bug - wouldn't that be nice!. 

So what likely happened to me is that, at some point in the past, my catalog became corrupted in a way that didn't result in a crash until the just the other day.  In the meantime, I've been backing up that corrupted catalog, feeling completely safe since I always tick "test integrity before backing up".   I haven't yet had time to see how far back I have to go to find a clean catalog, but I already know it's months.  

I hope this isn't a universal problem (well, in a way I do, since then Adobe would jump on it!), but I would appreciate any advice and also knowing if there are others dealing with this. 

As an aside: 

It's too bad you can't completely regenerate a catalog by exporting everything and importing to a new, clean catalog.  That works, of course, for  image folders, but I don't know of any reasonably easy way to export/import collections and published collections.  You can export a collection as a folder and import it to the new catalog, but this is very laborious if you use collections extensively.   Regardless, I don't think LR facilitates moving published collections from one catalog to another (you can treat them as regular collections, but no publishing info is carried over).  I don't understand why Adobe can't provide a tool for complete export/import. 

Thanks.

js

[dj_paige]

But if the integrity test fails on backup (method B), you never know it - no pop-up, no email, no nothing.   What you do have is a backed-up catalog that won't pass the integrity test when restored!

How do you know this is true? What is your evidence?

Just because you never received a notification at that time of a corrupt catalog, that doesn't rule out hard disk malfunction as the possible (and very likely) cause of the corruption. Your catalog was find when it was backed up, and then something went bad on the hard disk, and now you have a corrupted catalog file.

[johnshore]

Hi dj_paige - The answer is because this happens with numerous different backups, restored to different disks, and on different computers.  The catalog was not fine when I backed it up - there was a corruption lurking in it - one that is caught by the integrity test on startup. js

[Tensas]

This is a very serious problem

Every one of my zipped backup catalogs is corrupt. They have been variously backed up to a raid. Internal hard drive, and various external hard drives When unzipped, none of these will work. This is the same on three different computers running Lightroom Classic CC. I changed computers to be certain it is not a hardware problem.

The only back up which can be opened is a non zipped copy of the catalog I make after Lightroom closes.

The catalog has always opened as long as it is not recovered from  a zipped back up.

I have 157,000 photos in my catalog. The catalog is over 1 gig

The problem is the back up function in Lightroom always produces a corrupt backup zip file. After unzipping, it will not open.

D

[Just Shoot Me]

It is something on your system that is causing the corruption.

If it was LR it would be happening to every install of LR.

[johnshore]

https://forums.adobe.com/people/Just+Shoot+Me  wrote

It is something on your system that is causing the corruption.

If it was LR it would be happening to every install of LR.

I agree that's most likely, although it could also not happen on every LR install if it's the result of an obscure LR bug that only bites under unusual circumstances.  Regardless, my catalog was corrupted quite some time ago for whatever reason, but it continued to operate until it could no longer, meanwhile backing up what turns out to be a corrupted catalog.   Alternatively, it could be that the backup process corrupts every backup, which I highly doubt.

The alarming thing that I uncovered (locally, at least) is that the combo of

  (a) LR can run OK under some circumstances with a catalog that can't pass the integrity test,

and

  (b) when the integrity test is run as part of backup on exit, a failed test isn't reported to the user, and the backup goes ahead,

means that it's possible for a long trail of backups to all be corrupt.

thanks

js

[johnshore]

Regardless, of whether my problem is widespread, I think  it would be prudent occasionally to run the integrity check on startup - that way, you know for sure that your current catalog isn't corrupt. 

js 

[johnshore]

[posted elsewhere as well, since it's now clear that I'm not the only one with this very serious problem, e.g. see https://forums.adobe.com/message/10190475#10190475] 

More evidence that LR's catalog integrity test is not trustworthy.

While continuing to hope that my main lrcat can be fixed, I’ve been cobbling together a partial lrcat so I can continue working. Along the way, one of the pieces is an lrcat that I created by

1) managing to export a small part of my main catalog resulting in a file corrupt.lrcat (don’t know if the extracted lrcat is actually corrupt)

2) reconstructing that catalog with sqlite3 via: echo .dump | sqlite3 corrupt.lrcat > corrupt-lrcat.txt; sqlite3 -init corrupt-lrcat.txt repaired.lrcat)

This reconstructed catalog has the following properties:

1) it passes “test integrity” when the integrity test is run on LR startup

2) it backs up on LR exit with both test integrity and optimize enabled – no error messages from the test

3) Nevertheless, when I create a brand new catalog and then try to import repaired.lrcat, the import fails with “LR could not import this catalog because of an unknown error”.

Perhaps that’s a clue to what’s wrong with my full catalog. I’ll go ahead and upload repaired.lrcat in case someone is looking into my problem and thinks it's useful to look at this catalog fragment.

If nothing else, this confirms that the LR "test integrity" is not trustworthy!

Obviously, that's dangerous. In my case, it's why I didn't notice until now that my main catalog got corrupted when I switched to Classic CC (i.e., when the catalog format was updated). I know this because my last backup pre-Classic is OK, but my first backup from Classic is corrupt (likewise all subsequent backups).

[annw212125]

I am also finding that my LR Backups generate corrupted files, and the only way that I can backup my catalogue reliably is to do it manually.  I doubt this is a system problem on my computer for two reasons:

1.  This happens on all of my drives, all of the time, and I run a rotating schedule of three outside backup disks that I keep up to date.  The ZIP files generate reliable catalogue copies only once in a while -- perhaps once out of every 25 0r 35 attempts. 

2.  Looking over this forum I see that many of us are having similar problems.  If only one or two of us had issues, it would be different -- but many of us are experiencing the same difficulties, and that tells me there is something else going on.

Sure, the work around is to save your catalogue files manually, and that works all the time for me.  If my disks were corrupted that info would be corrupted as well, so I suspect the problem lies somewhere in the software that generates, or extracts, the ZIP file.

I am going to call Tech Support sometime next week, and am curious what they will say.  This will be my third call to them, and so far they have had no answer.

It is a serious issue, and I hope we can get to the bottom of it.

[GuyJr]

Many thanks to all who have posted here.  I didn't know I had a problem until I deleted all the collections in a cold fevered daze.  The assumption I could simply pull up a recent backup was flawed.  A January backup did work.  Interestingly . . . that backup was on the same drive as the catalog!  I suspect all the backups on my Network Attached Storage drives are toast.  That may be a clue to Adobe if they watch these threads.

Bottom line . . . do manual catalog backups until Adobe fixes their problem.

[HenryCHWong]

To avoid this situation, I think it is safe to back up LR catalog manually.

[Just Shoot Me]

A corrupted catalog is caused by some type of fault in the system. LR is not corrupting its own catalog file.

Yes there have been a few threads on this topic. But compared to the number of LR installs that number of corrupted catalogs is probably something like .00000000000001% of all LR installs.

[johnshore]

Just Shoot me (sorry, don't see the quote button) - 

I don't doubt that a system fault caused the corruption.   But apparently, LR can run fine on a corrupt catalog (I'm sure of this - many experiments) until something trips it up - meanwhile, I was backing up a catalog that had become corrupted. 

js

[HenryCHWong]

Have you used 3rd party software, like Sqlite, to check the integrity of LR database?

[photoliker]

My guess is that many people have corrupt Lightroom catalog databases and may not know it. I have a catalog that shoots me some obscure SQL error when accessing certain folders at random times. Won't see the error very often but when you're in Lightroom all day every day, I will see it every few months when accessing the folders that are corrupted in the sql database. It may or may not be hindering the performance of rest of the catalog, I am not sure.