Keywords on photos but missing from the list

[DaveDuprePhoto]

I recently organized my keywords to make things easier to find. However, I now have a situation where dozens of keywords still exist on files, but they do not show up in the Keyword List.  

 

If you see the image below, I have multiple keywords on a photo, but none of them show up in the list.

In addition, I can delete one of the keywords from the text box ("Aviation" for example), and then try and type it back in. As I type "Avi", "Aviation" will display for me to choose. That tells me the keyword is in there somewhere, but it still is not showing up in the Keyword List. I also have multiple smart collections based on keywords, and they still work.

I deleted the helper.lrdata file multiple times, and I tried resetting all my preferences. I also tried exporting as a catalog. That didn't work either. Still missing most of my keywords.

How can I get my keywords to show up in the list?

[DaveDuprePhoto]

An update:

I tried doing Metadata -> Export Keyword List. It got about halfway through and stopped. I waited several minutes before canceling. The CSV contained what was in my keyword list panel, but my guess is there is something wrong with the internal keyword list, and it froze the export.

Any way to rebuild the keyword list from all the photos?

One more thing, I checked my before and after backups. Sure enough, the keywords are missing in the after, and they are there in the before.

I'm thinking there is a bug in purge unused keywords since that is the last thing I ran after I completed the re-org. Feels like it removed dozens (maybe hundreds) of keywords even though there are photos that have the keyword.

I've been using this catalog for many years (at least 10, probably more, but I don't know exactly), and I have 40-50k photos.

[JohanElzenga]

The keywords shown in your screenshot are 'parent' keywords. The solid triangle shows that they contain sub keywords. That is where the keywords shown on your images are hiding. Click on a triangle to expand it and show the keyword hierarchy.

[DaveDuprePhoto]

To be clear, I know they are parent keywords. However, you need to note that there are no check boxes or '-' next to any of those parent keywords as there should be to indicate that a keyword down in the hierarchy is selected.

Also, if I search for Aviation in the keyword list, it will not be found. Expanding all keywords in the hierarchy, and Aviation (along with an untold number of others) are not in the list. There are there on the photos, but they are not in the keyword list.

Finally, if I try and export the keywords, the export process will never complete.

As I said above, I've been using Lightroom for over 10 years. I use keywords all the time.

[johnrellis]

Before moving to possible steps for recovering the keywords, let's do some troubleshooting steps; let us know the results of each step.

1. Turn off Catalog Settings > Metadata > Automatically Write Changes Into XMP.  That will stop possibly corrupted metadata from getting (further) written back to the photo files.

2 Test the integrity of the catalog by exiting LR, holding down Opt while you restart LR, and then selecting Test Integrity Of This Catalog.  That detects some catalog inconsistencies created by LR bugs. 

3. Try deleting the Helper.lrdata folder:

a) Do Catalog Settings > General > Show to open Finder / File Explorer on the current catalog folder.
b) Exit LR.
c) In that folder, delete the folder "<catalog> Helper.lrdata".
d) Restart LR, and it will rebuild the folder.

The Helper.lrdata folder caches information about metadata, keywords, folders, collections, and other things to speed the performance of LR and it can sometimes get corrupted. In recent versions, LR has used it more heavily to improve the speed of displaying metadata, and there have been a fair number of bugs with it.

4. Use the Export Keywords command of the Any Tag plugin (which has a 30-day free trial) to export the keywords to a .csv.  Does that command complete, and does the .csv look like it contains all your keywords? 

[DaveDuprePhoto]

1. Turn off Catalog Settings > Metadata > Automatically Write Changes Into XMP.  That will stop possibly corrupted metadata from getting (further) written back to the photo files.

This his been off on my catalog for as long as I can remember. 

2 Test the integrity of the catalog by exiting LR, holding down Opt while you restart LR, and then selecting Test Integrity Of This Catalog.  That detects some catalog inconsistencies created by LR bugs. 

Done this multiple times, and it always comes up fine.

3. Try deleting the Helper.lrdata folder:

As I said above, I've done this many times.

4. Use the Export Keywords command of the Any Tag plugin (which has a 30-day free trial) to export the keywords to a .csv.  Does that command complete, and does the .csv look like it contains all your keywords? 

I will try this, but my question is "What then?" 

[johnrellis]

"will try this, but my question is "What then?""

It depends on what the result is, whether Any Tag produces any error messages, whether the list is complete.

[DaveDuprePhoto]

Any Tag completed, but the file only contained the keywords that are in my truncated Keyword List. It did not include all the ones that I know are missing. In my original attachment above, you can see Aviation and P-47 on a photo of an old WWII warbird. Neither Aviation or P-47 are in the keyword export. 

[johnrellis]

Hmm, I haven't seen that kind of catalog inconsistency before -- with others, Any Tag can produce a clueful error message. Some more steps:

1. Sometimes doing Import From Another Catalog recovers from inconsistencies that Export To Catalog doesn't (and vice versa).  Create a new empty catalog, do File > Import From Another Catalog, and select the main catalog. Does that restore the keywords?

2. Another diagnostic step to see what information might be recoverable with plugins or scripts: Install this free plugin, Show Metadata:

https://www.dropbox.com/scl/fi/hagpmldch9q60ns4u4btq/showmetadata.1.9.zip?rlkey=06u2fautb3azdpgutw7m...

Select the photo with the missing keywords and do the menu command File > Plug-in Extras > Show Metadata > Catalog. That will open a text file containing all plugin-accessible metadata of that photo. In the Raw section, search for the line starting with "keywords = {" and copy that and the following indented lines here, e.g. 

    keywords = {--table: 4
        [1] = "California < US < Places (102148408)", 
        [2] = "Joe Blow < People (102148412)", 
        [3] = "Joe's Wedding < Events (102148416)"}, 

That will indicate whether it would possible to recover your hierarchical keywords using a small plugin script.

[DaveDuprePhoto]

A big part of me wants to just suck it up and go back to the last backup without the problem. Then I can make more incremental changes and backup after each change. I feel better about that vs the "I might be able to restore..."

Software developer me (30+ years) sees two things that indicate something went wrong.

1. A couple of times during the process, I ended up with a keyword in the list that indicated 1 photo had that keyword. However, clicking on the right arrow to show the photo came back with nothing. I ended up deleting the keyword.

2. I'm not 100% sure, but I think everything went south when I purged unused keywords at the end. Maybe it didn't properly deal with a hierarchy in certain cases and removed a parent keyword from the list along with all its children. In 99% of the cases, no parent keyword in my list has any photos assigned. It's always the lowest child keyword. The strange part is that it only removed it from the list. It didn't remove it from the photos. 

As an FYI, my keyword list was a bit of a mess. There were misspellings, duplicates, hierarchy was all over the place, far too many one-offs, etc. I wanted to get things into a better hierarchy with the top being:

events

activities

info

people

places

things

Doing this update require some temporary parent keywords and a lot of keyword moving. I must have done a little bit too much, too fast and discovered a bug in there.

[johnrellis]

Your initial evidence indicated a clearly corrupted keyword list, and the further troubleshooting confirms it.  Maybe it was Purge Unused Keywords in particular, though I haven't seen any other reports about that over the years. There are very infrequent reports over the years of the keyword list getting corrupted, but not in the way you're seeing.

What was the result of doing Import From Another Catalog?

If you have backups and if you're willing to live with losing the changes you've made since the last backup of the uncorrupted keywords, that would certainly be a conservative way to go.   If you'd lose a lot of changes, my step 2 above would quickly indicate whether the parent keywords are still accessible to plugins on a per-photo basis. If they are, I could quickly write a script that would recover them.

[DaveDuprePhoto]

I ended up biting the bullet and going back to the last good backup and then redo all my keyword changes. This time, however, I backed up a lot along the way. I have it all done now, and all my keywords are there. Only thing strange that I noticed is that some parent keywords would say there were photos, but if I searched none we found. For example:

California 405

    Disneyland. 400

I found that I added both California and Disneyland to a photo (no idea what I might have been thinking that day). I would click on the right arrow next to Disneyland to see all the photos. Then I remove California for all those photos. Now I have:

California 5

    Disneyland 400

Now, I go back to the keyword search and change from hierarchy to flat, and search for California. I expected to find 5, but none were found. To fix this, I created another California tag in a different parent, and then move Disneyland to the new keyword. Finally, delete the old California keyword. Lightroom will warn me about the photos that have this keyword.

Obviously, there must have been an odd catalog corruption that's been around a long time. Somehow, a photo was deleted but the reference from the keyword was not removed.

Hopefully, I don't have any other keywords that show more photos than actually exist. Wonder if there is a way to verify this?

Is it worth finding a way to share my catalog with Adobe to see if they can close a bug?

Your initial evidence indicated a clearly corrupted keyword list, and the further troubleshooting confirms it.  Maybe it was Purge Unused Keywords in particular, though I haven't seen any other reports about that over the years. There are very infrequent reports over the years of the keyword list getting corrupted, but not in the way you're seeing.

What was the result of doing Import From Another Catalog?

If you have backups and if you're willing to live with losing the changes you've made since the last backup of the uncorrupted keywords, that would certainly be a conservative way to go.   If you'd lose a lot of changes, my step 2 above would quickly indicate whether the parent keywords are still accessible to plugins on a per-photo basis. If they are, I could quickly write a script that would recover them.

By @johnrellis

[johnrellis]

"Hopefully, I don't have any other keywords that show more photos than actually exist. Wonder if there is a way to verify this?"

This symptom smells like an inconsistent <catalog>Helper.lrdata folder, which caches information about metadata, keywords, folders, collections, and other things to speed the performance of LR. It can sometimes get corrupted. In recent versions, LR has used it more heavily to improve the speed of displaying metadata, and there have been a fair number of bugs with it.  

I recommend deleting it, and LR will automatically rebuild it:

a) Do Catalog Settings > General > Show to open Finder / File Explorer on the current catalog folder.
b) Exit LR.
c) In that folder, delete the folder "<catalog> Helper.lrdata".
d) Restart LR, and it will rebuild the folder.

[DaveDuprePhoto]

I can do it again, but I've deleted that file several times during this process. 🙂

[johnrellis]

"I can do it again, but I've deleted that file several times during this process."

Maybe restoring from backups got a version that didn't match the .lrcat file somehow. 

[DaveDuprePhoto]

Hard to say. I restored to a new directory, so it should have rebuilt the helper.lrdata file. 

[johnrellis]

"Hopefully, I don't have any other keywords that show more photos than actually exist. Wonder if there is a way to verify this?"

I modified a script in my test library to check for such inconsistencies -- it verifies that every keyword assigned to a photo appears in the catalog's keyword list, and that the photos the keyword list says are assigned to a keyword actually have the keyword assigned. It doesn't make any changes.

To run the script, save the attached file to your disk and change its extension from ".txt" to ".lua". Then follow the instructions at the top of the file.

The script accesses the catalog through the LR SDK for plugins and doesn't directly access the catalog database. So it may not detect all inconsistencies.

[DaveDuprePhoto]

All good!

Keywords missing from catalog:

Keywords missing from photos:

[johnrellis]

Good that increases the confidence that there aren't any inconsistencies remaining in the keywords (but doesn't totally eliminate the possibility).