metadata status still broken

[DanielUSA]

Going crazy trying to deal with metadata inconsistency problems. I know that this has been an issue in the past, but it seems that it's still broken. In short, Lr is reporting metadata statuses for large numbers of files in my catalog that seem to be wrong. And when I try to fix them, eg, by selecting all files classified as having been changed in the catalog, and I then save their metadata to disk, the save seems to complete successfully (at least no errors and the file mod times as reported by macOS are updated), but they are still reported as unresolved. The only workaround I've found so far is to (a) save all the files to disk; (b) read their metadata back in (which is slightly worrying because on doing the read, Lr reports, incorrectly I believe, that the metadata will be overwritten). I've noticed other bugs too: the counts of files in the metadata filter panel don't correspond to the actual number of files listed; when I succeed in resolving files, the number still shows thousands of files unresolved but the panel with the tiles is empty.

Some more information:

-- I'm running the latest version of Lr on Big Sur. Adobe friends: if I could copy-paste the build number from the about panel I'd include it 🙂

-- This seems to happen on all kinds of files, DNGs, JPEGs and raws, and for even the simplest metadata updates (most notably keywording).

-- I'm not sure I've seen this problem for files stored on my Mac's SSD; it seems to only occur for files on my GRaid external drive.

-- I can reproduce the problem by selecting a single file that is reported as out of sync, then saving to file, noting that the metadata date in Lr matches the new date on disk, and then seeing that it's still reported as out of sync.

And yes, in case you're wondering, I do have more than one Lr catalog. I know switching to a single catalog might eliminate this problem, but I'm really loath to do that. I have one active catalog of images I'm currently using in projects, which contains about 50k images, and is kept on SSD, and another one of all images, in which I don't do any editing work except keywording, and that has about 200k images in it, and is kept on an external hard drive. 

[DanielUSA]

A followup to my own post.

I've noticed that when you read metadata from file, the develop action history gets a new action "From metadata."

It's not possible that this is actually reckoned as a metadata modification, is it? If so, this would be disastrous. It would mean that immediately after reading metadata from file, a file in the catalog is actually out of sync with the disk, and would appear as changed in the catalog!

[johnrellis]

"I'm running the latest version of Lr on Big Sur. Adobe friends: if I could copy-paste the build number from the about panel I'd include it"

Please do the menu command Help > System Info and copy/paste the first ten lines here. That will tell us precisely which version you're on (there's often confusion about "the latest").  

[DanielUSA]

I'm on the verge of giving up, so I'm trying to create a new catalog that merges my two existing ones. And it turns out that although the system info panel will open during import, copy doesn't work :-(. But I can screenshot it -- attached.

[GoldingD]

Just on external media?

Sounds like a MACOS cybersecurity issue. A right to access the external media may have not been setup (usually because the owner overlooked a pop up notice from the MACOS)

trying to remember a link to support the above. perhaps nother member will cortect or add to. This goes back to MACOS Catalina.

Ah, old link, refers to Catalina, but still on point.

https://nektony.com/duplicate-finder-free/folders-permission

[DanielUSA]

It's thoroughly inconsistent though. I've been writing from this catalog to the disk forever... And certainly no macOS warning...

[GoldingD]

The MACOS notification would gave come up once. It would have asked yes or no. It would not reappear.

Only a what if observation.

[DanielUSA]

Thanks -- did check just now, and Lr has access to all folders and to removable volumes. 

[johnrellis]

From the details you've reported, it sounds like there are multiple issues going on, so let's try to carefully tease them apart.

In general, LR's metadata status has always been unreliable. Some versions have had many fewer problem reports than others, but in the last year, it seems there are more reports.

If you're sure that you haven't edited a photo or its metadata in another app, then you can safely trust that the information in the catalog is the "truth", and you can do Metadata > Save Metadata To File. In past versions, that would clear a spurious metadata status indicator. 

In past versions, you could safely follow Save Metadata To File with Read Metadata From File, which was sometimes necessary to clear the indicator. However, LR 10 introduced a reproducible bug where Save Metadata To File wouldn't clear the metadata status, and a subsequent Read Metadata From File wouldn't either.  In addition, if the photo had a non-zero crop angle, Read From Metadata would lose the crop angle in the catalog:

https://feedback.photoshop.com/conversations/lightroom-classic/lightroom-classic-save-metadata-to-fi... 

There may be other conditions other than using Black & White or non-zero crop angles that trigger the spurious metadata status.  This bug is not yet fixed, so doing Read Metadata From File is definitely unsafe!

If doing Metatadata > Save Metadata To File doesn't clear the spurious status indicator, then I recommend to just ignore it.  It bugs the hell out of me to see it, though.

[DanielUSA]

Thanks so much for all this, John. Incredibly depressing though, as it doesn't seem to leave me with any good options, and I'm now worried that by saving and reading metadata I might have corrupted a large number of images. 

I'm rather astonished that a bug of this severity hasn't been fixed.

Any thoughts about the "From metadata" action?

[johnrellis]

"I have one active catalog of images I'm currently using in projects, which contains about 50k images, and is kept on SSD, and another one of all images, in which I don't do any editing work except keywording"

If the catalogs don't share photos, then there's nothing untoward about having multiple catalogs.

However, if you have photos that are in both catalogs, then that can definitely cause metadata status warnings. For example, if you edited a photo in one catalog and did Save Metadata To File, then opened the other catalog, you'd see the status Changed On Disk.  

If LR's metadata status worked properly, you could keep your head straight doing this, paying careful attention to the metadata status indicators and doing Read Metadata From File where necessary to transfer the information from one catalog to the other. 

But given the flakiness of LR's metadata status, I definitely don't recommend sharing photos between catalogs.

[johnrellis]

"when you read metadata from file, the develop action history gets a new action "From metadata."

It's not possible that this is actually reckoned as a metadata modification, is it? If so, this would be disastrous. It would mean that immediately after reading metadata from file, a file in the catalog is actually out of sync with the disk, and would appear as changed in the catalog!"

That History step just indicates the action you took.  Normally, Read Metadata From File makes the catalog metadata the same as what's on disk, though as noted above, there's a known bug with non-zero crop angles.

[DanielUSA]

Wow, this is a real mess. 

I have no saved/read 5,000 files. I've no idea how many have been corrupted; I'm sure many of them have crop angles and many are B&W.

I tried to import one catalog into the other, and hit another serious bug: the custom order of files in collections (that exist only in the importing catalog but include files that were in the other catalog) has been corrupted. 

[DanielUSA]

OMG, it's even worse than I thought. Save/read from metadata has dropped the entire metadata for thousands of files that had crop angles.

Why hasn't Adobe blocked this function until they fix it?

[DanielUSA]

John: a question for you. I'm now trying to get everything back to where I was. I've restored an old version of the catalog from backup (the one that got corrupted by the read metadata from file), and I'm now writing out all those files again from the restored catalog. 

The main thing this leaves though is that I spent a day keywording images in the other catalog, basically just adding two keywords to identify two people. Is there any way I can safely get that information to the other catalog (which will allow me to ditch this second one, since previously I'd been doing exactly what you mentioned, carefully watching metadata status on shared files)? I'm thinking that maybe I can create two collections for each of the keywords, and then delete all the other collections, export those collections as a catalog, and import without importing files or metadata -- is that likely to work or might I hit another bug?

[DanielUSA]

Sorry for the deluge of messages...

And actually a more basic question: I now realize I perhaps can't even import the photos from the old catalog at all, because even though I wrote all their metadata out to file, they won't import correctly if they have B&W or crop settings, right? 

So these bugs (the combination of read metadata from file being broken and catalog import messing up collections) mean you can't actually transfer files between catalogs at all and the develop settings are trapped in the old catalog?

[johnrellis]

Use the command File > Import From Another Catalog to merge the two catalogs. It uses the metadata stored in the catalogs and ignores what's on disk. I don't recall ever seeing any bug reports about Import From Another Catalog (but make two extra backups of each catalog just in case!).

[DanielUSA]

Thanks: I'll try that again. That was the command I was using that messed up the collections. I'm currently removing all the photos from the catalog being imported that aren't already in the importing catalog, in the hope that the import won't screw up the custom order of files in collections (since I'm assuming that bug is only activated when the collection includes files that exist in the imported catalog).

[johnrellis]

Also, please add your constructive opinion to the bug report I linked to above, and be sure to click Like and Follow at the bottom of the first post. That will make it a little more likely Adobe will prioritize the fix, and you'll be notified when the bug's status changes. Product developers rarely participate in this forum and won't see your feedback.

[DanielUSA]

So I did that, but the import seems to be buggy too. Here's what I did: 

-- in catalog Main, I created a collection called "all photos" and inserted every photo in the catalog into it.

-- in catalog Secondary, I imported Main, and then selected the photos in "all photos" and executed the action "remove from Lightroom"; as expected, this leaves that "all photos" collection empty and reduces the number of images in the catalog (under "All Photographs") by the number that were previously in that catalog. This should leave Secondary containing exactly those photos that are not in Main.

-- in catalog Main, I imported catalog Secondary, hoping to bring in the metadata settings of the images that were previously in Secondary but not in Main.

-- but the catalog import dialog asked me what to do with replacing metadata for over 20k images that it said were "existing". If, as I assume, that refers to conflicting files, there's some other bug here (or it's possible that I did something wrong, but was very careful...)

[johnrellis]

One potential gotcha: When you inserted all the photos into the collection, did you first do Photo > Stacking > Expand All Stacks?  If not, then only the tops of stacks would have been inserted into the collection.

If you're comfortable with Excel or Google Sheets, you might double-check the two catalogs don't have any photos in common.  Use the free trial of the Any Filter plugin to export lists of the files in both catalogs:

Then join the two lists, e.g. using Excel's XLOOKUP.

If this doesn't help, you could upload your two .lrcat files to Dropbox or similar and send me the sharing links in a private message.  I can double-check and see if the error occurs for me too.

[DanielUSA]

Thanks so much, John. I really appreciate your expertise and your willingness to help. I seem to have got things back into reasonable shape with all the files in a single catalog. Of course, the bugs are still there but I'm assuming (optimistically?) that save to file is actually working (since the file mod time is updated) even if the metadata panel still reports things as out of sync. 

I noticed another sync bug: some of the "files" that won't sync turn out to be virual copies. Interestingly, when I select "show in finder" they do map to DNGs, which I assume is a mistake. Looks as if whatever integrity checks Lr does (eg for optimize catalog) should also include a check that no image is classified as both a file and a virtual copy. Also (and I haven't seen mention of this), movie files have always been reported as being in conflict for me (and continue to be) even though I never update any metadata associated with them.

I did add to the bug report that you posted, and hopefully the Lr team will look into this. It does seem amazing to me that a bug of this severity is allowed to persist. It seems to be the kind of thing you'd do a feature freeze to resolve...

[DanielUSA]

Meant to say: I checked stacking, and that wasn't responsible for the discrepancy. Thanks for the suggestion though (and for teaching me about a very strange UX issue).