P: Duplicate detection fails when Filename/Size/ Match but Date does not

[zefelder]

I just was importing ~480 photos and I noticed that 6 or so of these photos were grayed out because they were "suspected duplicates".

They are definitely not duplicates. I made these photos today. I checked that the catalogue does not have any old photos incorrectly stamped with today's date.

I would assume that suspected duplicates at least involve comparing hashes of photos. For 6 out of 480 false poistives it should be a very weak hash even taking in account that I have ~300K photos in my catalog. So really sounds like a bug for me. Also I don't remember seeing anything like this in ~10 years that I use Lightroom. 

[Rikk Flohr: Photography]

Curious - are these Drone Photos?

LrC views duplicates like this:

If: 

1. Photo is the same physical size on disk
2. Has the same Date/Time Stamp
3. Has the same file name

It is a duplicate.

Select a file from your import queue - see if your Lightroom catalog has another file of the same name by searching All Photos for that file name. Compare the Date/Time and Size on Disk. 

[zefelder]

No, these are not Drone photos. The photos are from Sony ZV1 (a variant of RX100).

I checked that I don't have any other photos with the same date - checked both file creation date and EXIF.

Given I have tons of photos in the Catalogue there could be collisions by name+size alone (there on average ~50 identically named photos, and sizes are not very different either). Unfortunately I did not write down the names of the photos that were marked suspected duplicates before I imported them, so unfortunately I cannot provide more specific information. But I do suspect a bug, as I don't see any other reasonable explanation.

[Rikk Flohr: Photography]

As a test...

If you restart your computer and create a new blank catalog and attempt to reimport the 480 photo batch, what happens?

[zefelder]

That's a good idea. I tried something even better. I restored the latest backup of my catalog which is from a few days ago.

I can reproduce the issue. I looked at three of the "suspected duplicates" and for each of them I found a file in my catalog with the same name and the same size but a different date (such as a few years ago). I tried a few photos that were not suspected duplicates and indeed for none of them I had any similarly named and sized existing photos. 

So it is a bug. I am keeping the SD card and the old cataloge file in case you need any other data, let me know if you need it.

[zefelder]

Please note that this bug -- if it is not somehow specific to my particular setup/system -- is of a high impact as it may easily lead to irrecoverable loss of photos. E.g. in my workflow I typcally have see only "New photos" when importing. So I would never notice that some of photos were not imported when I select all and import. Then after reusing the SD card the files are gone forever.

[Rikk Flohr: Photography]

Can you duplicate if you perform the test as I prescribed? 

If not, if you reattempt on the same test catalog, does it show all images as duplicates?

[zefelder]

Just tried that. The issue does not reproduce with an empty catalog, there are no suspected duplicates.

I am certain it only reproduces if there are photos with the same size and name, even if taken at a different date.

[Rikk Flohr: Photography]

Thank you for running the test.

In your previous test - where you restored your latest backup catalog and identified "three selected duplicates". If those 3 suspected duplicates are added to the test catalog I requested and an import is performed again, does it fail on those images? It may be the only way to be sure those are a trigger. 

[zefelder]

I tried that. I openned an empty catalog and imported a couple of "suspected duplicates" photos and a few others. When I try to import again all these (and only these) photos are shown as "suspected duplicates" which is expected.

I however went one step further. I know that one of "suspected duplicates" on SD card which I did not yet import in the prevous step is named DSC00692.ARW and it has size 20,853,760 bytes. I also know that I have another photo named DSC00692.ARW, taken in 2015 and it also has size 20,853,760 bytes. So I did the following:

1. I imported the DSC00692.ARW taken in 2015 to the new catalog.

2. I openned an import dialog for an SD card that has DSC00692.ARW taken in 2022 and which is not in the catalog.

Result: DSC00692.ARW is now shown as "suspected duplicate" whereas it is not.

So this bug is trivially reproducible starting from an empty catalogue.

FWIW, I run LR classic 11.1.

[Rikk Flohr: Photography]

Would you be willing to share the test catalog -along with images as well as an image that is incorrectly identified as a duplicate? 

[zefelder]

I also could reproduce this bug on LR classic 10.3 that runs on a different computer (and also OS X rather than Windows 11). I started from an empty catalog of course.

[zefelder]

I can share the two photos, there is no need in anything else as the bug can be reproduced starting wiht an empty catalog. How can I send the photos (I obviously don't want them to be publically acessible)?

[Rikk Flohr: Photography]

I will require the test catalog with the single image imported. I will also need the single file that is identified as a duplicate.  In the Test Catalog - select the image and File>Export as a Catalog and then zip the resulting folder.  Zip the Raw File.  They will likely be too large to email so please upload them to a file sharing service (Creative Cloud folder, Dropbox, or similar). Once the two files are uploaded, you can direct message me the link via this forum. 

Thanks for your help. 

[zefelder]

I've sent you a direct message.

[Rikk Flohr: Photography]

The file needs to be a Zip not a RAR. Thanks!

[zefelder]

Sorry. Sure. Sent.

[Rikk Flohr: Photography]

I received the files - thank you.

The two files are identical in size and file name but not capture date so - this is potentially a bug. The file should import under the current documented criteria. 

Do I have your permission to include the file and catalog in the bug report?

[Rikk Flohr: Photography]

Setting status - adding bug number for cross-reference. 

[zefelder]

Yes, you can include the files in the bug report. I assume it is internal to Adobe.

How can we learn about the progress on fixing it? I'll have to change my workflow till it's fixed so that I don't lose photos.

This affects many unaware users in the same way (some photos suddenly don't get imported, lost forever on card reuse). Thus sounds like a high priority bug.

[Rikk Flohr: Photography]

Thank you. The files will be internal to Adobe.

This thread will be updated when a fix is issued. There is no ETA. 

[Rikk Flohr: Photography]

We are unable to reproduce this now using your test catalog files.

Request from Engineering: 
"Is it possible to get the system info of the user or screen recording for  the same?"

Is it possible for you to provide this?

[zefelder]

I could not reproduce this myself using the links that I sent to you.

BUT!

I could reproduce it using the original files. It also reproduces on the latest version 11.3.

I found that if I upload the RAW file to Google Drive and download it back, the issue does not reproduce anymore. Moreover if I simply zip this file and unzip on my computer, the issue does not reproduce. However if I compress it as rar, upload, then download and uncompress, the issue reproduces. It has to do something with file metadata, but not something that can be checked in windows explorer (file properties), as I don't see difference there in file size, dates, or attributes.

In order to send this RAW file intact to you I am sending link to a RAR file which contains it. If you can suggest another way of sending it *intact* - I'll do anything you'll ask.

Please fix this bug...

[zefelder]

Note that this bug is reproducible at least in Lightroom Classic versions 11.1-11.3 in Windows 10, Windows 11 and macOS Monterey (12.3.1). I believe I also reproduced it in 10.3 a while ago.

[Rikk Flohr: Photography]

The SDK will be updated later this week (June 13). The update contains a fix for this issue.  Please download and the latest version of your application via the Adobe Creative Cloud Application or your respective device app store.

If you do not see the update (Mac and Win) you can refresh your Creative Cloud App with the keyboard shortcut [Ctrl/Cmd]+[Alt/Opt]+[ R ]. 

Note: App store availability can take several days for the update to appear and be available. 

Thank you for your patience.