P:a No way to be sure if photos are in the catalog. zero.

[andybeales]

This post contains a video, for a full easy explanation of my situation. 
Video
https://www.youtube.com/watch?v=sThVzs0Q5pc&feature=youtu.be

The Issue in a Nutshell

You might say Import does not detect duplicate photos. So you have no way to point at various folders and be sure you imported that data. So I have no easy way to find out if my master catalog contains all my various drives (that I am sorting out) data.

I discovered this after doing a catalog import, I then did a standard import, not clicking import, pointing it at the photos folder of the catalog I just catalog imported, just to check if catalog import had worked. I selecting 'Don't import suspected duplicates', expecting to see zero or only a handful of orphaned files caused by incorrect deletion. It sees everything, every photo in the catalog! 

The main things I tried which I think relevant to say...

• Since pointing it at the original /photos location on the other catalog drive did not work... I copied it to the same drive = no difference. (Some people talk about external drives causing issue, it ruled that out)
• Pointed it at its own /photos directory = no new photos detected
• Pointed it at its own /photos/folders I copied out to the desktop = new photos incorrectly detected.
• Create a new empty catalog and did a catalog import of the mothership catalog, repeat = same result in all cases of above tests.
• Try with one single folder of just 78 images = the same
• Tried on another laptop with the same version of Adobe to rule out virus, bug, software, laptop, etc... = no difference, it's not my kit.

History
I started with a detailed Community Forum Post: https://forums.adobe.com/thread/2443613 and then since I discovered Adobe don't bother checking those, I then moved to a support case. Then I was asked to close my Adobe Case Number ADB-928182-W8X1 and post it here as a bug. After spending about 3 days working on this problem the lovely people at Adobe really touched my heart by giving me a one month credit for the Photography plan. How totally touching, moving, rewarding, understanding of them. No bug (or do they call it an enhancement request these days) provided, I feel like posting it here is a just a place for customers to vent, nothing more.

I think catalog systems are a terrible way to design software. I like how Exposure uses single tiny files in the images folder, as it means multi-users can also work on the same data together. Maybe it's time to move off Lr.

Version
I am using most up to data Lr and stuff, Lr Classic CC 7.1

[bob frost]

Have you renamed all your files already on your computer?

Bob Frost

[andybeales]

Hi Bob, I never rename or touch the RAW files before import from card, merge, or anything, I only deal with RAW. I let Lightroom import and manage them from the card. The only rename that is sometimes done is within Lightroom once a photo is ready to publish, at that point IMG_1234 might become a better clearer name for sharing and posting online. I rename in Lightroom using the Library tab and within the side panel. In terms of directory structure, again I never touch this except on occasion I might rename a few folders inside the Library folder tree within lightroom to have a more meaningful name on the end, e.g. YYYYMMDD turns to YYYYMMDD D.R. Congo. But when I compared data not seen as duplicate it was not a folder that had a meaning full name added, nor was it a folder that had any files renamed inside it, just 20170226.

[bob frost]

I seem to remember someone saying that LR compares the filenames of new files with those in its catalog. So if you have changed the name of the file in the catalog, it will think the same file with its original name is new. Should be easy to try. I may be wrong; my memory is not so good these days!

Bob Frost

[Jim Wilde]

I'm having the same memory issues, Bob.

I have to say that I've tried several times to recreate the issue that you're seeing, Andy, but so far without success. Every test I've tried so far, pointing the catalog to a copy of already cataloged images results in them all being correctly detected as duplicates (though that is NOT an instant process, it can take a while for the dupes checking process to complete....but that's seconds not minutes or hours). But maybe I'm not testing in the right way, and this is where my failing memory comes into play. I know that I've seen something similar in the past involving the Import from Another Catalog routine, but I can't remember the full details.

One question: does this problem (of pointing to another copy and not having them being detected as duplicates) happen with ANY folder, or only on folders that have been involved in a catalog Export or Import procedure? In other words, if you just import a folder of images from a card into Lightroom, close that catalog, copy that folder to the Desktop, restart Lightroom using the same catalog, then try to import that copy folder from the Desktop, does duplicates detection work correctly or not? 

[andybeales]

 Hi Jim, it's not file renaming, the examples I am looking at are nearly all not renamed. But your last question is a very good question...I think you might be on to something, I guess you're thinking.... 'Maybe the import doesn't detect duplicates if they were previously imported as a catalog import',  or 'maybe the import is changing the file meta in someway incorrectly and so if you point at the source again it doesn't recognise duplicates'...

THIS IS A BUG - HERE'S THE RESULT.
It's specific to data that has previously been catalog imported, even with a brand new empty catalog running tests today I can recreate it every time. Jumpy to the summary at the end of this post and read the 4 tests.

BACKGROUND FULL WORKFLOW AND PROBLEM IF YOU NEED
But just jump to the 4 tests at the end....

TEST 1 WORKFLOW
Firstly, like before, I noted a folder (a different one this time 20170219 UK ( I actually did it 3 times with 3 folders all the same results) that was imported in part of the previous catalog import, so it's in the new MASTER and the old TRAVEL catalog. I also checked the catalog is on the disk and in the catalog tree, and looks correct. The no. all match. Please note, these photos first arrived into the master catalog by a catalog import.

1. Point import within the MASTER catalog at its own 20170219 UK folder in the drives /photos folder, where the data should be found, and duplicates were detected correctly. It should see the data is already in the catalog.
2. Copy that same folder above (in the Master catalog photo folder) to the desktop (copy and paste) and point import at that, this time duplicates were incorrectly NOT detected, but they are in the catalog. It thought the images were new. The only change was copying and pasting it outside the directory the catalog points at. To confirm, this folder was previously brought into the catalog as part of the catalog import. I also copied it to the external drive or other locations, all the same result, it thinks they are new.
3. I then pointed import at the folder in the original location and duplicates were incorrectly NOT detected, it seems them as new, but they are in the catalog.

x3 repeats on different folders, all the same.

-----
JUMP HERE FOR 3 KEY TESTS AND RESULTS
------

TEST 1 - TEST ON DATA IMPORTED TO MASTER ORIGINALLY VIA CATALOG IMPORT
Just moving a folder that has previously been in a 'catalog import' in Lightroom Classic CC to a new location and point import at that, and it will not detect the files as duplicates and still want to import them. Hence you can't be sure if all your data is in by pointing at old drives or old directories. 

TEST 2 - CREATE NEW CATALOG > CATALOG IMPORT 1 FOLDER > REPEAT ABOVE
So I created a brand new catalog, brand new photos folder, did a catalog import selecting the one folder 20170219 UK. Then I repeated the above, and same result, it fails to detected the duplicates it should.

TEST 3 - CREATE NEW CATALOG > PHOTO IMPORT 1 FOLDER  > REPEAT ABOVE
So I created a brand new catalog, brand new photos folder, did a standard photo import rather than the initial catalog import, and then pointed at the other folder, it detects duplicates. So we have it, it's specific to the data that has arrived via a catalog import, not a photo import. 

TEST 4 - CREATE NEW CATALOG > PHOTO IMPORT 1 FOLDER > CATALOG IMP 1 FOLDER > REPEAT ABOVE
So I take it one step forward, create a new catalog, import one photo import as test 3 but this time then move on to do an additional import, this time a  catalog import of 1 folder. Everything was the same as test 3 but it failed to detect duplicates on the folder I imported via catalog import. So that's the problem, data that has been catalog imported previously is always seen as new data when doing a photo import.

SUMMARY OF BUG

Create a new catalog, do a catalog import of even just one folder, copy that same folder to any other location, try a photo import of it, and its contents won't be detected as duplicates, but it should be. If you did that same workflow but with a catalog that had an initial photo import, not a catalog import, and there is no problem. It's specific to data that has been catalog imported.

STEPS TO RECREATE

1. Create New Catalog on Lr Classic CC
2. Import a folder of photos with PHOTO IMPORT
3. Try to import that again with PHOTO IMPORT and you'll see duplicates correctly detected.
4. CATALOG IMPORT one more different folder to that same CATALOG.
5. Try to import that again using PHOTO IMPORT your see duplicates incorrectly detected. So you have no way when sorting out data and checking drives to know if your data is in the catalog.

[Jim Wilde]

Yes, I guessed that would be the case. I finally got some memory cells working and recalled the issue that I had seen (and reported to Adobe) several years ago, and tested it again (in much the same way as you did) to verify my memory.

Basically, what happens is that, as you may know, Lightroom's duplication checking is based on three criteria....file-name, capture date, and file length. When you import images these three bits of information are stored in a field (called the Import Hash) within one of the catalog database tables....and it's that hash that's subsequently used to compare against new imports. 

Generally that all works well, except that when you do the "Import from Another Catalog" the contents of the Import Hash for all the images being imported from the other catalog are not brought in, i.e. after the import from the catalog, the Import Hash is blank for all the imported images. Thus when you subsequently attempt to import those same images from another source, the duplicates checking fails because there's no Import Hash to check against. I have a feeling there are some other instances as well where something similar can happen, but will definitely need to trawl through some old emails and/or forum posts to confirm that.

I hope that's at least explained what you have been seeing. I'll contact Adobe again about this.....it may be a bit of an edge case, but it's still something that should be avoidable.  

[andybeales]

Hi Jim, it's just mind blowing to think it's been an issue for years. My heart nearly dropped when you explained it like that. Many people have been told (like me) by Adobe support at times to Create a New Catalog and do a catalog import it, as sometimes they think a catalog file might be broken or some other reasons. So if just about everything has that hash empty, I am many others have got no way to know for sure if I have all the data without that function working, especially since I was just merging 3 catalogs. Which leads to the question, how can I trust Catalog Import? lol

I was considering capture One or Exposure for a long time, because of speed and for the RAW engine. since this is in such a mess and I spent so long testing, trouble shooting, maybe now is the time to jump. 😞 I have wasted a whole week on this, what with copying NAS testing unwinding things, logically checking things, writing down file sizes, names, data, waiting for copies or import to run. What a nightmare!

Don't Adobe not have a bug forum or public bug listing? I had no way to check if something had been an issue before. Support just wanted to close the ticket as fast as possible.

I worry that the life and energy has gone out of Classic CC.

[andybeales]

Jim, what is feedback.photoshop.com's affiliation with adobe? I had the strange experience of a support guy closing a ticket with this page left open asking me to log a ticket here, but if feels like a independent forum?

[Jim Wilde]

This IS the official Adobe site for users to post bug reports/suggestions/feature requests, etc. It is managed and monitored by Adobe, and some team members are supposed to read all threads here. So technically you have formally reported an issue, I've added some details about how I think it occurs, so now it's up to Adobe to investigate and prioritise a fix. 

Regarding your earlier point about your concern that the life and energy has gone out of Classic, I don't really share that concern as yet. If it had I don't think they would be trying as hard (as they are reported to be) to fix those long-standing performance issues which have vexed many users over recent versions. So for me it's too early to say that the commitment to Classic has gone, let's see what happens throughout this year, I think by then we'll have a much better sense of their intentions regarding both Classic and LRCC.

[andybeales]

Thanks for clarifying. I just think it's strange they get the user base such as you and me to do all the hard work. Perhaps I am so frustrated I make the point too strong, but they clearly just divided their team on to two separate versions of Lr, and that's never good in software development. We can expect slower fixes and their attentions and efforts to be split 😞  From what I read Lr cloud version has lots of features missing compared to Classic, so I bet that is the focus. I think the speed issue took them years to address, probably because behind the scenes they were working on this cloud version, which is more a consumer focused product aimed at Mac Photos and Google Photo type of users wanting something more in the cloud.

As a side note, Import Catalog also has bugs if you repoint a catalog import at one you already did, to check you go everything, it can't distinguish between the files that were previously imported as virtual copies with changes added, it thinks those are new again. I just can't trust it to manage data, I think I'll have to use Photo Mechanics and change my workflow, which I don't want to do but hey...

Now I am down a rabbit hole with third party plugins like the Teekesselchen open source duplicate locator. Comparing folder by folder for 5 years and 192,000 photos. Let's hope they still bother with Classic CC, I think this should be a priority and not a side bug, as it's about being confident in your data.

Most importantly, It would be nice to have a real bug database online, so users like me can search and find this if its already log. But I guess that would make them vulnerable to people seeing how long this has been an issue, and cause more user discontent. So they create a community forum and let third party people answer it. Sounds like a way for developers who want to avoid official responses to engage disguised as helpful community members, and take the heat off Adobe for such things.

Still, should look on the bright side, they extended my Photography plan by one month for free, worth 10 pounds. x

[andybeales]

Tested again and still present in 7.2

[andybeales]

How does this forum work, is this site just a pool of discontent and a bucket of 'maybe we will work on this' forum? Do the hours and days of tests by the users get turned into a tracked bug with a reference no? Will we even get notified of a buy fix? I don't see how. Do I have to test with each version that keeps coming out again just hoping that someone resolved it. Do you have any freelance jobs in testing and QA going? 😉

ps. we all know the strategy behind the new non-classic version is to be able to make lots of money by charging reoccurring revenue for file storage, and take over the need for Dropbox, and increase profits. So Adobe's business focus has to be focused on bringing non-classic up to classics functionality, and things like this will be ignored.

[john beardsworth]

I wouldn't bother with conspiracy theories (cock up is always more likely anyway). But like any developer, Adobe evaluate bugs according to a range of factors and then prioritise them accordingly. When they are fixed, they are sometimes mentioned here, but I wouldn't bet on it.

Realistically, I wouldn't expect this bug to be fixed in the near or far future - it is pretty obscure and too rarely encountered. So you might find it more profitable to adapt your practices to ensure it doesn't continue to be a problem for you.

[andybeales]

Hi john,

I just wish support don't tell people like me to 'just create a new empty catalog and do a catalog import' as this causes the issue, or tell people who use laptops and desktops to use a travel catalog on your laptop and merging it, or anyone merging catalogs from each year into one master, so I think it affects a lot of people but they don't know it, or don't both checking old cards and drives.
The bug totally destroys anyone's ability to check easily if a directory, memory card, or old drive is present in my catalog without manually checking contents and totals of photos. So for me it's about nobody should ever rely on Lr to check if all the data is in the catalog.

I am not a conspiracy theory, that's a bit strong. It's clear business strategy sense that the guys at Adobe would double revenue if they push people towards online storage fees, so hence CC has to be the focus more and more... which is why I guess things like this will not be fixed. Resources are now split on two different products.

I already have changed my workflow. I can no longer have a master 1 catalog to rule them all. Only way is 1 catalog per project. 

[john beardsworth]

My preference for a travel catalogue has changed over the years. I used to rely on the Import/Export Catalog roundtripping, and I am pretty sure I would have remembered if I had hit this bug. But since the introduction of smart previews I simply take my single master catalogue on the road, with as many smart previews as I expect to need. So no roundtripping, no fragmentation of control over my pictures and workflow.

I really don't think the new "Lightroom CC" has any effect on fixing this bug - it looks like it's been hanging around for years.

[Rikk Flohr: Photography]

Lightroom Classic CC 7.5 was released today and should address this issue.  If you don’t see the update in your Creative Cloud App, hit the [Ctrl/Cmd]+ [Alt/Opt]+[R] to force a refresh. 

Give the new version a try and let us know if it solves the issue.