Problems with synchronizing, duplicates and renaming

[DavePinMinn]

Same HUGE mess of umpteen folders, but three NEW bizarre problems...  I need a really good answer that'll get this working...

I've got a setup now with year folders from 2006 to 2022, holding around 30,000 images.  Now we've get to get all the OTHER folders imported into the years.

PROBLEM 1

This morning, doing a Synchronize Folders showed X number of images import.  Clicked the show import dialogue and started the sync.  It did the metadata update, then should have opened the import dialogue.  Instead a partial spinning circle where the folders were, that stopped after half a rotation, and the normal “busy” rotating circle for the curso.  Never opened, never finished, just hung.  Crashed LR, retried a couple times.

And YEAH, since I KNEW the first thing somebody in here would say was “delete the preferences” I did.  Other than screwing up most things that were configured it did NOTHING.

Tried again on a different folder.  Worked FINE.

Tried another folder, got the same thing - no import dialogue, hung Lightroom, had to crash it.

If I DON’T tell it to display the import dialogue, it proceeds to import the images just fine.

Doesn’t seem to matter if it’s three images to import or 500, almost all the time, it hangs Lightroom.

That’s problem ONE.

PROBLEM 2

As I understand it, Lightroom thinks every image in the entire catalog will be unique.  HOW does it decide what “unique” is?  Is it JUST the filename?  Is it the filename and date?  PART of the filename?  Something else?  'Cause it's NOT working currently using the filename...

In this catalog, there are MULTIPLE occurrences of a filename.  They shoot Sony so the filename is something like DSC1234.ARW, and this image in the 2016 folder.  You import another bunch of images, and there’s a DSC1234.ARW going in the 2020 folder.  They’re NOT the same image, they're YEARS apart, but they have the same name.

HOW do NORMAL people handle this – people that DON’T rename every image they shoot?  In this system there are often at least THREE images with the same name in different years.

What’s BEEN happening is we find one of the umpteen folders of images and import it into the correct year.  BUT, a bunch of images fall OUT because they’re duplicates.  The “don’t import duplicates” gets turned OFF because they’re NOT duplicates, they’re DIFFERENT images with the same filename but years apart in the capture date.

So HOW do y’all handle the problem – rename every image?  Something else?  And how do NORMAL people without the expertise in here do it?  'Cause I don't see the people I'm helping having ANY clue about renaming images on import...

PROBLEM 3

In THIS case, I have a bunch of images that actually ARE duplicates going into the 2008 folder – it’s a very SMALL folder of 8 images, so it’s perfect to experiment on.  I elected to rename the images, which is where the NEXT problem showed up.

I created a preset to put a six-digit year (YYMMDD) on the front of the filename, followed by an underscore then the “Original Number Suffix”.  When I used this on one of these Sony images, DSC12345.? Instead of getting 161013_12345.jpg or ARW or psd, I got 161013_2.jpg or whatever.  EVERY TIME.  I absolutely CANNOT get Lightroom to use the original filename OR THE “Filename Number Sequence” - it will NOT put the number from the filename into the renamed image.  What I FINALLY did was use the WHOLE filename.  So, the rename is 161013_DSC12345.jpg.  Which works, but it’s ugly and longer than needed.  But I haven’t been able to get anything ELSE to work.

NOW comes the REAL fun.  In the 2008 folder of 8 images I have DSC12345.ARW.  In the 2020 folder I ALSO have DSC12345.ARW.  Same EXACT name, 4 years apart.  I attempted to IMPORT a DSC12345.ARW that matches the one in 2008 – it’s the same image.  It IS a duplicate.  Lightroom removed it from the import display as it should.

I RENAMED the 8 images in the 2008 folder so they looked like 080512_DSC12345.ARW.  I tried the import AGAIN.  And again, Lightroom said I had a duplicate – I PRESUME this time it was complaining about the one in the 2020 folder.  I renamed the DSC12345.ARW in the 2020 folder to 200419_DSC12345.ARW and a check of the ENTIRE catalog shows NO OTHER IMAGES WITH  DSC12345 ANYWHERE IN A FILENAME.  So importing an image named DSC12345 SHOULDN’T have a problem – RIGHT?

It does.  Lightroom STILL turn the image off EVEN THOUGH there’s NO duplicate filename in this catalog.

WHY ISN’T THIS WORKING?  Does the import object to a PARTIAL match on the same date?  What IS it doing?  The whole point of doing this is so we can be SURE when an image is shown as a duplicate that it actually IS a duplicate.

How DO I get this to work?

[SeanMcCormack]

Don't import suspected duplicates used to be filename, capture time and file size from what I remember. If any of these are different, it will fail. 

If a particular change causes a fail, it makes sense to just not to do that. I've seen where Lightroom loses the suffix on edits, on it's rather annoying. I'd suggest not using the date rename until you at least have the suffix established. Also use - instead of _. The - reads as a space online so if you use custom names, it's searchable. 

I rename every image on Import. It's a sticky setting and means every image has a unique name. Date-Custom Text-Original Filename Suffix. It may be that your lack of text inclusion is what's confusing detection of the suffix. 

[DavePinMinn]

Yes, in the SAME folder it appears Lightroom uses all those - and maybe pixel dimensions, I'm not sure.

In DIFFERENT folders I'm not sure what it uses if you try to import a duplicate.  BUT, you can turn off suspected duplicates and it'll happily import and once it's in, Lightroom doesn't care.  Not even synchronize folders picks it up as near as I can tell.

It's turning out there are a LOT of ways to screw up Lightroom if you do things wrong.  And if your Lightroom catalog somehow got completely messed up and you just switched to Bridge for a few years, it looks like EVERYTHING is wrong!

Thanks for the idea about the "-" instead of the "_". 

Like you, I rename every image on it's way into Lightroom using a YYMMDD_<4-digit-sequence>, and that's always helped keep me from doing anything DISASTROUSLY stupid.

BTW:  As far as renaming, I tested a couple images from different Nikon cameras this morning.  They renamed perfectly, pulling the camera's sequence number off the text.  No problem at all.  I ALSO tried an image from an Olympus - this one looked like _1234567.ORF and it too renamed perfectly, becoming 200813_2134567.ORF.  The ONLY camera I've seen so far where it DIDN'T work was the Sony I'm dealing with.  I don't know if it's ALL Sony cameras or what.  BUT, renaming with the date and the WHOLE filename does work.

[dj_paige]

Yes, in the SAME folder it appears Lightroom uses all those - and maybe pixel dimensions, I'm not sure.

In DIFFERENT folders I'm not sure what it uses if you try to import a duplicate.

The documentation is clear about what it uses, and @SeanMcCormack has stated it clearly. It applies across all photos that have been imported, in any folder, same rules apply whether you are talking about one folder or different folders. Having said that, I suspect this is a buggy operation, as over the years people have complained that it isn't working properly.

[SeanMcCormack]

And I'd agree on that @dj_paige , definitely issues with it where it doesn't work as expected sometimes and it's not always repeatable. 

[DavePinMinn]

So far, it's been an interesting learning experience finding out how users may create problems.

We've actually been successful at getting images INTO Lightroom.  One problem that still exists is because of how they were previously stored (trust me, you DON'T want to know), there are "duplicate" images - in at least one case yesterday, there were SIXTEEN image files of an image - at least 4 .ARW, 4 or 5 .psd, and 7 or 8 jpg.  This is NOT unique by any stretch.  Some have slightly different names, some are different dimensions, but finding the important ones to keep is/will be a manual process.

The other problem is there two users.  User 1 shoots 2 cameras, A and B1.  User 2 shoots B2.  All are Sony’s, B1 and B2 are the same model.  In Lightroom, there I haven't found any serial numbers for any of the 3 cameras, either in library filter or in Metadata panel.  So there’s no way I know of to tell the 2 “B” cameras apart.  And none of them have had the filename modified to make them identifiable – they all generate images of DSCnnnnn.ARW or TIF or jpg.  And they share lenses between the three cameras.

We've already found a half dozen sets of images that collide by capture date - both users shooting B1 and B2 at the same time so the images are mixed in Lightroom when viewed by Capture date. 

And, we've alread found several sets of images that collide by filename.  Different images with the same filenames going into the same folder.  For example, user1 shoots image 3731 on B1 and user2 shoots image 3731 on B2 three days later.  So there are whole series where file name collisions are occurring.

The only recommendation I have is that they use separate catalogs.  Unfortunately, it LOOKS like there may be a significant number of these collisions, and since there are duplicates anyway, this often results in 3731 having -2, -3, -4 and even more versions of the colliding images, possibly with multiple file types.

In this case, I don't think renaming the images would fix the problem because they'd still be jumbled by capture date.  Does anyone have an idea for handling this OTHER than separate catalogs?

[SeanMcCormack]

You can definitely create Smart Collections with Serial Numbers, and it's visible in the EXIF option in the Metadata panel. 

If the serial number is not coming in, that's on Sony. 

[GoldingD]

One question:

Those photos in those folders you want moved, were they already in the catalog? I ask this as importing would not be the normal process. if they had already been imported.

One comment

I for one, rename my photos at import, I use the syntax YYYMMDD_hhmmss. This eliminates any issues of identical names, and allows a cleaner sort. (note, LrC will add a suffix in case of photos shot in the same second).

Personally I absolutely hate using date specific folders, but that is my workflow.

[DavePinMinn]

No.  There were dozens of folders with images in them.  The user said they ORIGINALLY started in Lightroom, had so much trouble they quit, and switched to Bridge.  They ended up with dozens of folders of images on TWO different hard drives.  The user insists they have NO idea how they GOT all these folders with MANY MANY duplicates.  So far, we've managed to get about 42,000 images into Lightroom from these random folders,  I was able to START with a couple folders that Lightroom still had, but there was NO metadata ever saved, so all the metadata that's available is in images from Bridge.

Today I found another 109 (yes a hundred and 9) additional folders of images buried in a folder called 2020 inside inside a folder called "landscape backup".  Those contain another 860 GB of image files. There wasn't a lot of organization.

As for serial numbers, the user's Library Filter metadata showed at LEAST 10 cameras.  Every Canon, and Fuji, and Nikon showed a serial number.  The three Sony's - BLANK.  In the Metadata panel, everybody has a serial number.  Except the Sony's.

[SeanMcCormack]

That sounds like a world of pain Dave. 

If there's no camera serial metadata, it's gonna be pretty hard to separate them. If force a Read Metadata from file on a random file, does it update? I'm gonna assume that the serial number is actually in the metadata somewhere, just not where Lightroom is expecting it to be. Maybe try Jeffrey's EXIF Viewer: http://exif.regex.info/exif.cgi 

It may give you an alternative field to filter on. 

One suggestion on getting stuff where you want it to be: 

Save out the metadata for sets of files. Use delete and select the Remove option to get them out of Lightroom. Then import them again, setting your own destination and using the Move option to reposition them on disk.

[DavePinMinn]

It HAS been interesting... 

Unfortunately, there's no way to save out the metadata - the images weren't IN Lightroom.  Anything that HAS metadata got it from Bridge.  I'd estimate that with the years in years with folders for each and all the other stuff, we're looking at 200 folders minimum.  We're now down to 50 remaining because I took the 109 we most recently found, and reformatted the filenames so they're MOSTLY the year of what's inside.  Now I can take 10 or 12 files at a time, knowing they'll all go into the same YEAR folder in the new catalog.  So, first import is New Images with DON'T import duplicates turned on.  That puts all the NEW images into Lightroom.  This is done by year so if they're 2019 pile of images, SOME will be 2006, 2011, 2014, 2017 and the majority 2019.  So that first, new image import may take 5 or 6 imports.  THEN, the import of all the images that MIGHT be duplicates.  Same thing with different years, so that may be another half dozen imports.  Each of THOSE imports is marked with ared label so the user knows they have to SEE if they're duplicates or not. 

BUT, because of the collisions we regularly hit situations where 2 files with the same name are DIFFERENT, so those have to be manually checked.  Thousands of them.

Eventually, it'll get down to the slow, grunt work of going through thousands of images getting rid of duplicates.  Unfortunately - AGAIN - a lot of the "duplicates" AREN'T.  The user got in the habit, whenever they worked on an image of copying EVERYTHING into a folder.  So, the original image would be put in a second folder.  Then they'd do something in Bridge, so now there's a NEW RAW image that's been modified and the old RAW image that hasn't.  Then they'd go into Photoshop and SAVE that.  REGULARLY.  So there might be FOUR Photoshop files in that folder.  And along the way they'd save a bunch of jpg files, with little names in the filename so DSC1234 might have DSC1234.jpg, DSC1234a.jpg, DSC1234 copy.jpg, DSC1234 f.jpg, DSC1234.jpg, webDSC1234 copy.jpg, DSC1234-recovered.jpg, DSC1234-recovered-recovered.jpg and on and on.  Along with several .psd or .tif files with different names. 

THEN, they'd decide to do something ELSE and copy the WHOLE BUNCH into a NEW folder and start again.  More photoshop, more jpegs. 

Nowe we're trying to consolidate it all into a viable Lightroom catalog.

I've learned a TON of stuff about how NOT to do things...  EVER.  I've ALSO learned a LOT about things Lightroom does that aren't real productive in this situation, like not being able to stay in the import dialogue for multiple imports to different destinations.

I THOUGHT I could use the organize by date and just use a YEAR so EVERYTHING a folder would go into the correct year folder, but there's no template for JUST the year, and I haven't figured out a way to create one.

So, it's multiple imports from the same folders...