Lr folder contains duplicate raw files

Explorer ,
Nov 26, 2018 Nov 26, 2018

Copy link to clipboard

Copied

I have a Lightroom folder (Lightroom Classic CC 8) that is screwed up.

The problem started with an attempt to copy one folder (catalogue + images) from my desktop PC to a portable external drive.

First, I imported to the external drive catalogue - but mistakenly used MOVE instead of COPY.

I then tried to correct this by importing the images from external drive to desktop using ADD.

Inspection of the Lightroom folders then showed that neither the desktop nor external drive had a full set of images.

The good news was that I didn't seem to have lost anything crucial - images missing from desktop were present on the external drive (for the most part). The number of files in the problem folder was not large, so I backed up everything and set about working in in Lr manually gathering the important images into one folder. The folder had originally contained raw (CR2) files, Photoshop edits (TIF) and virtual copies of some raw and TIF files, totalling maybe 100-150 images in Lightroom.

I soon came across a strange problem - the Lightroom folder on my desktop PC contained several duplicated raw files. In image display grid and the filmstrip these duplicates have identical file names and (almost) identical metadata. The only difference in Lr-displayed metadata is that one has keywords; they are missing from the other. Some of these raw files have virtual copies. When I remove one of the duplicated raw files using Remove photo ... > Remove, both images are removed together with associated virtual copies. I still have the single raw file in File Explorer, but the virtual copies are entirely removed from Lr.

Can anyone suggest how I can get back to just one of each raw file in the Lr folder without losing virtual copies?

Views

1.6K

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct Answer

LEGEND , Nov 26, 2018 Nov 26, 2018
You might want to go over to Adobe and add a plugin to find duplicates.TeekesselchenTried it, works on v8. You download it, unzip it, move it to where you or LR wants it (same folder as other normal plugins works)Not destructive, adds a keyword and populates a collection with possible duplicates. This would trim down u=your search for duplicates, On the suspected duplicates, right click, and show in explorer (looking for images in wrong folder), and perhaps give the suspect a label color, in you...

Likes

translate

Translate

Translate
Adobe Community Professional ,
Nov 26, 2018 Nov 26, 2018

Copy link to clipboard

Copied

Have you made Lr backups all along the way? Wondering if it wouldn't be more efficient to open a recent backup and do it over?

Also do you have 'Do not import suspected duplicates' checked?

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Nov 26, 2018 Nov 26, 2018

Copy link to clipboard

Copied

So, when you accidentally clicked on ADD, LR did exactly that, it added instances of your images to the catalog, it did not copy or move the images, the images remained on the external hard drive. Now when you copy those images to your internal hard drive, to the original folder your Lightroom catalog was already setup to find the images in, you had an actual image file in that folder, AND an instance in LR of the image that is on the External hard drive.

Really FUBAR at this point.

Solution/Fix??

Actually a bit sticky, an issue will come up about bringing the LR edits with the images.

I keep thinking, typeing, rethinking, deleted a response. One that might get you your edits back, get rid of the clutter, get the kwywording back.

When you say, you see duplicates, are they in the same folder?

Oh, and if you disconnect the external hard drive, do they show up as missing?

Thinking about copying all image files from external to internal, back where they were, dissociating external hard drive from original catalog. Might even have to reverse the import.

Of course, next time, why not simply copy the h=whole entire folder containing your catalog and sub-bolders with images to the Ext hard drive?

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Nov 26, 2018 Nov 26, 2018

Copy link to clipboard

Copied

@davidg36166309 ... Thanks for that - you prompted me to read what the ADD input method actually does. It's a bit late for me because it doesn't do what I had thought.

The keywords are not a big problem because they are easy to redo.

I removed the external drive thinking you had got at least part of the answer, but the duplicates are still there. At some level Lr really does think that both of the duplicate files are in the same folder on the desktop PC.

The external drive catalogue opens OK. The only problem is this one folder which does not contain all of my files, just a subset. But the catalogue isn't complaining about that, just me.

It currently looks like I have to accept the loss of some virtual copies - that would be irritating, but I still have all of the raw files and almost all of the Photoshop edits.

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Nov 26, 2018 Nov 26, 2018

Copy link to clipboard

Copied

You might want to go over to Adobe and add a plugin to find duplicates.

Teekesselchen

Tried it, works on v8. You download it, unzip it, move it to where you or LR wants it (same folder as other normal plugins works)

Not destructive, adds a keyword and populates a collection with possible duplicates. This would trim down u=your search for duplicates, On the suspected duplicates, right click, and show in explorer (looking for images in wrong folder), and perhaps give the suspect a label color, in your main library folder, look for the red label ones, perhaps the non duplicate right next to it? and adjust keywords as required (could select both, edit keyword tags to suit) . this might require you to select the option to show photos in subfolders, and to list by file name.

Oh and it is run from /LIBRARY/PLUG-IN Extras/

oh oh oh!!!!!! like the little kid

And this link makes the above a whole lot clearer and comes with advice:

https://www.lightroomqueen.com/clean-duplicate-photos/

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Nov 27, 2018 Nov 27, 2018

Copy link to clipboard

Copied

davidg36166309 Many thanks. I have marked your response as "correct answer".

I shall investigate the plugin for finding duplicates.

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Nov 26, 2018 Nov 26, 2018

Copy link to clipboard

Copied

@cmgap ... Unfortunate, but there is no backup that helps. I normally backup on exit, but Lr had been running for more than 24 hours. As for 'Do not import suspected duplicates' I have always had this checked and it's still checked now.

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Nov 26, 2018 Nov 26, 2018

Copy link to clipboard

Copied

A different thought. If you open the external hard drives catalog. Does it behave?

If so, perhaps export as a catalog from it to new internal catalog, new location?

Move old one to a separate folder on external, tiol you are happy you can trash it.

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jan 09, 2021 Jan 09, 2021

Copy link to clipboard

Copied

Since I found Visual Similarity Duplicate Image Finder I stopped using those plugins as they scan EXIF data and are very unreliable. This tool provides true image analysis and can find not only duplicates but also similar photos.

Has support for 300+ RAW camera formats and 40 popular image formats - visible on the screenshot below.

main

 

There is a short Lightroom tutorial here: Lightroom Duplicate Finder 

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jan 09, 2021 Jan 09, 2021

Copy link to clipboard

Copied

This software is looking very promising to find duplicates. I am following its development keenly. (The [Scan Lightroom] button looks promising)

But, and this is a warning, or a Caution- The interface TBMK does NOT indicate which of the files has been edited in Lightroom-Classic.

ScreenShot357.jpg

You may have spent many hours in LrC developing one (or both) of these (eg. files as in the screen-clip), so which one will you delete? How will you check that? Delete the 'wrong' one and you have lost all your work in LrC unless you can re-link  the now 'Missing' photo in the Catalog Library to the 'other' file in another folder..

IMO- the Plugin Dup Finders are still currently the 'safest' to avoid 'damaging' the Catalog.

 

Regards. My System: Lr-Classic 10.2, Photoshop 22.3, Lightroom 4.2, Windows-10 Nikon DSLR.

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jan 10, 2021 Jan 10, 2021

Copy link to clipboard

Copied

Thank you very much for pointing that out, but I can see instantly if an image is edited in the preview panel. Actually there are even two preview panesl. One is multi-preview that shows all simialr images and the other one is the regular preview which shows two images side-by-side for more details. Clicking a preview image shows a full-screen preview too (this makes them 3 previews sorry) which you can also use to navigate and mark photos without having to close and open it every time. Overall I am more confident using this tool than any of the plugins that are available. 

If you want to stay on the safe side you can also move the duplicates to a temp folder prior to completely removing them. Definitely there is no way to "damage" your catalog.

Why I do not like plugins:

1. They can not list similar images

2. They rely on EXIF and list completely different photos as duplicates

3. They are limited only to LR and you can not check if a folder is already imported. The above tool can compare your catalog to a folder so you can check if images in a folder are already imported prior to generating new duplicates.

Of course if you are used to plugins then you will need some time to get used to the tool as it works in a completely different way, but if you play wiht it for a while you will understand that it is much more flexible. It can keep the higher-quality photos automatically, can makr photos by various properties and much more. 

I did use those plugins too, but since switching to this one I never needed to go back.

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jan 10, 2021 Jan 10, 2021

Copy link to clipboard

Copied

"I can see instantly if an image is edited in the preview panel."

I am not convinced! I do NOT see Lightroom edits in VS Dupe Finder

 

Here are screen-clips of two files I have- an original and a copy. The Copy version has been extensively edited (as you see it) in Lightroom-Classic:

ScreenShot365.jpg

Now, here is the same two files as shown in the VS Dupe Finder-

ScreenShot366.jpg

Without Viewing the Lightroom Catalog- Which of these two files in VS Dupe Finder should I delete without losing my editing work in Lightroom? VS suggests I delete the 'Copy'-

I did that (Delete the Copy), now this is what I see back in Lightroom-

A 'Missing' photo thumbnail, and files that cannot be found when I try to continue editing-So now I have a messed up catalog with many missing files and my editing work gone!

ScreenShot367.jpgScreenShot368.jpg

 

 

 

Regards. My System: Lr-Classic 10.2, Photoshop 22.3, Lightroom 4.2, Windows-10 Nikon DSLR.

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jan 10, 2021 Jan 10, 2021

Copy link to clipboard

Copied

Your reaction is pretty strange - if you like the plugins then use them, but trying to persuade me that the tool is not good is pretty strange. Exactly the opposite - the tool is great and can do much more than all the duplicate finder plugins available for LR combined.

 

Your claim is not quite true:

"I did that (Delete the Copy), now this is what I see back in Lightroom-

A 'Missing' photo thumbnail, and files that cannot be found when I try to continue editing-So now I have a messed up catalog with many missing files and my editing work gone!"

 

You can not have a messed up catalog as the tool does not modify the LR catalog. That is why it has displayed a meessage to you how to remove the missing entries automatically in LR after you have moved/delete duplicates. If you have not done that of course you have missing file - you have personally deleted it. Same happens if you delete a file outside LR. And it is a single file if what you describe is correct - not "many missing files". That is also visible from your screenshot.

 

Anyway, as I already wrote I am not persuading you to use the tool. If you don't like it just use whatever you like, but what your describe about "many missing files" is absolutely not true - I am using this tool for years. I do not understand why you are trying to turn this into an argument with your post. That is really pointless...we all use whatever is better for our productivity. If what I suggest does not work for you then just don't use it.

 

No hard feelings...Cheers. 

 

 

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Jan 11, 2021 Jan 11, 2021

Copy link to clipboard

Copied

Sorry to appear argumentative 😞

My discussion was not to "persuade" but to point out the one weakness of many Dupe Finders that are not a Lr plugin. (My discussions tend to get verbose- my problem.)

The VS Dupe Finder IS the best I have seen amongst many others, so my full support there for this software. And the [Lightroom] button (that no others have) is getting it closer to 'perfect'!

Thanks for the discussion. Keep on using VS. I will be also.

Cheers

 

Regards. My System: Lr-Classic 10.2, Photoshop 22.3, Lightroom 4.2, Windows-10 Nikon DSLR.

Likes

translate

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines