Skip to main content
dribble97
dribble97
Inspiring
November 26, 2018
Answered

Lr folder contains duplicate raw files

  • November 26, 2018
  • 3 replies
  • 3556 views

I have a Lightroom folder (Lightroom Classic CC 😎 that is screwed up.

The problem started with an attempt to copy one folder (catalogue + images) from my desktop PC to a portable external drive.

First, I imported to the external drive catalogue - but mistakenly used MOVE instead of COPY.

I then tried to correct this by importing the images from external drive to desktop using ADD.

Inspection of the Lightroom folders then showed that neither the desktop nor external drive had a full set of images.

The good news was that I didn't seem to have lost anything crucial - images missing from desktop were present on the external drive (for the most part). The number of files in the problem folder was not large, so I backed up everything and set about working in in Lr manually gathering the important images into one folder. The folder had originally contained raw (CR2) files, Photoshop edits (TIF) and virtual copies of some raw and TIF files, totalling maybe 100-150 images in Lightroom.

I soon came across a strange problem - the Lightroom folder on my desktop PC contained several duplicated raw files. In image display grid and the filmstrip these duplicates have identical file names and (almost) identical metadata. The only difference in Lr-displayed metadata is that one has keywords; they are missing from the other. Some of these raw files have virtual copies. When I remove one of the duplicated raw files using Remove photo ... > Remove, both images are removed together with associated virtual copies. I still have the single raw file in File Explorer, but the virtual copies are entirely removed from Lr.

Can anyone suggest how I can get back to just one of each raw file in the Lr folder without losing virtual copies?

This topic has been closed for replies.
Correct answer GoldingD

@davidg36166309 ... Thanks for that - you prompted me to read what the ADD input method actually does. It's a bit late for me because it doesn't do what I had thought.

The keywords are not a big problem because they are easy to redo.

I removed the external drive thinking you had got at least part of the answer, but the duplicates are still there. At some level Lr really does think that both of the duplicate files are in the same folder on the desktop PC.

The external drive catalogue opens OK. The only problem is this one folder which does not contain all of my files, just a subset. But the catalogue isn't complaining about that, just me.

It currently looks like I have to accept the loss of some virtual copies - that would be irritating, but I still have all of the raw files and almost all of the Photoshop edits.


You might want to go over to Adobe and add a plugin to find duplicates.

Teekesselchen

Tried it, works on v8. You download it, unzip it, move it to where you or LR wants it (same folder as other normal plugins works)

Not destructive, adds a keyword and populates a collection with possible duplicates. This would trim down u=your search for duplicates, On the suspected duplicates, right click, and show in explorer (looking for images in wrong folder), and perhaps give the suspect a label color, in your main library folder, look for the red label ones, perhaps the non duplicate right next to it? and adjust keywords as required (could select both, edit keyword tags to suit) . this might require you to select the option to show photos in subfolders, and to list by file name.

Oh and it is run from /LIBRARY/PLUG-IN Extras/

oh oh oh!!!!!! like the little kid

And this link makes the above a whole lot clearer and comes with advice:

https://www.lightroomqueen.com/clean-duplicate-photos/

3 replies

A
Anonymous
January 9, 2021

Since I found Visual Similarity Duplicate Image Finder I stopped using those plugins as they scan EXIF data and are very unreliable. This tool provides true image analysis and can find not only duplicates but also similar photos.

Has support for 300+ RAW camera formats and 40 popular image formats - visible on the screenshot below.

 

There is a short Lightroom tutorial here: Lightroom Duplicate Finder 

Rob_Cullen
Rob_Cullen
Adobe Expert
January 10, 2021

This software is looking very promising to find duplicates. I am following its development keenly. (The [Scan Lightroom] button looks promising)

But, and this is a warning, or a Caution- The interface TBMK does NOT indicate which of the files has been edited in Lightroom-Classic.

You may have spent many hours in LrC developing one (or both) of these (eg. files as in the screen-clip), so which one will you delete? How will you check that? Delete the 'wrong' one and you have lost all your work in LrC unless you can re-link  the now 'Missing' photo in the Catalog Library to the 'other' file in another folder..

IMO- the Plugin Dup Finders are still currently the 'safest' to avoid 'damaging' the Catalog.

 

Regards. My System: Windows-11, Lightroom-Classic 15.0, Photoshop 27.0, ACR 18.0, Lightroom 9.0, Lr-iOS 10.4.0, Bridge 16.0 .
A
Anonymous
January 10, 2021

Thank you very much for pointing that out, but I can see instantly if an image is edited in the preview panel. Actually there are even two preview panesl. One is multi-preview that shows all simialr images and the other one is the regular preview which shows two images side-by-side for more details. Clicking a preview image shows a full-screen preview too (this makes them 3 previews sorry) which you can also use to navigate and mark photos without having to close and open it every time. Overall I am more confident using this tool than any of the plugins that are available. 

If you want to stay on the safe side you can also move the duplicates to a temp folder prior to completely removing them. Definitely there is no way to "damage" your catalog.

Why I do not like plugins:

1. They can not list similar images

2. They rely on EXIF and list completely different photos as duplicates

3. They are limited only to LR and you can not check if a folder is already imported. The above tool can compare your catalog to a folder so you can check if images in a folder are already imported prior to generating new duplicates.

Of course if you are used to plugins then you will need some time to get used to the tool as it works in a completely different way, but if you play wiht it for a while you will understand that it is much more flexible. It can keep the higher-quality photos automatically, can makr photos by various properties and much more. 

I did use those plugins too, but since switching to this one I never needed to go back.

GoldingD
GoldingD
Brainiac
November 26, 2018

A different thought. If you open the external hard drives catalog. Does it behave?

If so, perhaps export as a catalog from it to new internal catalog, new location?

Move old one to a separate folder on external, tiol you are happy you can trash it.

cmgap
cmgap
Adobe Expert
November 26, 2018

Have you made Lr backups all along the way? Wondering if it wouldn't be more efficient to open a recent backup and do it over?

Also do you have 'Do not import suspected duplicates' checked?

GoldingD
GoldingD
Brainiac
November 26, 2018

So, when you accidentally clicked on ADD, LR did exactly that, it added instances of your images to the catalog, it did not copy or move the images, the images remained on the external hard drive. Now when you copy those images to your internal hard drive, to the original folder your Lightroom catalog was already setup to find the images in, you had an actual image file in that folder, AND an instance in LR of the image that is on the External hard drive.

Really FUBAR at this point.

Solution/Fix??

Actually a bit sticky, an issue will come up about bringing the LR edits with the images.

I keep thinking, typeing, rethinking, deleted a response. One that might get you your edits back, get rid of the clutter, get the kwywording back.

When you say, you see duplicates, are they in the same folder?

Oh, and if you disconnect the external hard drive, do they show up as missing?

Thinking about copying all image files from external to internal, back where they were, dissociating external hard drive from original catalog. Might even have to reverse the import.

Of course, next time, why not simply copy the h=whole entire folder containing your catalog and sub-bolders with images to the Ext hard drive?

dribble97
dribble97Author
Inspiring
November 26, 2018

@davidg36166309 ... Thanks for that - you prompted me to read what the ADD input method actually does. It's a bit late for me because it doesn't do what I had thought.

The keywords are not a big problem because they are easy to redo.

I removed the external drive thinking you had got at least part of the answer, but the duplicates are still there. At some level Lr really does think that both of the duplicate files are in the same folder on the desktop PC.

The external drive catalogue opens OK. The only problem is this one folder which does not contain all of my files, just a subset. But the catalogue isn't complaining about that, just me.

It currently looks like I have to accept the loss of some virtual copies - that would be irritating, but I still have all of the raw files and almost all of the Photoshop edits.

Terms & ConditionsAccessibility statement
Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices