2023 Elements Organizer import causing duplicate photos

@PhotoGeek2121 said:

Hi folks, with Gregs help I think we have figured out what is going on and how to solve it.  I have copied Gregs last msg above (text italicized and underlined below) and replied to each paragraph (my text not underlined). 

If you are each importing images into the respective catalogs from the same device or card, there will undoubtedly be duplicate files created

greg, can you say more about your statement above?

When you import files into a catalog using the APD (Adobe Photo Downloader), you are copying files from the phone or memory card, or USB drive to your computer's hard drive (which, in your case, is the NAS drive).  You have chosen to store your files in subfolders of Date Taken.  (I do the same.)  Once the files are copied to the hard drive, the files are imported into the catalog and their stored name and location are placed in the catalog's database.

Ok, this helps explain the “Two Step Import” see.  The first step starts quickly after “get media”,  then completes (copies images to NAS and deletes from iphone), then I see a pause (and sometimes I must restart APD Organizer) then see the images imported into APD Catalog Database.  It’s odd that the delay I am seeing starting step 2 is the APD catalog DB update (on local hard drive) and not trying to access my local networked NAS (step 1)?   (more below on this topic).

If you use the Organizer on one computer to import files from a device (e.g. your phone), they will be copied to the hard drive and will (depending on the options you choose) have a name such as Img0001.jpg.   That name will now appear in the catalog.  If you now try to import the same photo into the Organizer's catalog on the second computer, using the APD, the file will now be copied again to the chosen Date Taken folder.  However, the OS will not allow two files with the same name to be located in the same folder.  So, the file will be copied to the folder but be given a different name which will usually have a numeric suffix such as Img0001-1.jpg.  Since the computers are using different versions of Elements, they are using different catalogs.  So, when the second catalog attempts to import the same photo from the same device, it will not be recognized as already existing in the catalog, and the copy will be imported (and the APD will not filter it out as a duplicate).

I am using the APD provided post amble 4 digit number that advances over time if the base name (date)? is the same:

I rarely see a “-1” file name due to a windows duplicate. I also rarely (if ever) will upload a successful APD import using my second computer, if it was successful on computer 1, why would I run again on computer 2?  Its only now that computer 1 APD has stopped working that I have attempted to upload the images on that iphone using computer 2.  In this case those files have never been uploaded and are not on my NAS folders.  But if I had I agree that this scenario could cause duplicates.  

Again, my strict workflow is to keep iphone 1 and computer 1 separate from iphone 2, Ipad2, SD cards, and computer 2.  Separate APD’s on each (pse22 on computer 1, pse23 on computer 2).  Local catalogs, but shared NAS “Repository” with folders by year/date.  Over 100K files. I am certainly open to input on a better model.  I have seen some threads here that seem to recommend use a 3rd party app/tool to upload/import, then use the APD file/folder tool to “import” into organizer and then to “manage” the database from there?  Does this make more sense?  APD seems pretty intuitive, I am not sure the extended family can grasp more 3rd party apps and interfaces, and workflow. 

If you have your NAS set up as a watched folder for each catalog, the first computer will import the -1 copy and the second computer may import the original.  But because these are different (but duplicate copies of the) files, both may now appear in each catalog.

Now, having said all of that, you may be having a separate duplication issue with photos imported from a camera memory card.  I have only experienced this problem with my iPhones.  To determine whether this is the case, connect your iPhone via the APD and click on the Select Folders icon.  If you open up the folder hierarchy, you may find that you have two DCIM folders.  If they are both checked for import, this will cause duplicate files to be copied to your computer and imported (as separate files).  Uncheck one of the DCIM folders.

Greg, you are so correct!!!!!

At first I only see one apple folders to pull from, but think this is what you will see before you “trust the computer from the iphone:

After a couple of tries to select and view this “media folder to pull from” this view evolved to:

Notice that my 6 pix have been duplicated to 12, and the iphone sub folders have expanded to the structure seen above.  Further fully expanding to the “down carats” shows this structure where in fact I can select two individual folders.

Deselecting either folder changes the designated import targets to just the baseline 6 unique Pix!!!!!!

I was not aware of this (Apple?) unique nuance where my baseline images are structured within 2 file structures.  Nor was I aware that APD can make this visible (and selectable) using the folder icon in the (advanced?) import window.  Ty Professor Greg for lighting the path of understanding!!

I imported after “unchecking” the 202401_ source, both steps finished back to back with barely a hint of delay…   6 files only.  “Priceless”!!!!  The hangs/delays seem related to this dual source file artifact.     

There may be other possibilities for duplication, particularly given your comments about sharing photos over the iPhone with your wife.  When you share a photo from one iPhone to another, depending on what apps you use, you can end up with duplicate photos in your Photo Library.  However, again depending on the app and how you have saved the shared photo, you may or may not have the shared photo in the folder system from which Elements imports the photos.

A few final thoughts:

I have never used the After Copying Delete Originals option.  For me, that is fraught with danger and I see no benefit to using this approach, particularly given some of the problems you have described with importing.

I find that my iphones become littered with Junk that cannot be uploaded (live photos, slomotion, advanced files formats like HEIC, HEIF, etc).  It’s a constant battle to keep my “photo” icon clean.  Deleting helps that, I also was concerned about dups…   it also helps to keep my media sources clean, if its empty, its been uploaded…     

I believe that the problems you are experiencing with the import process completing only after you relaunch the program is because you are using a NAS which creates performance issues and is probably one of the reasons why Adobe does not recommend using this storage method.

This relaunch artifact seems directly related to the multiple folder sources on my iphone, when I clear those sources, the APD import runs flawless.  When I use APD to import from a SD card it runs flawless… both are using my local NAS repository (and gigabit network appliances).  Yes I can see windows file access pauses once in a while, but APD seems to have my Synology NAS on speed dial.    

As I understand, some of your follow up posts, you were not able to reproduce the duplication problem under some scenarios.  That suggests to me that the program is working correctly and this problem has been caused by some unusual workflows that you are using with the two computers (and two phones).

Yes, hidden Apple “feature” that requires some hand holding to mitigate!!

To further investigate what is happening with your import procedure,  use the Sort by Import Batch option to see what and when each pair of duplicates was imported.

The dups can be seen on the APD input screen, when you have a large number of files being imported it is easy to miss.  Agree that there are likely more root causes to the “duplicate” image “universe”…  but for now lets call this a win and close this thread, Ty Greg!!!!   

Also investigate the file location of each duplicate by clicking on the file location in the Organizer's Info panel on the right.

Hope this helps you to sort out the issue.

If you are each importing images into the respective catalogs from the same device or card, there will undoubtedly be duplicate files created

greg, can you say more about your statement above? 

Let me try to wrap my brain around this for a moment..  

The key issue here is about detecting duplicate images, a single catalog can do this for its imported images.  Then it will alert me if dups are present (I have not been using that option on the input screen!), I should uncheck those images (using icon at top) to keep them from being imported again.  I am not sure why dups should be occurring if I am using "delete origonals"...  Yes, if I cheat the process and reimport (cant use sd card with iphone) using USB on another pse2x catalog instance, and did not previously delete those imported images, then pse will not alert me to dups (and if I had heeded the warning! ugh), and unchecked, then yes I would be reimporting (to nas), those images....