Beginner workflow for scanned photos

[Dad25]

I scanned about 10,000 printed family photos over the course of the last few months, so they do not have any useful metadata. The photos were very disorganized, and I know that there were many duplicate prints. All files are on one external hard drive (with a backup copy on another one). To complete the work, I need to do some efficient clean up and add useful metadata.

When the project is completed, I will give a hard drive with a copy of the photos to each of my adult children. I want each photo to contain the following saved metadata, in the correct EXIF/IPTC/XMP fields (not just as keywords):

• Date the photo was taken (or at least a close approximation).
• Place names and GPS coordinates.
• People names, preferably in the “Person Shown in the Image” field.
• People tags (MWG face regions) if possible.
• Events.

I have no past experience with this and I would love to have your thoughts, feedback, advice or warnings on my planned “workflow.”

1. Duplicates. Find and delete visual duplicates (or near duplicates), using free software such as VisiPics.
2. Dates, Locations and Events. Create a new catalog in Daminion (free standalone). Assign dates, place names, GPS coordinates and events. Sync to the metadata of the files.
3. People names. Import all photos into a new catalog in PSE 2018 Organizer (since I already own it). Name and tag all of the people, using facial recognition. (Unfortunately, the actual tagging of the faces in their respective spots in the photo may not matter, since Organizer will not preserve the MWG face regions information in the metadata of the files.) Save the people names to the files’ metadata as keywords. (Unfortunately, this does not save the names in the IPTC extension field “Person Shown in the Image,” which I would prefer.)

In the above workflow, I have tried to use free (or already owned) software. I am not opposed to paying for software, such as Lightroom, if that would work more efficiently. However, I need the end result to have metadata saved in the files and not be dependent on a catalog that would force my family to purchase software in order to effectively read and use the photos.

Thanks!

[Tony_See]

In your instance (10,000 scans what an effort!!) I'd say keeping in simpler would be better.

Do EVERYTHING in Lightroom, but use plugins if you have to achieve other results not there by default in Lightroom.

|  KIS  | keep it simple

[Dad25]

Yes, I had no idea that we had that many photos! I would love to be able to do everything in one application. However, my very first step would be to eliminate dups and I understand that Lightroom does not have a way to automatically identify and bring together visually similar photos, so that is why I assume I would need to use VisiPics (or some similar program). Is there a plugin for LR that you would recommend that will do this?

[ItWasNotMe]

I've been through this with c30,000 images that I scanned. I second the suggestion to do it all in one application.

What I would do is also organise the photos into (approximate) 'date taken' folders, i.e. a limited number of photos in each folder.

I started with this, and also

• set the exif time and date in the photo as I went
• used a simple algorithm to set the time so any photos for a date would appear in the best guess of correct order when sorted on date/time.

Once I'd done that I worked through them chronologically doing the rest so could measure progress and be sure that I had done them all.

Once I'd "finished" a suitably large date range I wrote it to non-erasable media, e.g DVD/Blu-Ray depending on size of images.

Lightroom will allow you to write the metadata into the file (or XMP if the files are raw)

If you do Lightroom edits, they can also be written into the file, but either:

1. The recipient will need Lightroom to see these, or

2. You export copies that contain the edits and distribute those

When looking for visual duplicates you could first create 'stacks' in Lightroom and then choose the best from the stack

[Todd Shaner]

I agree with the suggestion to keep you're work inside LR. There are numerous LR plugins available as needed:

https://www.photographers-toolbox.com/

Any Comment Lightroom Plugin (and many more here)

Jeffrey Friedl's Blog » Jeffrey’s Lightroom Goodies (Plugins and Tools)

A good book on the subject. It says LR 5, but is applicable to the newer versions as well.

Organizing Your Photos with Lightroom 5 - The DAM Book

For finding duplicates here are two LR plugins with instructions on the Lightroom Queen's website, but they probably won't find all of the duplicates. When keywording and creating collections make sure to add specific information (place, people names, event, etc.) that can be used with the LR Text filter to help find the remaining duplicates. This is also useful for sharing the archive since many photo viewers have the ability to filter on keywords to find specific pictures.

https://www.lightroomqueen.com/clean-duplicate-photos/

Concerning sharing the finished photo archive you can use USB 3.1 flash drives. This one is available in capacities of 64GB to 256GB and is very small and economical.

https://www.amazon.com/SanDisk-128GB-Ultra-Flash-Drive/dp/B07855LJ99/ref=sr_1_fkmrnull_3?keywords=sa...

[ItWasNotMe]

I'm not sure that these plug-ins do what the OP is looking for, they seem largely metadata based matches; programme he is proposing matches the visuals.

I think he is trying to reduce the amount of work by eliminating the duplicates before he adds keywords etc.

[Dad25]

30,000? I have no right to be concerned about my measly 10k!

Yes, the reason I listed eliminating dups first is to reduce the work and to do some initial clean up. Also, I'm hoping that finding visually similar photos would allow automatically creating "stacks." Then, I could assign metadata (such as date taken, locations and events) to those stacks in batches.

[ItWasNotMe]

Its been a decade and more long project A real time bandit is when you given in to the temptation to "clean" them up.

I didn't have many duplicates until I started collecting images from relatives... so for me they were relatively easy to deal with.

I started sorting them out as I was scanning so a slightly different approach which was

0. Scan them into a standard folder

1. Create date based folder structure and move the image into that once I had loaded it into Lightroom, so ultimately the initial folder should be empty

2. Set the bare-bones metadata; date and approximate time on each as I scanned and imported it

3. Create some Task keywords; and set these on import

• Can be as simple as Develop or may be more detailed such as Straighten, Colour,  Crop... depends how you want to approach these tasks if you do them.
• I basically split into what I wanted to do in Lightroom and more heavy-weight stuff to do in Photoshop.
• As I've already said, if you take this route it can be really time consuming

4. Used many Smart Collections to create to-do lists, e.g

• Needs Developing  - keywords contain 'Develop'
• No GPS - Co-ordinates are not set
• No Title - Is blank
• and so on

Then I didn't need to do everything on an image at once, so if I got bored I could say do an hour setting GPS coordinates, if needed clear the task keyword otherwise completing the field would make them disappear from that to-do list, then having had enough of that move onto doing something else.

Other Points

A. Think about keywords and periodically review those that I had used and pull them back into some sort of standard

• Peoples names; they change over time, so did I use maiden name, married name, what if they divorced and married twice..., Formal Name or the one they use - Philip or Phil? There are loads of suggestions on how to do this. Eventually I decided to keyword using the name that I personally use most often, and then add all the others as Synonyms under that.

B. Decide which fields you really want to use, e.g. if you are using Sub location do you want to duplicate this in the Keywords? Has its benefits but is slightly more work.

C. Copyright notice or not - I decided to and then set up a Preset to add this to the images, so if its given any more I could just apply the preset

Also, as I might change an image several times, made sure I had a good backup strategy

[johnrellis]

I also recommend that you use LR as your primary digital asset manager. (I manage an extended-family collection of about 40K photos and growing, going back to 1950, and soon 1875.)  Some detailed feedback:

- Re PSE, I switched from PSE to LR in 2010, after several bad years of incredibly bad releases of the PSE Organizer: https://www.amazon.com/review/R332BXA5A70OL9 . While Adobe's support of LR, especially the Library, has flagged somewhat in recent years, and perhaps the support of Organizer has improved in recent years, it's still my strong impression that LR is significantly better supported than PSE Organizer.  While not perfect, LR's support of metadata has been much, much better than PSE.  LR does a reasonable job of following the Metadata Working Group standard, including using tagged regions for faces.    With just a little care, you'll be able to move your photos and their metadata from LR to other platforms.

- Every couple of years I poke my head up and look to see if there are better alternatives for digital-asset management (cloud- or desktop-based).  Over the years, advanced DAMs have come and gone, and biggies like Apple, Microsoft, and Google have fallen back to their core consumer markets and platforms.  To paraphrase Churchill, LR is the worst of all DAMs, except all the others.

- When thinking about metadata, only use fields and features of LR that get stored in industry standard fields in photos and sidecars. In particular, stay away from collections except as workflow tools (i.e. all photos that need GPS locations, photos still being deduped, etc.).  LR keywords are as capable or more capable as collections in nearly all respects: Lightroom Classic - help with setting up Collections for all my digital family photos | Photoshop Fa...

- You may want to include files of other formats in your catalog, e.g. genealogical documents, PDFs of family documents, etc. LR's product managers have mostly maintained their long-time party line that LR is targeted to the output of digital cameras and thus won't manage PDFs, Word documents, etc. My Any File plugin provides a reasonable method for importing such files into your catalog and managing their metadata.

- My Any Vision plugin uses Google's Cloud Vision to tag your photos with object and event labels ("mountain", "ski race", etc.), recognized text in the photos, GPS locations (sometimes), and dominant colors.  Think the Sensei feature of Lightroom CC (the cloud version) on steroids.  This can help you find photos with similar content and locations that would be missed by a program like VisiPics. It doesn't replace manual keywording, which remains essential, but instead supplements it and makes your photos easier to find. The plugin's Web page has pointers to other similar plugins and services that might be a better fit for your needs.

[johnrellis]

Also, if you're sharing the evolving catalog with family and friends, consider publishing the photos to a service like Flickr, Smugmug, or Zenfolio. This will let others contribute to the identification of people, places, events, and dates by adding comments to the photos, and LR will bring those comments back into your catalog. Unfortunately, LR provides almost no facilities for reviewing or searching such comments, so if you have a lot of them, take a look t the Any Comment plugin.

Flickr used to be my publishing platform of choice until their owner Yahoo's death spiral ruined it and they tried to be an Instagram wannabe.  "Flakr" has a horrible reputation now of unreliability.  I use Zenfolio, which isn't great either, but at least it lets people view the metadata attached to a photo without having to scroll!

[Todd Shaner]

ItWasNotMe  wrote

I'm not sure that these plug-ins do what the OP is looking for, they seem largely metadata based matches; programme he is proposing matches the visuals.

Agreed, which is why I said, "they probably won't find all of the duplicates." I'm working on a similar family photo archive project with about 5,000 image files so this piqued my interest. The image files are mixed flatbed photo scans, film scans, film capture using a DSLR camera copier, camera JPEG, and camera raw files. I just did a Google search on 'duplicate image finder' and ran across this review:

I found this thread with a review of Antidupl.net, Visipics, and Awesome Duplicate Photo Finder. None of these sound like a good solution for locating, reviewing and deleting selected duplicates in a large collection of 5,000+ image files.

https://sourceforge.net/p/antidupl/discussion/english/thread/e3675d12/

I downloaded another dedupe application called Image Comparer and ran a test on a folder with 1245 image files. It took only 3.0 min. to index the files and show like pairs. I have a large number of scanned handwritten, typed, and hand drawn sketch documents. Image Comparer found hundreds of "false positives" for these files, which makes it very difficult to parse from the real picture duplicates.  So not all that helpful!

Next I installed the LR plugin Deduplicator and ran it inside LR Classic 8.2 using the same 1245 image files. It took 6.5 min. to index the image files and found only 13 single image file duplicates. In the Duplicates collections folder it created only one image shows two actual duplicate files with the rest showing a single image file. So how do you locate the "matching" duplicate(s) if they have a different filename and date? Having recently worked with the 1245 image files in this folder I know there are many more duplicates and some have as many as four copies with slightly different edits.

Lacking camera EXIF data it appears none of these "duplicate finder" applications are up to the task of finding duplicate images based solely on analyzing the visual image data (i.e. perception hash algorithms).

Best suggestion I can provide is to bite-the-bullet and manually add verbose key wording to ALL of your image files. First use Face Recognition to speed adding name keywords. I ran it on a folder with 584 image files and it took 8 min. to complete. Once done with Face Recognition batch apply keywords starting at the highest category level and work down (date, family name, place, event, etc.). You can then use the LR Filter bar to search for specific subject types for duplicates, review them, and manually delete the unwanted copies. Keywords can be saved to the original image files using the CTRL/CMD + S keys and will be added to exported file copies.

[Dad25]

Todd, thank you for the links. I will explore for plug-ins that might help me. Unfortunately, as ItWasNotMe observed, both of those plug-ins mentioned by Lightroom Queen find dups based on metadata. What I need is a program that automatically identifies visually similar photos. My search, so far, has only turned up a plug-in called "Deduplicator," which says it is "based on perception hash algorithms." I do not know what that means, but perhaps I will try it.

[Dad25]

ItWasNotMe​ : I started sorting them out as I was scanning”

Unfortunately, I did not have that luxury. I suddenly had temporary access to a scanner and needed to do all of the scanning without being able to pre-sort or organize the photos. Currently, the only organization is that I put the physical photos into small boxes as they were scanned. The name of the digital folder just matches the label on the box. So now, if I needed to find a particular photo, I would “only” have to search through about 500 in a given box.

@Todd Shaner :

Thank you for sharing test results of those various duplicate finders. It sounds like Antidupl.net, Image Comparer, and Deduplicator will not do the work I am hoping for. So, I have downloaded and tried VisiPics. For my small sample (268 photos) it seemed to work pretty well. It took less than 2 minutes to find 14 duplicate groups with a total of 44 pics. That was using the “Loose” setting, so there were many false positives (non-identical photos). It is too bad that kind of technology is not built into LR. It would be very helpful in creating stacks of visually similar photos. When I put the setting to “Basic” it created 5 groups, with 2 identical photos in each group. Then I could automatically select the ones to delete or move. I think this will probably work as a way to pre-clean my collection before importing it into LR.

Eventually, once I have added metadata, perhaps I will want to use a LR plug-in to help find and manage possible duplicates from within LR. Victoria Bampton, in her blog (https://www.lightroomqueen.com/clean-duplicate-photos/) said: “The downside of an external duplicate detection app is it doesn’t know which of the duplicates you’ve edited in Lightroom, so you can end up deleting the wrong ones and creating a long job restoring broken links.”

@ johnrellis :

“LR does a reasonable job of following the MWG standard, including using tagged regions for faces. With just a little care, you'll be able to move your photos and their metadata from LR to other platforms.”

That is excellent news! So, if I correctly understand, you are saying that when I run face recognition and place tags on the faces of all known people in my photos, LR will save the location of those tags in the MWG face regions of the files? Do you happen to know whether or not the people names will be saved to the IPTC “Person Shown in the Image” field (and not just as keywords)?

[johnrellis]

So, if I correctly understand, you are saying that when I run face recognition and place tags on the faces of all known people in my photos, LR will save the location of those tags in the MWG face regions of the files?

Right.

whether or not the people names will be saved to the IPTC “Person Shown in the Image” field (and not just as keywords)?

LR stores the person names in keywords and in the XMP:PersonInImage field, which is where the IPTC Person Shown In The Image is stored in XMP metadata.  However, the names only show up in XMP:PersonInImage field in exported photos -- when you do Metadata > Save Metadata To File to write metadata changes back to the originals, the names only get stored in keywords, not in XMP:PersonInImage.  A senior Adobe engineer obstinately refused to acknowledge that as a bug: Lightroom: Person keyword only fills "people shown" IPTC Extended tag on export | Photoshop Family C...

[Todd Shaner]

Dad25  wrote

ItWasNotMe  : I started sorting them out as I was scanning”

Eventually, once I have added metadata, perhaps I will want to use a LR plug-in to help find and manage possible duplicates from within LR.

Give the Deduplicator LR Plugin a try. It's easy to use and creates a collection named 'Duplicates' with the found duplicate image files.

[Test Screen Name]

You don't mention it, so I will. I planning this project, think hard about your backup strategy. EXPECT the computer, and any hard disk to fail some time in the project, and plan so you lose no more than a way's work. Test your plan! This is annoying, time consuming stuff, but losing everything is worse.

[Dad25]

Thank you all for your time and helpful comments. Here is a summary of my conclusions and amended workflow. How does this look?

COMPLETED:

Duplicates. Find and delete visual duplicates.

The helpful discussion in this forum persuaded me that I would not find any effective way to achieve this from within LR (with or without a plug-in). Therefore, I moved forward, using the free VisiPics software. The results were great! The software took about an hour to process, plus I spent a couple hours selecting and deleting the duplicates. I successfully deleted 800 files. It is really a shame that LR does not have this technology. It would be very helpful, not only for removing duplicates, but for helping to show groups of similar photos for tagging, culling, etc.

PLANNED:

Import. Import all my photos into a LR catalog.

The above discussions had good advice about setting keywords on import, using Smart Collections, etc. However, the photos currently have no identifying data, so these suggestions will be more applicable once they are in LR.

People names. Name and tag all of the people, using facial recognition.

I have read countless online discussions about naming conventions, but still feel largely undecided. For now, I will just use “First Last” as of the time of the photo. I will add “(Maiden)” or some other term to distinguish those with the same name. Initially, I do not plan on using keyword hierarchies.

Dates, Locations and Events.

I don’t know exactly how to go about recording these pieces of information efficiently. Since LR cannot automatically “look” at and group the photos for me, I suppose I will just have to manually look at them in the image display area and dive in. I need to assign the date it was taken, place name, GPS coordinates and events. This will take many months.

Backups. After each significant work session, I will back up to a separate drive.

Another question:

Apparently, once I am completely done, and ready to create the collection to give to my family, I will need to “export” all the photos in order to have the people names saved in the XMP:PersonInImage field of the files. However, when should I do the Metadata > Save Metadata To File function? Should I do it frequently as I go, or wait until I have all of the above completed and then do it all at once?

[johnrellis]

Apparently, once I am completely done, and ready to create the collection to give to my family, I will need to “export” all the photos in order to have the people names saved in the XMP:PersonInImage field of the files. However, when should I do the Metadata > Save Metadata To File function? Should I do it frequently as I go, or wait until I have all of the above completed and then do it all at once?

For the purposes of providing photos to others, Export will correctly write the metadata into the exported photos.  Metadata > Save Metatadata To File isn't necessary for that.

However, Metadata > Save Metadata To File does have other uses:

1. If you want to use an external tool to manipulate the metadata, you need to run Save Metadata To File first so that the master photos have the correct metadata written from the catalog available for the external tool to read.

2. As a belt-and-suspenders, last-ditch backup mechanism. Hopefully, you've got a regular backup mechanism in place (e.g. Mac Time Machine), and depending on your backup program, you're having LR make regular catalog backups when it exits.  But backup regimes are notoriously prone to failure (due to human error and lack of testing), so some of us ensure that metadata is also always written to the photo files, providing a last-ditch backup in case the primary catalog backups don't work.  Rather than doing manual Metadata > Save Metadata To File commands, you can set the option Catalog Settings > Automatically Write Changes To XMP.  Years ago that used to cause performance issues for some people, but such reports are rare these days. 

[Todd Shaner]

Dad25  wrote

Dates, Locations and Events.

I don’t know exactly how to go about recording these pieces of information efficiently. Since LR cannot automatically “look” at and group the photos for me, I suppose I will just have to manually look at them in the image display area and dive in. I need to assign the date it was taken, place name, GPS coordinates and events. This will take many months.

With 10,000 family pictures and NO metadata currently assigned it's going to be a lengthy task.What you can do is create collections of specific photos, sync them to the Web and then create share links, which you can send to other family members. Currently they won't be able to add keywords, but they can add "comments" which will appear inside LR Classic and LR CC. You can then copy & paste words from the comment panel to the the Keyword Tags panel as shown below.

Sync Lightroom Classic with Lightroom CC Ecosystem |

With someone who is technically knowledgeable you could give them your lightroom.adobe.com login name and password. They can then add actual keywords inside the LR CC Web app that will appear automatically as keywords inside LR Classic, which saves you time. Just remember they can also delete, rate, and edit pictures, which you may or may not want them to do.

Get to know Lightroom CC on the web |

[johnrellis]

They can then add actual keywords inside the LR CC Web app that will appear automatically as keywords inside LR Classic

Unfortunately, in general keywords don't sync between LR Classic and LR CC. (In limited circumstances they sync from LR Classic to LR CC when a photo is first synced.)  It's one of the more popular feature requests:

Lightroom: Keywords don't sync between CC and Classic | Photoshop Family Customer Community

Unfortunately, Adobe has clearly indicated that no new functionality will be added to Classic - CC sync. But if enough people vote on that feature request, it's conceivable they'll change their mind. I received one of those Adobe surveys last month that asked several questions about Classic - CC sync.

Adobe has also stopped development over the past 18 months on cloud APIs to the Creative Cloud, and keywords were inexplicably omitted from the existing APIs.  If they just added keywords to the APIs (other metadata is already available), then plugins could provide the sync functionality.  Sigh.

The lack of keyword sync is one of several reasons why I use third-party services for sharing my photos.

[Todd Shaner]

johnrellis  wrote

They can then add actual keywords inside the LR CC Web app that will appear automatically as keywords inside LR Classic

Unfortunately, in general keywords don't sync between LR Classic and LR CC. (In limited circumstances they sync from LR Classic to LR CC when a photo is first synced.)  It's one of the more popular feature requests:

Lightroom: Keywords don't sync between CC and Classic | Photoshop Family Customer Community

Sheeeesh–I knew that, but guess I wanted to forget about it!

john beardsworth suggested using his Syncomatic plugin for this purpose. I guess it would work something like this:

1) From inside LR Classic create a collection and sync the image files to LR CC.

2) Add keywords to the image files from inside LR CC Desktop, Web, or Mobile apps.

3) From inside LR CC Desktop 'Save' the files as JPEGs into the same folder as the original files. Not sure how this would work if the original file is already a JPEG so you will need to test it on some original file copies so you don't corrupt or lose your original JPEG files.

4) Use the Syncomatic plugin in Lightroom Classic to sync the keywords from these JPEG copies to the originals. You can then delete the JPEG copies.

It sounds complicated, but will save you a lot of time if it works!