Tools and tips for tidying messy file and catalog in PSE2020 on W10

[Billy5FE5]

I have used PSE for many years and am currently using PSE 2020 on a Windows 10 PC.

During migrations and in configuring OneDrive I have made a number of mistakes in how I have organised files and the PSE catalog and I am losing track of where files are stored - I haven't lost any files, yet, but I have come close to it and need to tidy things up before I do.

Currently, everything is in one catalog which has almost 24,000 items, including many duplicates. The files are spread over an inconsistent file structure which is spread over my C: drive, the old drive from a previous version of Windows (E:) and in the cloud in OneDrive. (This situation is too confusing to explain here, but it might help to give you and idea about how confused I am. The interaction with OneDrive is particularly confusing to me just now and this is where I am most concerned about messing up so badly I lose data. I have had to restore from local backup once. The restore got all the data back, as far as I can tell, but it further confused the folder structure. The restored catalog was relatively clean, but I think the old folders were in the PSE watchlist and the old folder structure has reappeared in the catalog with content being downloaded from OneDrive on demand. )

I think that the best thing for me to do is to grit my teeth and painstakingly go through all the folders that Organiser shows as having images in them, understand where they are and whether they are duplicates or not and create a new, clean catalog on a better planned folder structure by hand - by moving the files to a new, consistent location and importing them to a new catalog. (Then I can think about deleting the old files, but that will be risky.)

I want to be able to use the new structure from my PC and a laptop, so my current thinking is to move the files to new folders in OneDrive and create a new master catalog on the PC. I would then create a seperate catalog on the laptop which would watch the OneDrive folder for changes and keep in synch that way.

I expect this will be a very long and tedious and anxious process so and efficient and reliable process is key. Before I start, does anyone have any advice about tools, tips and techniques for manually cleaning up a messy catalog? For example, tools that would let me compare folders and highlight differences? Anything in PSE that reports on duplicate files, broken links etc? Is there anything in PSE that would restructure files, e.g. create a new catalog based on a consistent folder structure based on the date shot or which would copy files over to a consistant structure?  

[Ged_Traynor]

Moved to the Photoshop Elements forum from Photoshop ecosystem

[MichelBParis]

Hi @Billy5FE5 ,

There are a number of things to discuss in your situation. Before that, I would start immediately with two suggestions:

1 - Create a new organizer full backup on a dedicated external USB drive to save the current sate of your catalog and managed photo/video files. Don't work on deleting or organizing at that stage or playing with OneDrive.

2 - If I understand your project to share the final catalog and library between two computers, my strong advice is to choose to store both on an external hard drive (I have been doing this for years). That means restoring the above suggested backup on a new small SSD external drive. You first create the new master folder as the 'custom destination' to hold your photo folders tree as well as the restored catalog folder. You don't lose anything of your current "organization", you have a backup and you can even access the new catalog from two computers having the same PSE version by just plugging the HD alternatively on both. Use the option to keep the folder structure.

Already a few tips:

- when you create such a full backup what happens with the original folders structure?

OneDrive: the files synced with OneDrive are on a specific OneDrive folder, they will be backed up and restored with the same path as before, but that won't the real syncing system drive, just a 'normal' folder.

Different drives: in the restore process on a single drive, the files of a separate drive will be restored under a master folder named after the source drive.

Duplicates:  Normally, the organizer should not contain duplicates in the catalog. In the organizer definition, they are totally similar in content and format, and are skipped at import or backup time. However, you can have visual duplicates, or files with the same name, bu not having the same date_taken AND file size in kilobytes. The backup you are doing scans the photos in your catalog, not on your drives. Duplicates on the drives are not backed up.

At that stage, we can start to discuss the cleaning and reshaping process of the catalog itself. Indeed, that may be a big task if we focus mainly in achieving a clean and meaningful folders structure. However, if we rather want to make the 'virtual' tag and albums structure better, allowing better searching and grouping of your photos, that should be easier and les dangerous.

What do you think about it?

[Billy5FE5]

Wow, two full, fast replies. Thanks to both.

@MichelBParis 1) Absolutely, I won't do anything risky without a full backup to an external hard drive. Fortunately (or sensibly) I had taken a full backup before my last mistake and was able to recover because of it, so the lesson has been learned.

2) I have read that the best way to access the contents of Organiser from two devices is to create a copy on an external drive and physically swap it over, so that probably is going to be the best way. I had hoped that if I used OneDrive I could find a better way of working that would:

a) give me a fully automatic, real-time backup process (I am lazy and need the backup to be as automatic as possible or I'lll start to skip it) 

b) give access to images from other devices (like my phone) from anywhere. This universal access isn't important, though, it's just a nice to have.

I have to confess that I don't understand your advice about the backup and duplicates. 

On the duplicates and not normally having them in the catalog, what I see on my catalog doesn't seem to be what you expect the catalog to be like. I have catalog entries that have the same name, size, date and time but point to seperate folders. For example, one entry points to E:\Users\BillyandJean\Pictures2020\2014 09 26\2014-04-19_14-17-03_999 and the other to C:\Users\billy\OneDrive\Pictures\2014 04 19\2014-04-19_14-17-03_999. When I look at them in file explorer, the files contain similar images and have the same size, date and time. The first is my original file structure but I honestly can't explain how the other one was created - it was probably part of my confused attempts to implement OneDrive.

If it is feasible, I would rather put the effort in to cleaning up the underlying folder structures rather than trying to solve the problem by tagging. I get it that tagging is safer and may take less effort, but this is just part of a general attempt to get the PC in order and I'd like to get this sorted at the underlying level. The question is whether or not it is feasible and worth the risk.

[MichelBParis]

2) I have read that the best way to access the contents of Organiser from two devices is to create a copy on an external drive and physically swap it over, so that probably is going to be the best way.

By @Billy5FE5

Doing a full backup and restore on a new external destination drive is the best solution. One big advantage is that it's the only 100% sure way to check if the transfer is ok. Note that in your case, you still have both the original catalog + data unchanged, plus a proven backup. You can start your re-organization job from the original computer disk or the external one, while keeping access to the previous situation.

I had hoped that if I used OneDrive I could find a better way of working that would:

a) give me a fully automatic, real-time backup process (I am lazy and need the backup to be as automatic as possible or I'lll start to skip it) 

b) give access to images from other devices (like my phone) from anywhere. This universal access isn't important, though, it's just a nice to have.

By @Billy5FE5

OneDrive is indeed a good solution to backup your files, but not the relationship between the catalog and the media files. I like it to share some kinds of documents and a selection of photos with different devices when away from my computer.

Your experience with duplicates is not easy to troubleshoot. The 'filtering' of duplicates when importing (or backing up) works very well in most cases. For instance, it's the same when you use the 'watched folder' services or when you manually import from one of the watched folders. Same scanning and filtering the files in the catalog. However, the filtering algorithm has been said by Adobe to be more complex than the two criteria mentioned above: date_taken and file size. However I have never seen it described in detail. So, there are still situations when the filtering does not work. Fortunately, they are not frequent. Recently, that happened with importing from smartphones or cameras which stored duplicates for some reason. That happened with files imported from cloned disks; probaly also from a mix of original files with restored ones... Good luck to find the reason unless you have lots of duplicates for the same reason.

[Billy5FE5]

@MichelBParis, thanks for your advice.

I have restored from (what I think is) a good full backup to a new catalogue on a new external drive. From checking a random selection of images and folders, it looks OK and I can use the new catalogue on both my laptop and desktop by physically moving the drive. So, (I hope) that I am in the good position of having a good catalog on one external drive and backups on another exernal drive. I will keep on taking full backups before doing anything risky, like moving or deleting files and folders.

That's a good first step, but I would like to positively confirm that every image on the PC is actually in the catalogue (or, rather, be able to account for any ommissions as I expect there are image files on the PC that I wouldn't have wanted to import to PSE).

So, I think that the next step, before starting to tidy up and delete the old stuff, is to identify all the images on the PC and reconcile that list against the newcatalog.

Can you recommend tools, techniques or processes for doing this? Or do you maybe think that it isn't feasible?

[MichelBParis]

@MichelBParis, thanks for your advice.

I have restored from (what I think is) a good full backup to a new catalogue on a new external drive. From checking a random selection of images and folders, it looks OK and I can use the new catalogue on both my laptop and desktop by physically moving the drive. So, (I hope) that I am in the good position of having a good catalog on one external drive and backups on another exernal drive. I will keep on taking full backups before doing anything risky, like moving or deleting files and folders.

That's a good first step, but I would like to positively confirm that every image on the PC is actually in the catalogue (or, rather, be able to account for any ommissions as I expect there are image files on the PC that I wouldn't have wanted to import to PSE).

By @Billy5FE5

There are indeed good reasons NOT to have every image on the PC in your catalog.

If you want to test which photo/video file on your PC is not in the catalog, you simply do a 'bulk' import of everything instead of importing selectively from folder branches. I don't recommend testing the 'bulk' option, because you'll get tons of images you don't want in the catalog. Those imported images will be easily selected afterwards if you choose to select by 'import batch' sort order. However, deleting thousands of unwanted files is very slow in the organizer. The good side of that bulk import is that it will skip duplicates and unsupported file formats. If you import selectively from chosen drives and master folders, that will work the same as if you put them in the 'watched folders' list.

Theoretically, [= if the filtering of duplicates at import time has worked] you can:

- 1 selectively import from various disks or master folders

- 2 select by import batch

- 3 delete unwanted files and optimize the catalog

- 4 backup and export on a new drive or location

- 5 Finally use the explorer to delete the original moved folders and the original catalog, while keeping the backup folder.

[Billy5FE5]

I put the external drive on the PC which has the files to be cleaned up. The good catalog is in G:/PSE  and the old folders are in E:\Users\BillyandJean\Pictures2020\Users\BillyandJean\Pictures2020. To keep things short I'll just call these the G: and E: folders. The first sub-folder on G: is /100OLYMP and has 97 items. The same folder in E: also has 97 items. So far, so good.

I did a "Get files and folders" from the E: sub folder /100OLYMP, hoping that it would skip 97 items and import nothing, but it imported 31 and skipped 66, leaving 31 apparently duplicate entries in the catalog.

Easy enough to tidy up, but I cannot see why some files are seen as duplicates and some aren't.

These are the file explorer properties of files that Organizer imported because it couldn't see that file already exists. 

The "size on disk" and "date created" are different, but files that were correctly skipped show similar differences and so I don't think these fields are the the cause.

The file info in Organizer looks like this - again, I don't see a significant difference between the E: and G: versions. 

If the import function isn't identifying duplicates reliably, this approach doesn't seem feasible. Any idea about how to figure out why some files are skipped and others imported? 

[MichelBParis]

@Billy5FE5 

I have not found the reason of the differences.

I'll take more time to think about it.

In the meantime, here is the best note about the various dates and times in the organizer in John R Ellis's faqs;

https://johnrellis.com/psedbtool/photoshop-elements-faq.htm#_All_the_different

[Billy5FE5]

@MichelBParis Thanks - if you think of anything, I'll be glad to hear from you.

I think I'll post another question on this forum specifically about how PSE identifies duplicates.

In the meantime, my next line of investigation will be from the FAQ you referred to, which says "If you’re using unknown month, day, or time values in the Edit > Adjust Date And Time command, you’ll likely run into problems." I have a lot of media files that I created by scanning old photos - I would have editted the date and time information and may have set some fields to "unknown". That doesn't immediately look likely for the specific files I looked at in the case above, but it is worth looking into - it might avoid new problems in future.

The FAQ also recommends Exiftool so I'll look at that and see if it will help me analyse the situation.

Thanks again.

Onwards!

[Greg_S.]

@Billy5FE5, Michel has given you sound advice. 

I am just curious whether you use keyword tags, face recognition and other Organizer features such as Albums.  It may affect what approach to take in the future.

And while your main concern seems to be where your files are stored on your computer, the Organizer doesn't really care where they are stored unless, as in your case, you intend to use the same catalog on a desktop and laptop. Michel is an expert in that workflow.

When you get around to comparing different copies of folders, there are a number of utilities that can help you.

[Billy5FE5]

@Greg_S. Thanks for your interest in this.

I use tags a lot. Most of my images are tagged one way or another and I often rely on them to locate images. Facial recognition, yes I use that too, but I don't rely on it and I try to tag images by hand as I import them. Albums, I don't use much at all.

I take the point that Organiser doesn't care where files are stored. I am starting to understand that I may have been taking the wrong approach with the backup. I thought it would be best to use OneDrive for backup for everything I use (apart from PSE, I have Office 365 files, mostly, with an old iTunes library to complicate things) and wanted to include my photo library in the same approach.

Maybe, instead of having one approach to backup, I should rely on Organiser to backup the photo library and exclude the photos from the PC backup regime.

Above I said to @MichelBParis that I wanted to work on the folder structure rather than using tags and albums, but maybe using tags and albums is the best and safest first step on the way to fixing this mess. (Once it is sorted at that level, I could look to understand the underlying folders.)