NEED HELP TO RESTORE A PROBLEM CATALOG

[josephunit2n]

I keep active catalogs on an external drive.  I keep catalog backups on another external drive.  One catalog's structure I do not like, want to correct.  First, it appears in both "Custom Location" AND "All Users".  If I open that catalog name in custom location, many photo thumbnails have no pictures.  If I open the exact same name in all users, ALL photos appear, and are connected to source files - only -- some are stored on that desired external drive, even more are stored on the computer's C drive, and I do not want this, want to consolidate ALL onto the external drive.  SO:  I opened the all users catalog, confirmed all photos connected to source files (in either one, or the other location, as mentioned) - and ran a full backup, to the external "backup" drive (successfully - no messages about need to reconnect photos)  - THEN I restored to the other external drive (for active catalogs), but to a new folder/new location.  Backup ran successfully -- BUT -- when opened (either in Organizer, or directly from the "catalog.pse21db" file) the catalog has MANY thumbnails without photos (like the catalog I mentioned in "custom location").  NOW, I looked, and in the Adobe location on C drive, found that catalog - opened it from the "db" file, and the correct catalog DID open, with all photos linked.

WHAT IS WRONG WITH MY METHOD?  HOW CAN I GET THE BACKUP FROM THE COMPLETE CATALOG, STORED ON C DRIVE?  ---  OR --- is there a better way to achieve my purpose, to get it all stored only on the proper external drive?  

I'm very worried, don't want the catalog to become corrupted, photos lost - PLEASE HELP

[Greg_S.]

Please give us a screenshot of what you mean when you say photo thumbnails have no pictures.

How are you naming the catalogs?  Is it possible that what you think is the same catalog on different drives, is in fact two different catalogs with the same name?

And why do you have more than one active catalog?

[josephunit2n]

"photo thumbnails have no pictures" means grey squares, with question mark in the corner;  I agree, confusing!  Same catalog name, same photos - but the catalog.pse21db file is in 2 places - in a folder of that catalog name, on the proper external drive (when opened, those many grey thumbnails) - AND in the same catalog name on the computer C drive, in the Adobe location (when opened, ALL photos - same photos - are properly linked to source files, but some in folders on that external drive, some source files on the C drive).  Anyway, I thought the backup I was taking was of the catalog with all pictures (it was displayed in Organizer, when I initiated backup - no "must reconnect missing files" message appeared), but when the backup was restored/opened, it was the instance with the many missing source files - the files on C drive, only those with source files on the external drive were properly linked.  WHY is this happening?  HOW can I get a backup of the catalog in the C drive location, which has all thumbnails correctly linked to source photo files?  Or, if this is for some reason a useless concept, how can I get that catalog on the C drive, with all linked photos, some on the external drive, some in that C drive location - to be ALL on the external drive?  If you show me how, next question will be - can I safely delete that same, but incomplete, catalog that resides in another folder on that external drive?  How best to do that?  Manual deletion (I don't like, because the source photo files are in a number of "date folders" and I'd worry my deletion might not be complete)?  Can this all be deleted (after the proper catalog is on that drive, in another location) using "remove" in the Organizer?  This is getting ahead!  First, let's get the intact catalog completely stored in a new location on the external drive.

[josephunit2n]

How am I naming it?  It is:  "Joe's Images 12_15_2020", same naming format I use for other catalogs.

[josephunit2n]

Waiting for your direction.

[Greg_S.]

Hang in there. You have provided a lot of information that we need to wade through. 

[MichelBParis]

Hi@josephunit2n 

I am trying to find what is wrong with your issue. It seems to me that there is a misconception somewhere about catalog managements. I would start with asking what you mean by "One catalog's structure I do not like, want to correct. "

Your general reasoning seems ok and it may help to better understand what you mean by "structure". Changing the location of files and file folder trees on different drives or master folders?

[josephunit2n]

I mean:  worried about catalog's stability, integrity, going forward - with source photo files some on C drive, some on intended (for active catalogs) external drive; PLUS, I've never seen, can't understand and do not like - that the same catalog "catalog.pse21db" is found on BOTH external catalog drive AND in same name catalog folder on C drive in the Adobe location (if I open that "catalog.pse21db" on the external drive, the catalog opens with many thumbnails not connected to files - I believe these are the ones stored on C drive;  if I open the same "catalog.pse21db" found in the folder of same catalog name, in the Adobe C drive default location, ALL photos appear, all thumbnails populated, and my random scan shows - as said - some source photo files are on that external drive, some on the C drive).

My reference to "structure" is this:  I want to have the catalog and all files stored ONLY on the external drive - I imagine in a new folder/location - and next, I would like to delete/eliminate/remove the other "catalog.pse21db" found on that drive, completely, including all associated files.  Finally, I would prefer (last step) to also delete this and other catalogs stored in the Adobe default location on C drive (unless you advise these are necessary, for some reason - I wonder - when the Organizer gives a pop-up message "time to backup catalog" - and I click "ok" - does it create those catalogs on C drive?  -- if so, and if I may safely delete them once this issue is sloved - I will never again click "ok" to that  backup request, instead make my own backups to another external drive designated for that purpose).  Among many reasons, one reason I want active catalogs and all associated files on the designated external drive, not C drive:  the external drive is RAID 1, dual mirrored drives for security (of course C drive is not, therefore not long term secure).  Also, C drive is solid state, not as well suited for mass file activity (photos) as is the external, mechanical hard drive.

[josephunit2n]

Nice to hear you Michel!!

[MichelBParis]

I mean:  worried about catalog's stability, integrity, going forward - with source photo files some on C drive, some on intended (for active catalogs) external drive; PLUS, I've never seen, can't understand and do not like - that the same catalog "catalog.pse21db" is found on BOTH external catalog drive AND in same name catalog folder on C drive in the Adobe location (if I open that "catalog.pse21db" on the external drive, the catalog opens with many thumbnails not connected to files - I believe these are the ones stored on C drive;  if I open the same "catalog.pse21db" found in the folder of same catalog name, in the Adobe C drive default location, ALL photos appear, all thumbnails populated, and my random scan shows - as said - some source photo files are on that external drive, some on the C drive).

By @josephunit2n

Hi @josephunit2n 

Le us start with with a simple fact:  the file 'catalog.pse21db' is not a catalog. It's a a sqlite database file, the main component of a catalog. Catalogs are folders. The name of the catalog is the name of the folder. The 'catalog.pse21db' is the main sqlite database in the folder; there is another one:  thumbs5.cache which is used to store the thumbnails, as well as many other components working together the main database. the "pse21db" extension means the internal format of the database is built for the 21 version of PSE, called PSE 2023. That explains why double clicking that catalog.pse21db opens the organizer with the right flavour of the sqlite database structure and the correct organizer version. That also explains that any catalog database file of the same PSE version is always called the same. Going on further, it's a universal fact that if you have two identical copies of the same catalog, whether on paper or numerical form, they all link to the same data. Two or three copies of the same catalog of the Louvre museum will always indicate where is Mona Lisa situated even if they are stored in different houses. You can always copy or move catalog folders, their links stay identical, even if they are in differeint drives. That's a good thing, but it explains that you can have several unwanted catalog folders on various drives or folders with the same name. Of course, unless you want catalog folder backups, it is generall redundant and misleading.

To answer your other question, you may delete unwanted folders directly from the explorer or from the catalog manager.

Before going further, I'd like to know if that makes things clearer now ?

[josephunit2n]

Somewhat;  I understand it's a database file - for brevity, I called it "catalog".  From the proper catalog folder (name:  Joe's Images 12_15_2020), IN THE EXTERNAL DRIVE, if I open from the database file - I get the catalog in Organizer, with only photos connected that have been more recently stored on that drive - all others (stored on C drive, I believe) are not connected;  If I go to that same catalog folder/name on C drive in Adobe location, again open from database file found there - same catalog opens in Organizer, but with all photos linked to souce files  - some stored on that external drive in same date/folders as in the first instance, the rest stored in that Adobe location on C drive.

Not sure it matters, but what I suspect happened:  I had the catalog on the external drive - in Organizer it appeared in "Custom Location" and I might have used "move" to put it into "all users" - recently, I discovered this doesn't just move the catalog name to "all users" (as I had presumed), rather, the catalog was moved to the Adobe location on C drive!  Then, later, when I added more recent photos, I likely "pointed" their storage to the external drive.  Regardless, I want to consolidate ALL to the external drive, don't understand why my method failed (I opened that catalog in Organizer, "all users" which displayed catalog with ALL photos connected, ran backup to another external drive [all photos confirmed, since no "must reconnect photos" at the start of backup], then did successful "restore" to a new folder/location on the external drive for active catalogs - however - when it automatically opened [same result, if I open from database file at that restored location], the catalog displayed just like the instance I have described, from "Custom Location" in Organizer - that is, ONLY photos recently stored in external drive location, displayed - all others, those stored in Adobe location/C drive - were NOT linked!).

**  I don't know why this is the result (could the source photo files on C drive be somehow copy protected?).

**  If the answer is not known, is there a better/another way to accomplish my purpose?

[josephunit2n]

Hmmm - a thought:  If my assumptions are even correct - could/should I attempt to "reverse" that "move" process?  That is:  first delete all traces from the external catalog drive (or - not, just point to a new folder/location), then use "move" to direc the catalog in Adobe "C" back to that new location on the external drive.  Would this resolve it all, or create only more storage complications, possible corruption?

[josephunit2n]

First, I do await your guidance.

All these years, I've had a love/hate relationship with P.E, enjoying working within the program, fearing failure from the storage/cataloging system.  I really must learn what went wrong in this instance.  I keep full catalog backups on an external hard drive, always feeling secure that these would restore all photos and catalogs, in the event of catastrophic failure.  Now, after a "perfect" backup (organizer displaying all photos linked to source files, no "must reconnect missing files" message, "backup complete" message at completion), upon Restore to another external drive, the restored catalog is missing a majority of the photos.  I am completely shaken!!!!!  Are all my backups, that I've placed great faith in, actually useless??  PLEASE help get to the bottom of this, help me restore faith in the security of my wife's and my many photos.

[Greg_S.]

First, it is highly unlikely that your working catalogs have suddenly become corrupt, but it is possible that you have created a corrupt backup.  It is even more unlikely that the Organizer will lose any of your precious photos.  Those are all stored on your hard drives (perhaps on different drives and perhaps in duplicate, either with or without backup copies).  The only time a photo is deleted from your hard drive is if you gave specific instructions to delete the file, either while deleting files from a catalog or while working in Windows File Explorer (which should never be done).

Based on the limited information you have given us, my guess is that you in fact have multiple catalogs with the same name (on different drives).  And once you started using multiple catalogs, it looks like you were opening them at different times and performing different actions with them.  For, example, you import some new files into Catalog 1.  You then open Catalog 2 (which has the same catalog name) and the newly imported files are not found there.  Coupled with other actions you may have taken, it appears that you have a number of missing files in the catalog.  (A missing file is one with the yellow question mark icon that you described earlier as a photo without thumbnail.)

You still have not provided us with sufficient information to help you.  The devil is in the details.  So let's start with some more basic questions (and it would be helpful if you gave us some screenshots):

1. If you open the Catalog Manager, is your current catalog identified under the All Users radio button or under a Custom Location?
2. If under a custom location, what is the full path of the custom location shown? In my screenshot below, my custom location is my L drive (and because I don't have any catalogs on the root of the drive, there are no catalogs shown)

You should also open each catalog you can find and confirm where the catalog folder is located.  You can do this by selecting System Info . . . from the Help menu:

BTW, the default location for your catalogs is C:\ProgramData\Adobe\Elements Organizer\Catalogs and the automatic backups made by Elements are found in the _AutoBackup_ sub-folder.

3.  Next we need to know where all of your media files (photos and videos) are stored, and what steps you have taken to move any files around outside of the Organizer.  I am not asking about copies of the photos that may have been made as part of a backup made via Elements.  Those backup photos will usually have a name that has a B000 prefix.   It is my understanding that you have media files on one or more external drives and also on your C drive, and that your goal is to place all of your media files on your external drive.  To make sure you know where the files in each catalog are stored, make sure you use the Folder panel in Tree View.  This will show the exact same hierarchy as Windows File Explorer. If you only see drive icons, right click on each drive and select Show All SubFolders from the context menu:

If all you see is a flat list of folders in alpha-numeric order, you are in the List View.  You can change to the tree view by clicking on the hamburger icon next to the My Folders heading at the top of the panel:

4.   For further confirmation that you have more than one catalog with the same name but with different contents, tell us how many media files are displayed in the media grid when nothing is selected.  (There may be some minor differences depending on whether you have any stacked images or version sets, but this will give us a better idea of the scope of your problem.)  The number of media files is found at the lower left of the Organizer window:

5.  How many external drives are you using?  Are they all always connected to your computer? 

6.  How many backups have you made?  What is your backup practice?  Do you delete old backups after making new ones?  Do you make full backups or incremental ones?  Do you use any other software to backup your computer files?

I shall await your responses before making any specific suggestions for you.  But you may want to take a look at the discussion in this thread.  It is a little difficult to follow because of some translation issues, but I suspect that the problem addressed is similar to yours.

[josephunit2n]

Greg, answering your questions:

• In Catalog Manager, the EXACT SAME catalog name appears under All Users, AND Custom Location
• Custom Location, full path:  P:\Photoshop Elements\Catalogs\Restored_Active Now 11_10_20\Joe’s Images 12_15_2020
• All Users location, full path:  C:\Program Data\Adobe\Elements Organizer\Catalogs\Joe’s Images 12_15_2020   [note:  name of this catalog is ALWAYS Joe’s Images 12_15_2020]

WHERE MEDIA FILES ARE STORED:                                                                                                                                            In the catalog opened in Custom Location (310 photos), the most recent photos (appearing in thumbnails) link to photo files stored in that same path, with a date folder at the end (P:\Photoshop Elements\Catalogs\Restored_Active Now 11_10_20\2023 xx xx\ )  where x’s are the date of a given photo.  Many more photos, older than these, are only grey thumbnails w/ question mark, no source photo file.  My belief:  the more recently added ones are here because I became aware and correctly “pointed” them to P drive storage, this catalog, when they were uploaded.

                In the catalog opened in All Users (346 photos, due to some recently added only to this location),  THE SAME RECENT PHOTOS as in Custom Location are also stored in the same P drive location, BUT all the others, those grey/question mark in Custom Location, now are properly populated thumbnails, source files in:  C:\Program Data\Adobe\Elements Organizer\Catalogs\Joe’s Images 12_15_2020\2022 xx xx\  (where the end, 20xx xx  xx is the date of the specific photo).

•  How many external drives are you using?  Are they all always connected to your computer?  I use one external drive for all active catalogs (P drive);  I have another to store catalog backups (O drive).  P drive is only connected to computer before I open Photoshop Elements.  O drive is only connected when I intend to save a backup  (and, connected at other times for unrelated purposes, when Photoshop Elements is not open).
• How many backups have you made?  What is your backup practice?  Do you delete old backups after making new ones?  Do you make full backups or incremental ones?  Do you use any other software to backup your computer files?  I have made one full backup of each of our catalogs ( 8 various catalogs).  To make a backup, I have P drive connected, Organizer open to the intended catalog, in Organizer, under File, select Backup Catalog.  Then, I always run a full backup, pointed to the existing backup location, and authorize “over write existing backup”.  I run the backup, until it ends, with “backup completed successfully”.  I use NO other software for backups.

TO RECAP:  My questions are:

• How may I get this one, “fractured” catalog and all associated files, stored and running only from my external “P” drive?
• Why did my backup/restore action fail?  What must I do in the future to create complete, full backups?

THANK YOU FOR YOUR ATTENTION

[MichelBParis]

Hi @josephunit2n 

Thanks for the detailed answers to Greg's questions. I won't interfere with the debugging process with Greg, but I believe you are on the right track and that it will be possible to solve your issue.

However, what you have already answered discloses a very important information:  it seems that your media files trees are subdirectories of the catalog folder. This should never happen. The media folder tree(s) and the catalog folder ideally should always be independent; at least, the media trees should never been included in or under the catalog folder. And that happens very naturally for users following the "default" locations offered by Adobe and Windows:

- The catalog folder situated in a hidden system folder, not displayed in the Explorer unless you set the Explorer to show it.

- The media folders being generally suggested in your preferences to be under the Windows 'Pictures' default category.

There are good reasons to choose 'custom' locations in your preferences for both your catalog location and the default importing or saving paths for your photos.

- if the photo files are NOT included under the catalog folder, you can copy and move that folder where you want and the size of the catalog folder is not huge. Better for alternative backup solutions.

- You can store your catalog folder beside the media folder (what I call the "library") in either external drive you choose, which enables acessing both from various computers by plugging them alternatively - provided the external drive is recognized with the same drive letter.

So, have a look at your organizer preferences to check where are the 'default' paths for storing new imported files and for saving them. Then decide where you want to store the catalog and the "library" folders.

[josephunit2n]

A lot to digest, I will study this - waiting for answer to primary question

[Greg_S.]

Michel has already pointed out the likely cause of the problem.  I will put it more bluntly.  You should NEVER store your photo files in the catalog folder.

I am still having difficulty understanding some of your answers.  But let's see if I have this correct:

1. You have two different catalogs with the same name.
2. You have one catalog on the C drive and one on the P drive.
3. The catalog on the P drive was created from a backup and restore of the one originally on the C drive.
4. After restoring the Backup to the P drive some of the photos in that catalog are displayed as missing.
5. The original catalog on the C drive is intact with the exception of some files that you presumably imported to the catalog that is stored on the P drive.  (This may be a few hundred photos.  You still haven't told us the total size of your catalog.)
6. Your ultimate goal is to have all of your catalog database files and all of your photo files on your P drive.  A secondary goal is to have none of these files on your system C drive.

I am still trying to understand how the photos on the P drive became missing.  If it happened as part of the backup/restore process, then the likely cause is the fact that the photo folders are part of the catalog folder.  The answer to this may also depend on the options you used to Restore the catalog on the P drive.  Presumably, you chose to restore the catalog to a new location.  But did you simply choose the root of the Pdrive or did you specify an exact path?  And, perhaps more importantly, did you select the box to Restore Original Folder Structure?

I am going to ask @MichelBParis to jump back in after you have responded again, because depending on your responses, I may have to do some testing of the best approach for you.  I rarely use the Organizer's backup/restore process, whereas Michel has more recent experience with it.  I am sure that neither of us have any experience with a catalog that contains photos stored in the catalog folder.

And one final question for now:  Have you moved any files around using Windows File Explorer.

And one final thought perhaps for Michel:  Could part of the problem relate to the fact that the external drive is a Raid 1 mirrored drive.  Could there possibly be confusion in the identity of the mirrored drives?  In researching this issue, I came across @josephunit2n's old thread.   And that led me to John Ellis' article which talks about this problem with RAID drives.

[josephunit2n]

Greg:  thank you for your input;  I'm away from my home system, today, therefor will fully respond soon as possible;  I do see you still have some erroneous assumptions regarding this issue.  Briefly,

1) "

1. "You have two different catalogs with the same name." - I would say SAME catalog, in 2 locations ("C" and "P"), same photos (except a few more recent ones added only to the "C" catalog).
2. "The catalog on the P drive was created from a backup and restore of the one originally on the C drive. " NO! The catalog on P has been there since its creation.  What you're confusing:  recently, I tried "backup and restore" of the completely connected catalog on C, to a brand new folder/location on P (having nothing to do with the already existing same catalog in diffent folder location, on P) - and, this attempt failed, as that recent "restore" resulted in restoration to that new P location NOT of the fully connected catalog from C, as intended, rather that restored catalog mimicked the version already on P, that is, all photos with sourc files on C, were not connected - grey thumbnails) -- I have deleted this "restored" catalog from that new location, as it was of no use.

3) "

"The original catalog on the C drive is intact with the exception of some files that you presumably imported to the catalog that is stored on the P drive.  (This may be a few hundred photos.  You still haven't told us the total size of your catalog.)"  NO!  The catalog on C is fully intact, all photos linked (not at home now, I recall around 346 photos, total), though some link to "date folders" on P, most link to source photo files on C.  I did more recently import some photos targeted to P catalog (the only ones in that catalog that are presently linked to source files), and these DO also appear - correctly linked to the same source files on P, IN THE CATALOG ON C --  therefore, again, the catalog on C has all photos, correctly linked

6)  Statement #6 is exactly correct!

When I am back at home computer, I will review Michel's assumption that the date/photo files on P are within the catalog folder (I think this is NOT so, but I will confirm)

Again, thank you so much -- I'll be here again, soon as possible.

[Greg_S.]

I'm sorry, Joseph, but are you seriously telling us that we have spent goodness knows how much time trying to move a catalog that has a total of ~350 photo files?!  If that is the case, then my advice is as follows:

1. Copy all of the photos you want to place in this catalog to your P drive, using Windows Explorer.  If you have multiple folders, place them under a parent folder such as My Special Photos;
2. Go to the Catalog Manager and select the Custom Location option;
3. Browse to and create a custom location folder something like P:\My Elements Catalogs\;
4. Create and open a new catalog with a name that you have never used before;
5. Import the files into the new catalog, using the Import>From Files and Folders option; if you have previously saved any metadata to the files, you will be given the option to include those tags in the new catalog.
6. Perform whatever other organizational tasks you require for the photos such as face recognition, Albums, Events, Places etc. from scratch.

You will now have all of your 350 photos and the catalog folder on your P Drive.

[josephunit2n]

Greg, first apologies for delay, my personal commitments kept me away from computer last few days.  I just finally did a careful review, to answer the storage questions exactly:  

Opening the catalog (catalog.pse21db) on C drive, Adobe default location, most recently added photos are in date folders on P drive INSIDE the folder of the catalog name (Joe’s Images 12_15_2020);  slightly older photos are also on P drive, in date folders OUTSIDE of that catalog name folder (Joe’s Images 12_15_2020) ; the rest, the majority, are on C drive, in that default Adobe location, in date folders, INSIDE the catalog name folder (Joe’s Images 12_15_2020) .

Even with your new suggestion, creating a new catalog, -- with this now exact storage information -- I need to know why I can't take a backup of this complete catalog, then get a complete restoration at a site of my choosing.  I ask not only as another means to restore this catlog complete on the P drive, but also general knowledge as to why my backup/restore fails - I must know how to make successful backups for all catalogs, going forward.

[josephunit2n]

PLEASE PROVIDE SOLUTION USING ADOBE TOOLS:

Yes, your remedy will work for this small catalog.  I asked this question with other catalogs in mind, having thousands of photos, where manual rebuilding is not an option.  I intended to use the directions given to me here, to correct those catalogs, as well.  

Those other catalogs fall into 2 categories:

1.  Some have ALL photos stored in the external P drive as desired, only the catalog data file with "catalog.pse21db" is on the C drive ; HOW COULD I GET THIS MOVED TO P DRIVE, AS WELL?
2.  Two have the status of (1) abobe, plus all photo source files are also on C drive (in other words, everything on C drive;  HOW COULD I GET THIS MOVED TO P DRIVE, AS WELL?

With these answers, my questions will be complete, and I will be able to properly "rationalize" all catalogs!

[Greg_S.]

Some have ALL photos stored in the external P drive as desired, only the catalog data file with "catalog.pse21db" is on the C drive ; HOW COULD I GET THIS MOVED TO P DRIVE, AS WELL?

There are at least two options:

1. Simply use Windows File Explorer to cut/copy the entire folder containing the catalog data from the C drive and paste it to the P drive.  To identify the correct folder on your C drive, open the catalog and go to Help>System Info . . . .  The exact location of the folder will be identified.  Copy the entire folder which will have the same name as the catalog. 
2. Use the Move button in the File>Manage Catalogs dialog.  However, to use this option, the catalog to be moved cannot be open.

Two have the status of (1) abobe, plus all photo source files are also on C drive (in other words, everything on C drive; HOW COULD I GET THIS MOVED TO P DRIVE, AS WELL?

Do a full backup of the catalog and restore using the New Location on the P drive for the restore.  You can maintain the existing folder structure for the restored catalog.

See this HelpX article for more details.

For those catalogs still on the C drive, if the photo files are stored in the catalog folder, I would move the files using the Folder Panel in the Organizer so that the files are in a more standard location.  However, if those same photos are in more than one catalog they will become missing in the other catalogs.  This is why it is highly recommended to use only one catalog. 

[josephunit2n]

Thank you!  To your last point, I do not have any photos in more than one catalog.  Busy weekend ahead, I will not get to this for some days, as I want extended, uninterrupted time time for the work.  Hopefully, all will conclude well, otherwise I'll send follow up question (not a threat OR a promise !!!  ).

[Greg_S.]

@josephunit2n said:  To your last point, I do not have any photos in more than one catalog. 

My point is that you do not need to put your photos in more than one catalog.  There are other ways to meet your needs.  I have one catalog containing over 250,000 photo and video files.  I can find any photo I want within a matter of seconds.