Change drive letter breaking links to some, but not all, files

Report · Jun 01, 2023

I have my PSE 2020 catalog and image files on a USB drive that I want to swap between machines. If I had thought it through, I would have set the drive up as P: on both machines to avoid conflicts, but I didn't. I created it by restoring from a backup on to my laptop with the drive mounted as E:

Predictably, when I go to my PC, E: is already assigned and the drive is mounted as G:

I was expecting that either:

- PSE would take the location of the files relative to the location of the catalog and everything would work

or

- PSE would treat the links as absolute addresses and all of the files would be pointing to E: and have broken links.

In fact, all the files which I migrated from a previous version of PSE seem to be pointing to G: without any intervention, but the ones that I have imported since moving to PSE 2020 are pointing to E:.

I guess that I can fix this by:

- Setting the drive up as P: on both machines

- Restoring from backup to P: to create a new catalog where everything points to P:

- Deleting the old version that refers to E:.

However, I am cautious about trying to fix a problem that I don't understand. Why are some files linking OK and some aren't?

Report · Jun 01, 2023

Not an easy one...

@Billy5FE5 a écrit :

I have my PSE 2020 catalog and image files on a USB drive that I want to swap between machines. If I had thought it through, I would have set the drive up as P: on both machines to avoid conflicts, but I didn't.

Indeed...

I was expecting that either:

- PSE would take the location of the files relative to the location of the catalog and everything would work

or

- PSE would treat the links as absolute addresses and all of the files would be pointing to E: and have broken links.

The location of the files in the database is stored in two separate pieces.

a- as you are expecting, the path of subfolders under the letter of the master folder of that drive

b- the internal database code of the drive, which describes the properties of that drive:

- the most important one is the internal serial number read by Windows from the specifications of your physical drive

- the drive letter assigned by Windows either on the first connection of the drive or by yourself as you could have done.

- other info like drive type (builtin or external), drive denomination in Explorer...

So, the full location of the file is the full path starting with the drive letter followed by the path on the drive: "E:/myphotos/Year2022/.....

The organizer keeps track of the various drives used in your catalog in a separate table. From the media table you have an internal (never displayed) numerical identifier for the drive from which the two pieces are found, letter drive and path, and which are concatenated to produced the full path you would use to specify a file yourself in the explorer.

Users who are proficient in sqlite database management and Windows commands could solve your problem by editing the media and/or volume_table. That what explained in an old conversation some ten years ago but my bookmarks have disappeared in the forum upgrades...

More info:

https://johnrellis.com/psedbtool/#_Detecting_Problems_with

Report · Jul 03, 2023

I finally got round to trying the backup and restore to sort this out and have more inconsistancies.

The catalogue was in a folder, PSE Elements, on a removable removable drive (let's call it the cataolg drive) to another (the backup drive).

I changed the drive letter on the catalog drive to P: then restored from the backup drive to a new folder on the same drive (P PSE Elements).

I now have a new catalogue on P:/P PSE Elements, but many of the images on the new catalogue are pointing to files in the old folder. As far as I can see, the files exist in both folders - when I tried to manually fix this by moving the folders that are pointing to the old location (within the folder view of PSE) I got an error message saying that the folder already existed.

How can a backup and restore leave this situation where some of the files are pointing to a location that is different from the folder I specified?

I am absolutely baffled by how PSE is behaving and have lost confidence in it - if I could wave a magic wand and switch to another solution I would, but I suspect that any migration process will be complicated by these anomalies.

Billy

Report · Jul 03, 2023

Hi @Billy5FE5

I think it's necessary to come back to your first post.

"I was expecting that either:

- PSE would take the location of the files relative to the location of the catalog and everything would work

or

- PSE would treat the links as absolute addresses and all of the files would be pointing to E: and have broken links".

1 - A catalog (which is a folder in PSE) can be copied or move nearly anywhere on various drives and folders; it always keeps the location/description of the files it does link to in its database. The location of the catalogued files is totally independent from the location of the catalog file.

- PSE would treat the links as absolute addresses and all of the files would be pointing to E: and have broken links.

2 - PSE does treat the links as absolute addresses, but:

a - the addresses in the database don't refer only to the drive letter name which can be changed by external factors as in your own case. Remember that when the organizer was created, most drives were floppy CDs and you could not expect a given CD to keep the same letter when other CDs or devices were plugged in. So the organizer conceptors chose to identify the drive primarily by the internal serial number of the drive as registered by Windows.

So, in Windows, the drive is described by 3 properties, the drive letter, the internal serial number and the drive label as shown by the Explorer. The most reliable item is the internal serial number which nobody knows, but which is accessible through a DOS command. So, in many cases, the organizer is able to keep the link to the original drive even if its letter has changed.

- b - In a backup/restore process, you can want to restore files situated in different drives, different folders or even from network locations.Users generally expect to restore on the same various locations as the original ones, or to a new common master folder, where different original drives are converted to new subfolders with the name of the drive. In that case, the addresses have to be updated to the new absolute location. Of course, to restore on various original drives, they must be available for the restore.

tip: the text command in Windows powershell to find the internal serial number is VOL X: to get the hexadecimal value of the drive with letter X.

In practice, to share an external drive containing both the catalog folder and the media files folder trees, it's best to use letters much farther from the usual a, b, c, d... like x, ky ,z.

Don't forget that in the case of a backup/restore, the real addresses have to be updated to reflect your own choice between original or 'custom' location.

Report · Jul 03, 2023

Thanks for all the time you have taken to respond to my question. I appreciate the effort but I don't understand what you are suggesting I need to do to fix this and the latest response doesn't seem to be addressing my current situation.

I have images in one folder. I backed that up and restored it to a different folder. Instead of having all the items in the new catalogue pointing to the new folder, some of them (only some of them) point to the old folder! Is that not a weird thing to have happened? Surely the restore ought to have linked only to files in the new folder?

Any ideas about how I can create a clean version of my catalogue where all the images point to the same structure?

I think that it is beyond me to work out how the old structure works but I thought that backup and restore was the drastic but dependable solution - basically rebuilding from scratch - but even that hasn't given me a consistent structure.

Report · Jul 04, 2023

I have images in one folder. I backed that up and restored it to a different folder. Instead of having all the items in the new catalogue pointing to the new folder, some of them (only some of them) point to the old folder! Is that not a weird thing to have happened? Surely the restore ought to have linked only to files in the new folder?

By @Billy5FE5

Sorry if I am not able to give you a simple solution.

To answer your question above:

The drives and folders structure of your media files is recorded in the main database (catalog.pseXXdb) in the catalog folder. It's quite possible that there is an identification error for your drives in the current catalog. If so, creating a backup will reproduce the same error. The backup folder will contain two components which can reproduce both the volumes identification and the media folder tree structure:

- a renamed copy of the catalog.pseXXdb, now catalog.buc

- a tly file: backup.tly which is a special text file recording the structure of the original media folders.

So, if the error is already present in the catalog before the backup, it will be reproduced in the restored files.

Generally, if there is no error in your source catalog, you'll find no missing (disconnected) files and you'll see that the view of your folders will be the same in the Explorer and the 'folders view' in the organizer. Then the backup/restore will act as expected.

If there were errors prior to the backup, there is no 'miracle' solution to correct them, even for expert users with sqlite commands knowledge (not my case). So, it depends if the number of missing files is important or not. You might select and delete the missing files; perhaps locate the missing folders, maybe move them to a known, safe location and import them again from there?

Report · Jul 05, 2023

I guess haven't been able to explain the problem properly and I undertsand why you wouldn't get it, because the situation looks so bizarre to me. I am in the process of repeating the backup and restore, so I'll let you know if this is repeatable or not.

The point is that there are no unconnected files. Every item on the newly restored version of the library is connected to an existing file. The weird thing is that a lot of them, but not all, are connected to the old folder structure rather than the new folder - it is as if there were unconnected files but PSE has been able to search the drive and find the old ones and connected to them.

I know that PSE has a feature that will attempt to find and repair broken links so maybe this is not an impossible explanation.

I am thinking about how how I can test this theory. I would like to restore on to a completely clean platform, but I don't have one spare. Even if I bought a new drive, there are image files kicking about on the laptop and the PC that might get in the way. (And I don't feel like paying for a new drive just to test this theory.)

Report · Jul 05, 2023

You say:

"I know that PSE has a feature that will attempt to find and repair broken links so maybe this is not an impossible explanation."

I don't believe that would work. The automatic relinking is extremely slow and needs to be guided.

What is possible based on the way the organizer identifies the drive from its internal serial number primarily instead of the drive letter is the following intended behaviour:

- The organizer stores both criteria (letter and serial) in the database in a table describing the known drives being recorded in the catalog. The hidden internal identification of that drive item attributed by the sqlite engine can be searched by either its letter or serial property.

- for some reason, the removable drive (CD or ext. drive) gets a new letter. When it is plugged in, the organizer reads the new properties, unchanged serial and changed letter. The idea is to rely primarily on the serial to recognize the initial recording. The true (hidden) sqlite identification number is found; no new drive identification is created in the volume_table; the former record in that table is kept and the letter drive is updated.

This is how I understand what takes place when a drive has changed only the letter. Just my supposition.

- The main media_table also has an (hidden) sqlite identification and records the properties of each media files, precisely its location known by two criteria:

- the folder path as a text input

- the volume (hidden) sqlite identification as described above.

To locate the file in the Explorer, the organizer creates the full path: calculated letter (eg P:/) followed by the text path = P:/My Pictures/.../.../pic....jpg and sends the result to the Explorer.

In my experience, this is what happens correctly in many circumstances, but there are frequent cases where that can't work.

The problem with relational databases like sqlite is that they don't allow any inconsistencies in a table, for istance in the volume_table you can't have two conflicting items for example two with the same serial and each a different letter depending on the corresponding indexes. Another example; you are using a cloned SSD drive to replace the old conventional one. It gets the same serial number. Don't try to use both drives at the same time!

A screenshot showing the volume_table displayed by an sqlite utility might help in understanding the confusion.

I can't be sure, but I think that such an internal workflow would allow a change of letter without disconnecting files AND to allow a backup process resulting in a restored catalog with ALL the files AND the database updated for the new letter for all its files before and after the letter change.

Report · Jul 05, 2023

I repeated the backup, change volume name and restore to new folder with the same results.

I am confused about why the volume table is relevant here when the files are all linked to the correct drive (this time the drive is X). The problem is that many (the contents of hundreds of sub folders) are linked to folders outside the destination folder that I specified in the restore operation. How the restore knows anything about those locations is beyond me.

However, I have installed SQLite Studio and will try to give you the information that will help debug this, if you are still willing.

Above is the data tab of the volume table and I will happily send more information that would be helpful.

Report · Jul 06, 2023

Very good thing you now have this sqlite utility.

A few tips:

- the first two items, n_drive and database relative are always present even if no media are linked to them

- find the serial name of a driver from its letter: command prompt "vol X" where X is the drive letter.

- the displayed result in in hexadecimal and is recorded in decimal, use the calculator in programmer mode.

- the 5th item (X) may be correctly stated as 'built_in' if it's an external USB drive for instance

- no idea about the 6th one... probably no link to your present isue.

I'll take more time to try to simulate the same issue. Could you show screenshots in the explorer and catalog folder view of files which are found but not in the expected folders?

Report · Jul 06, 2023

This is how the folders look in Windows Explorer.

The original structure, now under X:/PSE Library, has several hundred folders and they start like this:

The restored catalog, which I restored to the folder "PSE library on X", is similar (this time I ticked the box to retain the folder structure, so it all sits under a sub folder "PSE library")

(Ignore the folder /P PSE Library - that was the previous attempt to backup and restore and I used P as the drive letter.)

I haven't done a file by file check yet but it looks like everything that was under "E:/PSE library" has been recreated under "X:/PSE Library on X/PSE Library". I'll use TreeSize to compare the two later to check that they are absolutely identical, but last time (when I restored to P) there were no unexplained differences between the two folders - the restore seems to recreate the folder and files OK.

The mystery comes when I look at folder view in Organizer.

First, let me show that I am using the new catalog on /PSE library on X

The folder view of this catalog shows some files are linking to the old structure, outside of the folder that I specified in the restore.

There are too many folders to show the whole list, but the ones on the old structure start off like this, so it seems to be the oldest folders that are pointing to the old catalog.

The folders that look OK, in other words pointing to x:/PSE library on X, are later in date:

The date here is probably significant because this is when I upgraded to PSE 2020 from an older version of PSE (I am not sure which version it was). Images imported by PSE2020 seem to be restored correctly. Images imported in previous versions are being linked to a folder outside the destination folder of the restore.

I agree that automatic repair by PSE isn't the cause. However, I have run out of ideas about what to try next.

As an aside, I worked in IT tech support for many years and I someone had reported this to me, I would be very sceptical that the report was accurate so I appreciate you sticking with this. If you want any more information or evidence, let me know.