P: Wrong timestamp stored in catalog causing wrong metadata status (all Windows versions)

[Samoreen]

Hi,

The problem I describe below is not new (I observed this at least since version 3) but this time I took the time to investigate more deeply...

From time to time, LR tells me that the XMP file of a given image is no longer in sync with the metadata in the catalog. Most often, this is correct because I made changes without recording the XMP file (Ctrl-S - I'm not using the automatic XMP updating mode). But very often, this information is simply wrong. I didn't change anything to the image and suddenly, the little down arrow appears in the upper right corner of the thumbnail.

Hitting Ctrl-S may or may not fix the problem. Sometimes, the little down arrow reappears after a few seconds or minutes although I didn't do anything (hands away from keyboard and mouse).

I recently did the following test for multiple images unduly displaying the "metadata status changed" flag. I compared the following values :

1. Windows "last modified" timestamp for the XMP file.
2. Value of the xmp:MetadataDate field in the XMP file.
3. touchTime column value for that image in the Adobe_images table of the catalog (which is a SQLite database).

The touchTime value is stored in a special format, so I had a hard time converting it to a readable date/time value. However, I will not explain this and how I navigated the database in order to access this timestamp (this requires some knowledge about databases).

Result:
For all the images tested, values #1 and #2 were always strictly identical. The touchTime value was always off (sometimes about 10-15seconds, sometimes much more). So no wonder that LR thought that the XMP file and the metadata in the catalog were not in sync.

Moreover, the difference in time can be negative of positive. So LR displays the up or down arrow accordingly (meaning that the XMP file is older or newer than the metadata in the catalog, respectively - which is wrong in both cases).

I explained above that sometimes, the image reappears as "not in sync" just a few seconds or minutes after I did a Ctrl-S. In that case, a quick look at the database showed me that the touchTime field had not been updated. So the time difference causing the image to be flagged as "not in sync" was still there. In that case, the problem can be fixed by reading the metadata from the XMP file which was actually correctly updated. This operation updates the catalog and everything is in sync again.

Anyway, there's something very wrong in the computation of the touchTime value of the Adobe_images table. That seems to be obvious. This wrong timestamp generates in turn a wrong metadata status.

I made another interesting test :

1. I started from a situation were 0 image was flagged as "not in sync" with the XMP file.

2. I purged all 1:1 previews and started a Build all 1:1 previews.

3. In Library mode, I setup a filter to show only the images that had the Metadata Status set to "has been changed". I got a cup of coffee and waited.

At the beginning of the generation process, no image was displayed in the grid, as expected. While LR was building the previews, images unduly tagged as "not in sync" started to appear. I got about 200 of them. For all these images, the metadata status was just plain wrong. These were finalized images not modified since a long time and for which the XMP file had been timely updated after the last modification. I checked the touchTime field for some of them and each time it was different from the Windows "last modified" timestamp and from the xmp:MetadataDate field of the XMP file as mentioned above.

So now I know what's going wrong but I'd like to have this problem fixed after all these years.

Thanks in advance.

[Samoreen]

Thanks very much, John.

[MarcoKlompalberts]

With all the detective work Patrick has done, you would think that Adobe would at least have a clue where to look for it.
Having the same issue here.
Once in a while, I will save to file, to make sure that everything is in sync.
I will usually select (with smart collections) all photos from 1 year and Save To File.
But I am hesitant to Read From File, because what if the save didn't go as planned?

Really hope that this will be fixed.

I am on a Windows 10 PC with the most recent update of Lightroom CC Classic installed.

[MarcoKlompalberts]

By the way, I noticed you can search for (and have smart collections with) metadata status.
It would be handy if you could also search for metadata date. That way I would be able to Save To File and afterwards Read From File if the metadata date is after the last save time
(hope that thought makes sense 😉 )

[Samoreen]

Hi Marco,

This bug exists since the beginning of Lightroom and I have been experimenting with the Save/Read workaround a lot. AFAIK, this procedure is safe. Never had a problem with this. However, this fixes the problem only for a (sometimes very short) while. As soon as the files are accessed again (for example when rebuilding a discarded preview), the problem may appear even if you didn't make any change. Just viewing the images is enough.

I am using a single rule dynamic collection (Metadata Status is no up to date) to regularly fix this mess. Open the collection / Select All / Save.

As for a fix by Adobe, give up any hope. The usual rule applies : once a bug has survived one or two years, you can be assured it will never be fixed. They just don't care and this is a demonstration that the bug  doesn't  prevent the product from being sold anyway. High code quality and professional conscience are no longer a preoccupation at Adobe.

[3badDogs]

IMHO, the LR dev team and their processes (QA in particular) is no where near the quality of the PS development team. Adobe have the capability but seem content on collecting their monthly fees and not delivering what was promised. I am still confounded why no one has not yet brought a class action law suit against them. Someone who owns a business impacted by these bugs surely would be a good lead plaintiff.

[Samoreen]

Still not fixed in 7.5 but I would be inconsistent if I said that I am surprised. :--((

[ssprengel]

If this is caused by LR updating something in the metadata, perhaps the metadata version number, as a background process that is scanning all the photos, then it is not a bug.

Do we know what is changing, if anything, by comparing the XMP file before and after a Write Metadata to Files?

[Samoreen]

Not fixed in LR 8.

[MBeckerAC]

Same on Mac OS.

[DMEEPhD]

Yes, I see this in Mojave (and it was present in High Sierra), but I've only seen it with LR7.5 and LR8.

[MBeckerAC]

Dear Photoshop-Lightroom development team.
This is a serious flaw and technically can be solved with some basic knowledge. So it might be a good idea to hire some engineers with knowledge in backup software.

[FeedbackCommunityMember]

For months now I have struggled with a situation which is quite similar to the one described by Patrick in this thread (details, if needed: metadata conflicts with large files and a NAS). 

Thanks to Patrick for the work! The explanation is very helpful and makes the situation easier to handle.

However, I am deeply disappointed by Adobe's way of (not) handling this:

- The issue apparently has been known for many years.

- Three years ago already an analysis was provided in this thread with a relatively simple first approach for a fix.

- Despite users doing Adobe's work in this case, the issue has not even been properly addressed. 

This does not only cause paying users to waste time every day due to the false error messages. It also causes additional loss of time due to the necessity to investigate the issue in the first place. The latter could be avoided if Adobe at least documented the issue properly. It is anybody's guess how many people have spent plenty of time trying to understand the problem and find a solution.

Request to Adobe & staff: If you are not willing or able to fix this issue, please show some basic respect for your customers and respond to this issue and document it in an appropriate place--one that is easy to find even for a newcomer, e.g. in the FAQ.

[Rikk Flohr: Photography]

I've asked the team to review this thread again. .

[Samoreen]

Thanks.

Given the information provided in this thread by John R. Ellis and me, the problem should be easy to fix. At least, it should be easy to be more tolerant when comparing the timestamps. A very small difference shouldn't be taken into account.

[FeedbackCommunityMember]

Thank you for responding.
I think it would be appreciated if you or the team could keep us posted. 

[Smit K]

Hi Patrick,

We have been trying to reproduce the issue and have been successful with soem DNG files.

Could you confirm if you are facing this issue with only DNG files or all types of files.?

Thanks,
Smit
Lightroom Classic CC Team

[FeedbackCommunityMember]

Are you guys reading the posts? Even I can say that yes Patrick has confirmed file format doesn't make a difference.

[Samoreen]

Hi,

The file type is not relevant. As described above, the problem is easy to reproduce. So I reprint :

I made another interesting test :

1. I started from a situation were 0 image was flagged as "not in sync" with the XMP file.

2. I purged all 1:1 previews and started a Build all 1:1 previews.

3. In Library mode, I setup a filter to show only the images that had the Metadata Status set to "has been changed". I got a cup of coffee and waited.

At the beginning of the generation process, no image was displayed in the grid, as expected. While LR was building the previews, images unduly tagged as "not in sync" started to appear. I got about 200 of them. For all these images, the metadata status was just plain wrong. These were finalized images not modified since a long time and for which the XMP file had been timely updated after the last modification. I checked the touchTime field for some of them and each time it was different from the Windows "last modified" timestamp and from the xmp:MetadataDate field of the XMP file as mentioned above.

[Sreenivas R]

Hi Patrick,

First of all sorry that this has not been acknowledged or fixed for many years. Anyways let us work together and see if we can find the root cause and fix it.

Based on your investigation I created a test catalog and imported a single image into it and went through the steps multiple times. But I was not able to reproduce the issue. I think difference in touchTime doesn't seem to be the cause behind the bug. I could be wrong here but from the first looks that is what I feel. For instance I make a edit (star rating change) at 7:00 and do a metadata save at 7:10 I find that touchTime remains at 7:00 and xmp file gets 7:10 timestamp written into it. Now even with this difference the icon doesn't show up even after building previews multiple times. 

I also find many lines of code written comparing many time stamps before the status icon gets displayed. So I want be  sure I get to see the exact state in which the icon gets displayed for you. For this I need help from you as follows.

1. Create a test catalog with few images (please make sure you are using photos which you can share with me). It would be best if you keep the images next to the catalog itself.

2. Make edits and save the metadata. Make sure non of them are showing the metadata icon. Quit LR. Make a copy of the photos and catalog into a separate folder named "base".

3. Open LR and go through any steps needed to get the metadata icon to get displayed incorrectly either on one or more photos. Quit LR. Make a copy of the photos and catalog into a separate folder named "error".

4. Zip the base and error folders and share it with me via dropbox or weTransfer.

You can provide the link here if it is ok with you or I will mail you privately for the link.

Thanks

[Samoreen]

Hi Sreenivas,

Thanks for the effort after all this time. I'm almost surprised.

Before I make the requested test...

Did you actually make the following test on a regular catalog (please note that I'm working with a dual screen configuration - main window on the main screen and grid view on the secondary display) ?

1. I started from a situation were 0 image was flagged as "not in sync" with the XMP file.

2. I purged all 1:1 previews and started a Build all 1:1 previews.

3. In Library mode, I setup a filter to show only the images that had the Metadata Status set to "has been changed". I got a cup of coffee and waited.

At the beginning of the generation process, no image was displayed in the grid, as expected. While LR was building the previews, images unduly tagged as "not in sync" started to appear. I got about 200 of them. For all these images, the metadata status was just plain wrong. These were finalized images not modified since a long time and for which the XMP file had been timely updated after the last modification. I checked the touchTime field for some of them and each time it was different from the Windows "last modified" timestamp and from the xmp:MetadataDate field of the XMP file as mentioned above.

This makes the bug reproducible for every machine I run it on. For me, it's certain that the error occurs when purged previews are regenerated when I visit a folder for which image previews have a good chance to have been purged (believe me, I have much experience with this bug). IMHO, working on a catalog containing only a few images is not enough (unless you force the previews to be regenerated).

[MBeckerAC]

Dear Adobe team.

1. This software bug is there since YEARS !

2. It is EASY to reproduce: PLEASE read Patrick Philippot's comment.

3. Please hire some expert from backup software companies - they know how to determine, whether a file has changed or not.

In other words: Please act and correct this!

Not so kind regards
Michael

[Samoreen]

Sreenivas,

Would you mind answering my question ? Did you actually run that test ?

[MBeckerAC]

Dear Lightroom-Team,
anything new?

[Samoreen]

The more I think about this bug and others that are lasting since years, the more I'm convinced that the incapacity to fix them is related to the programming language used to develop a big part of LR : LUA. Beside many other flaws, this language lacks the necessary features to effectively track, spot and fix bugs. After all these years, I'm still wondering why this disastrous choice has been made.

Anyway, there should be a way to fix this bug.

Wake up !

[MBeckerAC]

Dear Adobe Team.
I would appreciate any - ANY ! - kind of feedback to this problem.

Thank you and kind remarks,
Michael