P: Wrong timestamp stored in catalog causing wrong metadata status (all Windows versions)

[Samoreen]

Hi,

The problem I describe below is not new (I observed this at least since version 3) but this time I took the time to investigate more deeply...

From time to time, LR tells me that the XMP file of a given image is no longer in sync with the metadata in the catalog. Most often, this is correct because I made changes without recording the XMP file (Ctrl-S - I'm not using the automatic XMP updating mode). But very often, this information is simply wrong. I didn't change anything to the image and suddenly, the little down arrow appears in the upper right corner of the thumbnail.

Hitting Ctrl-S may or may not fix the problem. Sometimes, the little down arrow reappears after a few seconds or minutes although I didn't do anything (hands away from keyboard and mouse).

I recently did the following test for multiple images unduly displaying the "metadata status changed" flag. I compared the following values :

1. Windows "last modified" timestamp for the XMP file.
2. Value of the xmp:MetadataDate field in the XMP file.
3. touchTime column value for that image in the Adobe_images table of the catalog (which is a SQLite database).

The touchTime value is stored in a special format, so I had a hard time converting it to a readable date/time value. However, I will not explain this and how I navigated the database in order to access this timestamp (this requires some knowledge about databases).

Result:
For all the images tested, values #1 and #2 were always strictly identical. The touchTime value was always off (sometimes about 10-15seconds, sometimes much more). So no wonder that LR thought that the XMP file and the metadata in the catalog were not in sync.

Moreover, the difference in time can be negative of positive. So LR displays the up or down arrow accordingly (meaning that the XMP file is older or newer than the metadata in the catalog, respectively - which is wrong in both cases).

I explained above that sometimes, the image reappears as "not in sync" just a few seconds or minutes after I did a Ctrl-S. In that case, a quick look at the database showed me that the touchTime field had not been updated. So the time difference causing the image to be flagged as "not in sync" was still there. In that case, the problem can be fixed by reading the metadata from the XMP file which was actually correctly updated. This operation updates the catalog and everything is in sync again.

Anyway, there's something very wrong in the computation of the touchTime value of the Adobe_images table. That seems to be obvious. This wrong timestamp generates in turn a wrong metadata status.

I made another interesting test :

1. I started from a situation were 0 image was flagged as "not in sync" with the XMP file.

2. I purged all 1:1 previews and started a Build all 1:1 previews.

3. In Library mode, I setup a filter to show only the images that had the Metadata Status set to "has been changed". I got a cup of coffee and waited.

At the beginning of the generation process, no image was displayed in the grid, as expected. While LR was building the previews, images unduly tagged as "not in sync" started to appear. I got about 200 of them. For all these images, the metadata status was just plain wrong. These were finalized images not modified since a long time and for which the XMP file had been timely updated after the last modification. I checked the touchTime field for some of them and each time it was different from the Windows "last modified" timestamp and from the xmp:MetadataDate field of the XMP file as mentioned above.

So now I know what's going wrong but I'd like to have this problem fixed after all these years.

Thanks in advance.

[MBeckerAC]

Hi Adobe.

Another try to help you:

May it be that you store a "Cache Flag" in the metadata?
So - whenever an image goes in or off the cache, you change the metadata, i.e. this flag..

If so - this is ill-designed.==> You have to correct this and store such a flag elsewhere, e.g. in the catalogue!

Kind regards.Michael 

[Samoreen]

Hi,

Not only this bug still has not been fixed in version 9 after all these years but things now appear to be worse than usual. After installation, I found myself with hundreds of files with their metadata status allegedly marked as "not saved" which is plain wrong. When using the usual trick that temporarily fixes the problem (Ctrl-S, then Read Metatadata from file), I had to "read the metadata from file" multiple times for some of these images before the metadata status got the correct value. And as usual, the status returned to "not saved" for some files after a few seconds.

I regret that the developers of LR have not enough self-esteem and pride to definitively fix this infamous bug lasting since the early versions of LR and disturbing the workflow of many users. I personally would be ashamed of this.

[Samoreen]

I confirm that in version 9 the "Ctrl-S - Read Metadata from file" workaround is no longer a safe way to (temporarily)  circumvent the metadata status bug. Occasionally, the image remains flagged as "metadata status changed" after this operation. So, a new bug has been added to the original.

Also, it seems that now, visiting a folder or a collection containing images for which the preview has been purged, systematically causes the metadata status flag of these images to be reset to "changed" when the preview is rebuilt. I got hundreds of images unduly tagged this way after reviewing some folders and collections this morning.

So, this flag that is so important for managing the XMP files effectively (which I consider as a complementary way of backing up my edits, among other uses) is now definitively unreliable. Good Golly, Miss Molly, LR continues to rock. Just a little more effort guys, and this bug will celebrate its tenth anniversary.

I just pre-ordered a Capture One 20 license.

[Samoreen]

Hi,

OK. If the development team are not able to understand what has been explained above, I can describe this bug another way : if an image has not been viewed in the browser since a long time (long enough to cause the preview to be purged), just viewing this image again without changing anything in the image edits can trigger a reset of the metadata status flag.

Is this enough to let us hope for a fix before this bug celebrates its 13th anniversary ?

[MBeckerAC]

Dear Adobe staff.

I would appreciate any comment.

Thank you and kind regards

[bob frost]

Is that not because simply looking at an image is 'touching it' and therefore the LR catalog item 'touchtime' will be changed. That puts the catalog at odds with the touchtime stored in the xmp sidecar unless you have set update xmp automatically. I have that set, and never have this problem. Win10.

Just a guess!

Happy New Year,

Bob Frost

[MBeckerAC]

The problem is not keeping the xmp sidecars up to date - but the "meta data changed flag" is resulting in re-uploading the images to the publication services.

Re-uploading as a result of just and only touching (i.e. viewing) an image is not only superfluous, but a serios performance problem. For me, if I am viewing images, this results in 100s, sometimes 1000s of images to be re-uploaded

Another aspect:
If the touched image is still in the preview cache ... then the "meta data changed" flag ist *not* set.

So, it looks like touch time is not the reason and not an explanation for this software bug.

[Samoreen]

Michael is correct,

The problem is that the timestamp recorded in the catalog is not kept in sync with the time stamp recorded in the XMP file, as explained at the beginning of this thread. I was just trying to make this clear to the people who should have fixed this since many years. Since the technical explanations that have been given by me and John + the test allowing to reproduce the problem seem to be too complicated for them, I was trying to give a more simplistic way of tracking and reproducing the bug.

Also, I should say that I have been contacted by people from India who seem to be in charge of this case. The questions they asked and the tests that they asked me to do have demonstrated that they merely didn't understand what has been explained and reported in this thread. Anyway, they have stopped answering my questions.

[Samoreen]

Still not fixed in version 9.2. Are you surprised ? I'm not. Lightroom is like Arthur Rimbaud's Drunken Boat.

As I was going down impassive Rivers,
I no longer felt myself guided by haulers:
Yelping redskins had taken them as targets
And had nailed them naked to colored stakes.

[FeedbackCommunityMember]

Hello, I just wanted to say that I experience the bug for many years and now I finally found the exact description of reason why Lightroom wrongly behaves. I was getting mad after all the cmd+s and still seeing "metadata has been changed". I desperately await once there is a comparable application to Lightroom so I can move away from it for good. Since it takes more 10 years to fix such a small bug to Adobe, I believe that another company (hopefully Affinity) will introduce their solution sooner. 

[Samoreen]

Hi,

There's a new related bug with version 9.2.

Previously, when using the Metadata | Read metadata from file command, the up/down arrow at the top right corner of the thumbnail was reset (which is expected). Now it shows a down arrow, meaning that after reading the metadata from the XMP  file, the catalog is still considered as not in sync with the XMP file.

The original bug is already annoying enough, it's not necessary to add insult to injury.

[Samoreen]

After multiple tests, I confirm. Fixing the wrong state of the metadata status arrow by using the Ctrl+S | Read metadata from file sequence is no longer working while in the Library module. However, when switching to the Develop module, the status arrow is reset.

So, not only that old bug is still not fixed but it just became worse.

Adobe, please tell the programmer who is in charge of this to keep his/her hands off that code.

[FeedbackCommunityMember]

Now one year later this is still a problem. I have a little addendum to the problem. This is actually much much much worse when the actual files resides on a mounted network drive.

But why in the first place relay on timestamps to determine when files have been changed - silly. Use a MD5 sum instead. Just MD5 sum the file - save the MD5-sum in the database and when determine if file is changed - do an MD5 sum and compare to the saved in the DB. Thats all there is to it...used this method a lot for various development projects.

The catch about timestamps is that most backup solution "touch" the file when doing backup, silly i know, but thats often the way they do it - this means after each backup everything is changed. This forces me to have the files in an "unbackuped" space - and after changing stuff they will be rsync'ed to a backed up space - requireing double internal storage. I know disk space is cheap, but still a problematic way of doing things....

[Samoreen]

*Bjarne Christensen 

Hi Bjarne,

I have given up any hope about this issue. One may wonder why such a bug has not been fixed after all these years. Laziness ? Maybe. But I think there's a more basic reason. Don't forget that a good part of LR is written in LUA, a language that doesn't make debugging and maintenance very easy. The problem with such languages is well known. Once the original developers have been moved to another project or have left the company (although I think that this code has been at least partially subcontracted to offshore companies), the new developers who are in charge of maintaining the program are very often unable to understand the existing code, especially if it has not been properly documented.

I'm pretty sure that this is the reason why so many "old" bugs have not been (and will never be) fixed. They just can't. They are out of control or are not skilled enough. Otherwise, this would demonstrate that they just don't care, which wouldn't be a surprise either.

[mlimke]

I just had some success with the "failure to make metadata changes stick" problem and thought I'd share. This is when you see "unsaved changes," select the photos, save metadata, and then they quickly become unsaved again on their own.

I am on Windows 10, using the Creative Cloud licensing, so I'm up to date. Most of my catalog was created with an older version of Lightroom, a standalone app from 10 years ago, I'd guess.

Today, I found that this sequence helped:

1. Select the problem photos
2. From the menus, click Photo > Develop Settings > Reset (this is also on the right-click menu)
3. Save the metadata to disk with Cmd/Ctrl-S

The change have stayed put and now I have no more unsaved changes

This did away with photo modifications such as white balance and cropping, of course, but for me, that was an acceptable tradeoff. I did fiddle with the related menu action "Update to Current Process Version," but it wasn't consistently workable. Your mileage may vary.

[Ian Lyons]

This issue has been around since the very first builds for Windows in June 2006 (i.e. Public Beta 4). It was identified then by the Engineering Lead and multiple times by other Adobe staff ever since as a limiation of Windows. The same issue occurs with History and other areas with lists.