P: Photos are marked for republish even though there is no change.

[Dr. Fred Mbogo]

I noticed after updating my new v10 converted catalog that LrC was acting slow in the Library module, so I disengaged the usual culprit for this, the "Automatically write changes into XMP" setting. Then to my dismay, I noticed that it proceeded to mark the majority of the photos in the current grid view as needing a metadata update, even though the catalog metadata was fully saved to XMP prior to the upgrade, as far as I was aware.

Then I paged down, and the pattern repeated: Lightroom scanned all the now-visible photos and found that almost all of them also needed to be updated on disk.

And I did it again. And again.

Eventually I wrote a script to send "Page Down" events to the program periodically to ensure that Lightroom looked at every photo in the catalog, then let it run overnight with the library filter set to "Metadata Status = Up to date", so that it would give me the list of photos that need no metadata update. The next morning, I scrolled the grid back up to the top and let it go again, to catch any photos it missed on the first pass.

In the end, it marked over four-fifths of my catalog as out of date. This beggars belief, since I normally keep "Automatically write changes into XMP" engaged.

Then I did an experiment: I ran exiftool on one of the photos marked as still needing an update, saving the result to a text file, told Lightroom to save the metadata (⌘-S) and ran exiftool on the result, saving the output to a different text file, and diffed the two outputs, and found only timestamp and program version differences!

Observe:

5c5< File Modification Date/Time     : 2020:10:23 18:31:53-06:00---> File Modification Date/Time     : 2020:10:25 10:46:00-06:007c7< File Inode Change Date/Time     : 2020:10:23 18:31:53-06:00---> File Inode Change Date/Time     : 2020:10:25 10:46:00-06:0026c26< Instance ID                     : xmp.iid:9acec219-b6a4-4918-a592-9fc6a0ab3486---> Instance ID                     : xmp.iid:b5e76784-9638-44a1-a8b6-4e17799d003f28c28< Metadata Date                   : 2020:10:23 18:31:53-06:00---> Metadata Date                   : 2020:10:25 10:46-06:0040,41c40,41< History Instance ID             : xmp.iid:eac2ee0b-0b2d-4143-8da1-cafb424d66cc, xmp.iid:9acec219-b6a4-4918-a592-9fc6a0ab3486< History When                    : 2014:04:27 20:29:30-06:00, 2020:10:23 18:31:53-06:00---> History Instance ID             : xmp.iid:eac2ee0b-0b2d-4143-8da1-cafb424d66cc, xmp.iid:b5e76784-9638-44a1-a8b6-4e17799d003f> History When                    : 2014:04:27 20:29:30-06:00, 2020:10:25 10:46-06:00

For that particular file, the "Metadata Date" inside Lightroom is in July of 2019!

You may then wonder why the file modification time in the diff isn't in 2019, but instead two days ago. I dug into a recent pre-upgrade backup of my photos, and indeed, the prior file's mtime was in 2019. So, not only did LrC v10 decide it needed to do a bogus update to the file's metadata, it touched the file prior to actually being told it was okay to do so!

This upgrade has entirely invalidated all of my photo backups. Everything has to be backed up again, all because LrC is being stupid about touching files unnecessarily.

Surely the only defensible case for updating the application version number in the file's XMP metadata is that I've changed the photo, so now the program is properly reporting the last application to update the metadata?

I see other posts in the forum here on related topics, such as complaints that publish services are forcing a re-publish of unchanged photos. I'm posting this because I think I've diagnosed this to a deeper level than most users.

[Rikk Flohr: Photography]

A list of repeatable steps would go a long way towards solving the issue. 

I publish collections several times a week and do not encounter the issue so having some step-by-step would help immensely. 

[DVABAU]

There are no steps. Just opening a published folder in the library module is all it takes then it’s off to the races.


Regards,

Dana

<Personal Information Removed for Privacy>

[DVABAU]

Played some more, it usually immediately marks any photo I select as modified, but not always, and it seems to randomly select other nearly photos in the filmstrip, but not always, as it is also random at times. It also threw bunch into the "deleted photos to remove" bucket for the first time.

[John R Ellis]

"A list of repeatable steps would go a long way towards solving the issue."

I think one cause of the issue is the combination of older photos, perhaps older process versions, combined with catalog upgrades (see my next post). So it could be very difficult to come up with a simple recipe reproducing the bug.

My largest catalog currently has 176 published collections with 7147 photos needing republishing. Almost all of those predate LR 11, and most of those modified photos don't involve changes to non-develop metadata (keywords, etc.).  

So I think if the engineering team wants to track this down, they'll most likely have to obtain copies of one or more catalogs that have been struck by the bug and put them under the microscope.  I'll hold off on my annual ritual of republishing all those photos, in case Adobe wants to look at my catalog.

[John R Ellis]

I'm pretty about one cause of this issue:  When Adobe adds or changes the representation of develop settings in a new version of LR and a user installs the new version, the catalog entries for the users' photos aren't immediately updated with the new representation.  Rather, they're updated incrementally as LR re-renders the photos, such as when the user does a 100% zoom, invokes Library > Previews > Build 1:1 Previews, or exports the photos. 

When LR eventually does update a photo's representation, it notices that the new internal settings aren't the same as the old, and it marks the photo as Modified To Re-Publish.

This happened in LR 2015.6 when new settings were added for the Guided Upright Tool:

https://community.adobe.com/t5/lightroom-classic-bugs/p-photos-incorrectly-considered-as-quot-change...

I examined some of the photos in my current catalog that are getting incorrectly marked as Modified, and for most, the only change in the representation of develop settings was converting to the new representation of masks in LR 11.

If my educated guess about the cause of the bug is correct, then the fix is to change the incremental-update code to avoid triggering the "photo modified" condition.

[John R Ellis]

@Rikk Flohr: Photography, there's an existing (very long) bug report spanning 2016-2021 here:

https://community.adobe.com/t5/lightroom-classic-bugs/p-photos-incorrectly-considered-as-quot-change...

[DVABAU]

Then why would it happen repeatedly to the same photos, especially those that are never touched? Something else is going on, at least in my situation.

Regards,

Dana

[John R Ellis]

"Then why would it happen repeatedly to the same photos, especially those that are never touched?"

It may be that the buggy code doing the incremental update is creating invalid catalog entries for these photos that is confusing the Mark As Up-To-Date. That's the nature of bugs -- by definition, they cause the program to behave in ways we don't expect.

[tomtel49]

In reading thorugh many of these replies, some points I can "kind" of agree with (partially), BUT some important fact remain.

This has been a problem for years (since 2016 from some of the postings I see - we are now in 2022) - so it is not anything new - may worse as time grows?

This does NOT ONLY affect old collections, it affecrts new photos collections as well

It seems to be RAMDON in how it selects photos.  Unless you make hundreds of screen shots each time you see this issue, including the changes while updating, it is hard to monitor.

When updating, it will add photos and delete photos in spurts.... when they were not even listed in the origianl group "needing updates" - I included screen shots of this in the second comment of the original post.

Using the "Mark as up-to-date" only saves a person having to "re-puiblish" (i.e. time), but does NOTHIN permanent to the photo that keeps it from getting marked as NOT needing 're-publishing".

One only needs to open LRC to find this issues - that is the ONLY step needed to replicate (if you have the issue).   As an example, I opened LRC yestersay (no changes to anything - just looking) and had to use "Mark as Up-To-Date" to clean up some foldes.  EVERYTHING was as is shoudl be - no updates needed.   Later that afternoon I went back to LRC and checked the folder that I had corrected earlier in the day - and GUES WHAT - photos needed re-publishing when nothing has been done to change anything other than open the Pulishing folder and look.  See screen shot attached.

It seems "random" is selection.   Get new photos that have not displayed in group needing replushing and old ones that have been "re-published".   Also seems to be true for photos that have been "Mark as Up-To-Date".

Bottom line - getting to be a very frustrating issue if you keep on experiecing this issue - especially after a long, long period,.  If one keeps updating then a great consumer of time.

If one does nothing.... then what are the consequences?

If one updates, are changes actually being made?  If so, then why do same images need updating again later?

[John R Ellis]

There could well be multiple specific causes, e.g. the LR 2015 addition of the Guided Upright Tool, the LR 11 new masking, and perhaps other unknown causes.

[tomtel49]

Another comment today on this issue.   I when I checked everything today, I made sure all was marked as "Published" using the "Mark as up-to-date" option.  

This evening when I opened LRC and checked on somthing I am working on (not working on images in LRC), I had numerours folders (again) that had images marked as needing to be published.  I used the "Mark as up-to-date" option to get everything to display as Publised - as should have been.  

HOWEVER - and this may have some significance.... I have a folder that is made up of images from other folders (which are all sub-folders to the same group (SEE ATTACHED IMAGE).   I CANNOT use the "Mark as up-to-date" option on any folders in this folder - THIS MAY BE normal - I don't know at this time.  All the other primary folders in this grouping to include these "Recap" folders get their copies of the images in the "Publish Services" FROM the source folder.   This particular sub-folder got all the images from the "Publish Services" folder (i.e. a copy. not the original).  

I forst tried to use "Mark as up-to-date" and it did not work, so I then did each sub-folder.  After dooing that and getting all back as should be (Published), the "Recap" folder still had all the images needing to be re-published - and YES, I still could not use the "Mark as up-to-date" option.  I assume this is because they are "copies", but when all alse was updated, they should have reflected that.... SEE ATTACHED IMAGE

[DVABAU]

With a known bug as old as this, it's remarkable that Adobe hasn't built some sort of debugging tool that affected users can install that would identify exactly what the heck is going on and then just fix it.

Regards,

Dana

[dj_paige]

I will add my concern about this bug, as it has happened to me.

The last time it happened was when the catalog was upgraded from the Lightroom Classic version 10 catalog to the Lightroom Classic version 11 catalog. Suddenly, virtually every publish service I had showed all photos "Modified Photos to Re-Publish" when in fact none of the photos had been modified in any way (edits or metadata), the photos in some of these publish services had not been touched for years.

[DVABAU]

Assuming the system is actually making some sort of random change to the file which sets the flag, is there any way to lock the photos in a collection so that they can't be changed without unlocking?

Regards,

Dana

[John R Ellis]

"With a known bug as old as this, it's remarkable that Adobe hasn't built some sort of debugging tool that affected users can install that would identify exactly what the heck is going on and then just fix it."

See here for the last comment I've found here from Adobe about this (2017):

https://community.adobe.com/t5/lightroom-classic-bugs/p-photos-incorrectly-considered-as-quot-change...

[DVABAU]

That just links to more descriptions of the issue and the embedded link for a fix links back to itself. If it's the read metadata and copy metadata fix, that did nothing for me anyway.

Regards,

Dana

[John R Ellis]

"Assuming the system is actually making some sort of random change to the file which sets the flag, is there any way to lock the photos in a collection so that they can't be changed without unlocking?"

My understanding of the problem based on years of experience with LR, writing plugins, and dealing with this class of bugs: LR has a background task that checks for photos in published collections that might have changed since they were last published. The task has a comparison function that compares the current develop settings with the settings at the time the photo was published (it probably compares hashes).  The comparison function gets confused by new releases of LR that add new develop settings (e.g. Guided Upright Transform in 2015.6, masking in 11.0). 

This process appears "random" for two reasons:

- We don't know the heuristics the background task uses for scheduling its checks.

- Photos get updated incrementally to the new internal representations of develop settings as they are re-rendered, e.g. to build previews for Library (e.g. 100% zooming) or for exporting or sometimes just for scrolling through Grid view thumbnails, if the preview cache has lost their previews for some reason.

With the 2015.6 instance, you could permanently clear the "edited" flag by doing Save Metadata To File, Read Metadata From File, Mark As Up-To-Date.  But in 11.1, that only temporarily clears the edited flag. Apparently, the background process eventually gets around to rechecking the photo again and still thinks the develop settings have changed.

I wrote a plugin script that invoked publishedPhoto:setEditedFlag (false), on the hypothesis that perhaps there was a problem at the user-interface level with the Mark As Up-To-Date command.  But that script had the same problem -- the photos would get moved to Published and then sometime after get moved back to Modified.

So as of now, the only known way to permanently mark these photos as Published is to actually publish them.  (That's what I've been doing with about 7K photos about once a year, having given up after LR 2015.6 on getting Adobe to fix the bug.)

[John R Ellis]

"That just links to more descriptions of the issue"

It links to the last comment from Adobe about the 2015.6 bug:

"As usual, you, our users, have done a fantastic job of addressing this issue and bringing it to our attention.  Due to it’s obscure nature and that a solid workaround has been discovered we will not be expending any time or resources to fix it. It will only affect those users who are planning to migrate from a version <6.6 to a version >= 6.6 so that number is extremely small."

[DVABAU]

"So as of now, the only known way to permanently mark these photos as Published is to actually publish them. (That's what I've been doing with about 7K photos about once a year, having given up after LR 2015.6 on getting Adobe to fix the bug.)"

Am I missing something? Publishing them to my hard drive has no effect on the issue; it simply flags them again as needing to be republished.

[DdeGannes]

Is it possible that the flow of info e.g comments by viewers does not happen automatically? Hence the necessity to republish so it can be viewed in LrC?

[John R Ellis]

"Publishing them to my hard drive has no effect on the issue; it simply flags them again as needing to be republished."

That's a good detail to know. I publish to Flickr (Adobe plugin) and Zenfolio (Friedl plugin), and republishing is working for me at clearing the modified flag.

Though the different publish services can determine which metadata changes trigger "modified", a couple of us determined that in 2016, at least then, changes to develop settings always triggered "modified", regardless of the plugin.

But it sounds like there may be a relevant difference here between the hard-drive service and the Flickr and Zenfolio services.

[DVABAU]

Just to be very clear, all I need to do is open LR in the library module and click on a published folder and the mayhem ensues, even when I'm not in LR.

Regards,

Dana

[tomtel49]

CORRECTION TO MY ABOVE COMMENT (ONE WITH SINGLE SCREEN SHOT - REFERECING THE IMAGES BEING COPIED FROM PUBLISH SERVICES RATHER THAN SOURCE FOLDER).

This comment was wrong on my part as I made a SEVER mistake.   Going throught these so much, for some reason I thought I had already published this folder, but in reality I HAD NOT published, so the status was INDEED CORRECT - all the images NEEDED to be Published - as they had not been published.  MY MISTAKE.

The folder (Recap photo folder as seen in the screen shot) has been Published, so now I will wait and see if I get the error when random images start showing up to be "Re-Puplished"

Since the images had NOT been Published - the "Mark as up-to-date" WOULD INDEED NOT WORK since the images had NOT been published - this whole comment of this particular instance was in error on my part.

[John R Ellis]

Here's a recipe for reproducing some of the symptoms -- upgrading a LR 10.4 catalog to LR 11.1 causes published photos with the old masking applied to be incorrectly marked as Modified. But this recipe doesn't reproduce other symptoms multiple people (including me) have observed:

- Published photos without the old masking applied get marked as Modified.

- Doing Mark As Up-To-Date only works temporarily -- marked photos move back to Modified, sometimes immediately, sometimes within a day.

If Adobe wants to debug these two symptoms, they'll need a copy of my catalog.

-----------------------------------------------------------------------------------------------------------------------

1. In LR 10.4, make a new catalog, with Catalog Settings > Standard Preview Size = 1024.

2. Import two raws.

3. Add a Graduated Filter to one raw and an Adjustment Brush to the other.

4. Set up the Hard Drive publishing service to publish to a local folder, with Image Sizing set to Resize To Fit: Long Edge = 1000.

5. Drag the two raws to the publish service destination.

6. Select the publish service and do Publish.

7. Exit LR 10.4.

8. Start LR 11.1 and open the catalog created in step 1, upgrading it.

9. Select the two photos and do Library > Previews > Build 1:1 Previews.

10. Observe that the two photos now incorrectly appear in Modified Photos To Re-Publish.

11. Right-click the two photos and do Mark As Up-To-Date.  Observe that they don't move back again to Modified Photos.

[DVABAU]

Funny as I don't have a recipe other than starting up LR. Without even going into published collections, they change.

Regards,

Dana