P: Photos incorrectly considered as "changed" by republishing and smart collections

[ianmao]

[Update from John R. Ellis:

At least some instances of this bug are caused by the new develop settings added by CC 2015.6 for the Guided Upright tool (2015.6 was released 6/8/2016). When 2015.6 or later first renders a photo at 1:1 zoom that had been imported by 2015.5 or earlier, it adds those develop settings before rendering. Then it compares those develop settings with the old, notices they are different, and incorrectly marks the photo to be republished.

Here's how to work around this instance of the problem: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis...

See here for a detailed recipe to reproduce the bug, along with an analysis of the problem and suggested fix: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis... ]
-----------------------------------------------------------------------------------------------------------------

Sometime photos in my published collections began to randomly mark themselves as modified to republish.   Photos I haven't touched in years, in galleries I haven't recently changed, all of a sudden appear under 'Modified Photos to Re-Publish.'   If I even scroll through a collection, dozens of the images begin to jump up to 'Modified Photos'. I can select the photos and send them back to 'Published' with 'Mark as Up-to-Date,' but then more immediately jump up to Modified.

[weinelm]

I have the same problem. All collections mitgrate to republish without me changing anything. As others have said it seems to be when LR generates previews.  Steven Settlemyre's suggestion didn't work for me.

Windows 10, Lightroom CC 2015.7

[eschurr]

I have been experiencing the same problem for years with my Publish to Hard Drive folders.  I have over a dozen smart folders that only include photos that meet certain criteria:  the color of the photo must be Green (this indicates that its editing is final) and the capture date must be in a certain range (a particular calendar year).  This publishes all my final photos for each year into separate folders. I very rarely edit a photo i took years ago, but if i do i go to the smart folder to re-publish the photo.  Usually i find that other photos are in the "modified" category, too, and it's always baffled me but i've never worried about it too much because there were only a few thousand photos in each folder and typically only a few dozen mysteriously marked "modified" photos.

However, i recently changed my setup so i now have one smart folder that publishes ALL of my  "green" photos (30K+) to the hard drive on another PC on my network.  Now the problem is magnified and significant because the "modified" set of pictures is often thousands.  That means when i want to publish any of my new pictures (or ones i've re-edited) I have to publish literally thousands of pictures.  This takes a long time and consumes a lot of CPU, memory, and network bandwidth.

I found a post in an Adobe LR forum in  which a user thought that unchecking "remove person info" would fix this problem but it has not worked for me.

i'm happy to provide more details if it would help.

[dj_paige]

Last night, I came home from work, edited 3 photos, and then tried to create a smart collection where the criteria was "Edit Date" is Today. Much to my surprise, 91 photos were in the smart collection. Now I suppose it is remotely possible that I accidentally selected and edited these 91 non-contiguous files, but I consider this extremely unlikely. Also, it is remotely possible that Auto Sync was on when I edited these three photos, but that would show up in the history of all of these 91 photos, and there is nothing in history to indicate Auto Sync was on, or that new edits had happened since these were published via a Publish Service that left it's date-time stamp in the history.

This morning, I woke up, launched Lightroom, and created the smart collection anew. Since I had not used Lightroom overnight (I was sleeping), and Lightroom was closed overnight, I assumed that the smart collection with criterion "Edit Date" is Today would produce zero photos (as no photos had been edited Today, a few had been edited yesterday). The same 91 photos appeared in the Smart Collection.

Possible causes:

1. Bug in Lightroom
2. I was sleepwalking overnight and re-edited the same 91 photos, but don't remember doing it
   
3. Space aliens

I think it's a bug.

[Steven Settlemyre]

I am also having this issue. I created a series of new Smart Collections that simply write out JPEGs to folders on my hard drive. I "Publish" the Smart Collection using the Publish button, and randomly, after a photo is really published, it immediately appears in the "Modified Photos to Re-Publish" section. This is a huge inconvenience and an unacceptable bug that will kill many peoples' workflow.

Adobe Lightroom CC - 2015.6.1 build 1083169
Macbook Pro - Mac OSX El Capitan 10.11.6

[Califdan2]

Yeah!   It's not obvious what LR decides is a "Change".  Obviously things like star ratings, titles, captions, exportable keywords or anything you do in the develop module cause it to go to Modified, but other more subtle activities can also do it. I don't know what else causes it to "jump" as you say, but it could be something as mundane as just opening the image in the Develop module or syncing with a mobile device or even publishing in an entirely different publish service.  I'm not saying that I know any of these 'changes' cause it to jump, but something does and it's not always clear what it was.  You may want to check what other plugin's you have active as sometimes work they do in the background could trigger LR to mark the image as having been changed.  One thing the gets me a lot is if I move a KW from one parent to another or fix a spelling error on a KW or any of its parents, or add/remove a synonym.

I've also seen situations where the act of publishing an images caused it to be marked as changed and have it pop right back to modified.  I think the issue here was the inability to turn off the cross transfer of comments.  So you have an image and publish it which sends it to the service.  Then LR asks the service for comments and the service replies with "none" or returns a new comment which in turn marks the image as modified when the comment is added to the metadata.  Circular logic.

It doesn't 100% solve the problem, but some 3rd party Publish Service plugin's, such as those from jfriedl, have a panel in Publish Service set up where you can check off the things you want to be considered a "Change" or un-check them so they won't be considered a change.  For example in his Facebook Publish Service plug-in you can check or un-check any of:  caption, copyright, creator info, GPS location, instructions, job identifier, exportable Keywords, label, provider, rating, source, title, image date.  when I switched from LR's Facebook plug in to JFriedl's and unchecked all but caption, Keywords and Title it reduced the problem you describe by over 90%.

Dan

[FeedbackCommunityMember]

I found this other issue (and solution) that seems to work for this issue as well.

https://feedback.photoshop.com/photoshop_family/topics/photos-publish-directly-to-modified-photos-to...

[FeedbackCommunityMember]

I am having this problem only in the last couple of months. Simply opening an image in the Library module to view it causes it to be marked for republishing.

Im using a rMBP 13" with OSX 10.11.6
Lightroom 2015.7

[FeedbackCommunityMember]

The same thing happened to me today.  I am working on a batch of 150 to so photos; then suddenly 135 got marked as needing republishing.  They are showing the mysterious "metadata was edited externally" flag.  Now I have to update the links and proofread the whole InDesign book again.

I've checked the edit history on a sample of these and there are no edit steps on the photos since last published.  I have not changed the Publish service itself.  I have not changed any metadata either as far as I know (unless I hit something odd by accident).  No other application has written to these files - they are placed in InDesign but I am not using any other editing application.

Running Lightroom 2015.7, InDesign build 11.4.1.102, MacOS 10.12 on a high spec system with 32GB memory.

[FeedbackCommunityMember]

I have the same problem. After publishing a smart-collection the number of photos for republish is growing up continuous.
Usually it affects photos which are visible, when I scroll in film-strip or published photos.
It seems as if LR generate new thumbnails or previews.
That's very angry!

[FeedbackCommunityMember]

I'm having the same problem-

 I'll go to a publish collection and while I'm looking at the published photos, they will suddenly appear as Modified Photos to Republish without any input from me.  This can happen one at a time or all at once.  I can mark them as Up to Date and clear them from republish status and then more will appear as Modified Photos to Republish! 

[FeedbackCommunityMember]

This is at least the third post on this topic in the last couple of months. It clearly started with the 6.6.1 update. I'm only publishing to my HDD and pretty sure my hard drive / OS is not updating image metadata. Does anyone from Adobe read these forums?

[johnrellis]

"Considering that this seems to have happened 3 times now with 3 separate database updates I don't think it should be considered an "obscure incident" and will probably happen with more database updates."

I agree. Either there isn't an internal mechanism for handling the addition of develop settings properly, or more recent engineering teams don't know how to use it.

[weinelm]

I'm absolutely disgusted by Adobe's response to this issue. We are not asking for a favour. We are paying customers.

Unfortunately many people foretold that the subscription model would lead to this kind of behaviour. It seems they were right.

[Samoreen]

Look again at this thread

https://feedback.photoshop.com/photoshop_family/topics/wrong-timestamp-stored-in-lightroom-catalog-c...

Both problems are related and the bug described there is still not fixed AFAIK. As long as is is not, all related problems are not fixed either.

[Samoreen]

> I have never seen so much arrogance.

Wrong! At IBM, just before their fall.

[Samoreen]

> Due to it’s obscure nature and that a solid workaround...

I beg your pardon ? How you dare writing this ? I have spent hours investigating this bug and I have clearly explained what was wrong in the database. You have the necessary information to definitively fix this issue. If the Adobe developers are not able to correct this problem with all the information given here, they must return to school or look for another kind of job. I'm afraid, the real cause is : "We don't care. Pay your subscription and shut up.".

The write/read metadata trick is not a solid workaround because the problem comes back anyway.

This post is pure provocation. Really. I have never seen so much arrogance.

[johnrellis]

Very much agreed.

[Samoreen]

Got it! Actually, it's a side effect of the bug reported here which is lasting since version 1 :

https://feedback.photoshop.com/photoshop_family/topics/wrong-timestamp-stored-in-lightroom-catalog-c...

I just remembered that I have been recently hit by this bug and I found myself with a lot of images for which the metadata status was wrong. So I used the write/read XMP method to fix the problem. Unfortunately, LR considers this as an editing operation although no setting has been actually changed.

Conclusion : bugs that are not timely handled are never fixed by themselves. They just generate more trouble with time. We have now version 6 and the time stamp bug is here since version 1. Time to do something.

[johnrellis]

"So I guess that something happened during the past week that caused a lot of images to be incorporated. This event probably spanned over several days, otherwise I wouldn't still have thousands of images in the collection."

I repeated the recipe I posted earlier for showing that the introduction of Guided Upright in CC 2015.6 caused photos to be marked for republishing spuriously: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis... In that situation, LR is adding new develop settings to the photo for Guided Upright whenever you first zoom in on the photo, fooling publish services into thinking the photo has changed.

But the "touchTime" and "touchCount" of these photos isn't changing when that happens.  So I don't think Guided Upright / Zoom is the cause of spurious changes to Edit Date as viewed by smart collections.

[johnrellis]

I refreshed my understanding of SQLite's types. Columns are dynamically typed, and any type can be stored in any column.  The type declarations of columns are hints ("affinities") about how values should preferably be stored. Adobe_images.touchTime is missing a type declaration:

CREATE TABLE Adobe_images (...    touchTime NOT NULL DEFAULT 0)

so according to the documentation it should have type NUMERIC, where values are stored as integer or real if possible. 

DB Browser for SQLite 3.9.1 is buggy and by default stores values in "touchTime" as text rather than numeric values.  According to the documentation, that shouldn't matter because the text value will be converted to numeric in expressions that require numeric values.  But likely there's some subtlety here that causes LR to treat text values in "touchTime" representing numbers differently than numeric values.

Most likely, SQLite Expert Personal 4 is doing something similar, storing the values you enter for "touchTime" as texts rather than numbers.

So I think any further experiments need to be done with "sqlite3".

[Samoreen]

Hi John,

I did all the data manipulations described above with SQLite Expert Personal 4. So, it may have yet another behavior when manipulating the touchTime field. If I look at the Design tab for the Adobe_images table in SQLite Expert, the touchTime field has no specified type. So I guess it's a string.

Anyway, this morning, my "Last 7 days edited images" collection has much less images in it. So I guess that something happened during the past week that caused a lot of images to be incorporated. This event probably spanned over several days, otherwise I wouldn't still have thousands of images in the collection.

[johnrellis]

My curiosity got the better of me and did some tests. After first, I observed the same thing you did -- changing "touchTime" of a photo didn't change the contents of an "Edited Today" smart collection.

I was making the change to "touchTime" using Database Browser For SQLite (Mac, 3.3.1). But when I started making the change with "sqlite3", it worked exactly as expected, with the value of "touchTime" corresponding exactly to "Edit Date" in smart collections.

Then I noticed that when I changed "touchTime" with Database Browser and then dumped the database with sqlite3's .dump command, the value would be quoted, e.g.

'509059250.086592'

But when I changed "touchTime" with "sqlite3" and dumped the database with .dump, the value was not quoted, e.g.

509059250.086592

My understanding was that SQLite represented everything internally as strings, so I'm confused as to what's happening with Database Browser.  But using "sqlite3" confirms that "touchTime" really is "Edit Date" in smart collections and "lastEditTime" in the SDK API.

[johnrellis]

Mysterious.  "touchTime" maps to "lastEditTime" in the API but not "Edit Date" in smart collections. We're clearly missing something.

[Samoreen]

Maybe I made a mistake. I'll try again and let you know.

No mistake. Changing the touchTime field has no effect on the membership to the smart collection.

There's another table, AgLibraryImageChangeCounter,  that has a date field : changedAtTime. It is not stored in the Cocoa format as the other time stamps in the database but as a string.

Not many images in that table and I can't make any connection with the image I'm testing. There's an image field that is obviously there to identify the image but I can't connect it to the rootFile field in Adobe_images or to the id_local field in AgLibraryFile.

[Samoreen]

Thanks, John. I'll have a look at this plugin. However, as explained above, I already tried to set the touchTime field to a date value that should exclude the corresponding image from the smart collection. This had no effect.

Maybe I made a mistake. I'll try again and let you know.