P: Photos incorrectly considered as "changed" by republishing and smart collections

[ianmao]

[Update from John R. Ellis:

At least some instances of this bug are caused by the new develop settings added by CC 2015.6 for the Guided Upright tool (2015.6 was released 6/8/2016). When 2015.6 or later first renders a photo at 1:1 zoom that had been imported by 2015.5 or earlier, it adds those develop settings before rendering. Then it compares those develop settings with the old, notices they are different, and incorrectly marks the photo to be republished.

Here's how to work around this instance of the problem: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis...

See here for a detailed recipe to reproduce the bug, along with an analysis of the problem and suggested fix: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis... ]
-----------------------------------------------------------------------------------------------------------------

Sometime photos in my published collections began to randomly mark themselves as modified to republish.   Photos I haven't touched in years, in galleries I haven't recently changed, all of a sudden appear under 'Modified Photos to Re-Publish.'   If I even scroll through a collection, dozens of the images begin to jump up to 'Modified Photos'. I can select the photos and send them back to 'Published' with 'Mark as Up-to-Date,' but then more immediately jump up to Modified.

[johnrellis]

One more thing to try.  Download and install this plugin: https://dl.dropboxusercontent.com/u/21811200/showmetadata.1.1.zip (see here for directions on installing a similar plugin).

When you run it on a selected photo, it displays a window with all the catalog metadata that plugins can access, not all of which is written to the file by Save Metadata To File.

1. Select a photo, and do File > Plug-in Extras > Show.  Copy the displayed "before" metadata from the window.

2. Double-click to zoom and verify photo is marked for republish.

3. Do File >  Plug-in Extras > Show again and copy the displayed "after" metadata.

Post the before and after metadata here.

[flingo]

That's a cool tool!

Results:
http://messmer-online.de/Metadata-Before-After-Zoom.zip

Metadata Before and After differ a lot in the Delevop section. But if you sort the Develop section, you'll find that the only difference are the following new items in the 'After' version:

PerspectiveX = 0,
PerspectiveY = 0,
UprightCenterMode = 0,
UprightCenterNormX = 0.5,
UprightCenterNormY = 0.5,
UprightFocalLength35mm = 35,
UprightFocalMode = 0,
UprightFourSegmentsCount = 0,
UprightPreview = false,
UprightTransformCount = 6,
UprightVersion = 151388160, 

I did not even enter the Develop mode!

[johnrellis]

Interesting, these are the only differences.  The Develop settings are for the Guided Upright tool, which was introduced in CC 2015.6 / 6.6.    Here's what may be happening:

1. You zoom-in on a photo last edited in CC 2015.5 / 6.5.   

2. LR decides it needs to render the photo, and as part of that, it adds the develop settings for Guided Upright (at their default values).

3. It compares the new develop settings with the old, sees that they're different, and marks the photo for republishing.  (Though publishing services can define which metadata fields should be considered when decided if a photo should be republished, I believe that changes to develop settings always trigger a republish.)

A potential fix is to make the comparison of old and new develop settings smarter, so that a nil-valued (missing) Guided Upright setting is considered the same as the default value, and thus "no change".

Which publish services have you observed with this problem?

[flingo]

That sounds reasonable and it explain why I don't see this effect when I zoom-in photos that were last edited w/ the latest Lr version.

I using the Photo StatLr plugin (upload to Synolgy Photo Station) which was written by me.

The publishServiceProvider.metadataThatTriggersRepublish()  function returns { default = true}, i.e. any metadata change will trigger re-publish. I think, most of the Publish plugin will use that setting.

[johnrellis]

"The publishServiceProvider.metadataThatTriggersRepublish()  function returns { default = true}, i.e. any metadata change will trigger re-publish. I think, most of the Publish plugin will use that setting."

I'm not expert on implementing publish services, but I think that changes to Develop settings always trigger a republish, regardless of what that function returns.  I say that because this is what all of Friedl's publishing services show:


I think Friedl designed the LR SDK's publish-service architecture, under contract to Adobe, so he would know (only 85% confident about that).

[flingo]

That's exactly how I read the API documentation on this callback:

It allows the plug-in to specify which metadata should be considered when Lightroom determines whether an existing photo should be moved to the "Modified Photos to Re-Publish" status.

No develop settings mentioned here. That means that any publish service will suffer from this (d)ef(f)ect.

I think we now can hand this bug to Adobe on a a silver platter, can't we?

[johnrellis]

At least some instances of this bug are caused by the new develop settings added by CC 2015.6 for the Guided Upright tool. When 2015.6 or later first renders a photo at 1:1 zoom that had been imported by 2015.5 or earlier, it adds those develop settings before rendering. Then it compares those develop settings with the old, notices they are different, and marks the photo to be republished (incorrectly).

Here's a recipe for reproducing the bug:

1. In CC 2015.5, create a new catalog with Catalog Settings > File Handling > Standard Preview Size = 1440 pixels. Uncheck the option Catalog Settings > Metadata > Automatically Write Changes Into XMP.

2. Make a folder with 50 copies of this 5472 x 3648 raw file: https://dl.dropboxusercontent.com/u/21811200/DSC05456copy1.ARW

3. Import the folder into LR and wait until the standard previews have built.

4. Set up the Flickr publish service with default settings, except Resize To Fit: Width & Height = 1000 x 1000 (to make the uploads go faster).

5. Make a Flickr publish album containing all 50 pics and publish it.

6. Exit LR CC 2015.5 and start LR CC 2015.7 on the same catalog.

7. Using the plugin SDK, verify that all of the photos have the develop setting UprightVersion == nil (this is one of the settings added for Guided Upright in 2015.6):

        

This plugin is handy for examining the internal metadata and develop settings of a photo as exposed by the plugin SDK.

8. While viewing the published Flickr album in grid view, select one of the photos, go to Loupe, and then zoom 1:1.  The photo will show up in Modified Photos To Re-Publish. And often the next photo in grid view will also show up, presumably because of LR's pre-rendering policy.

9. Repeat step 8 a number of times.

10. Verify that the photos that appear in Modified Photos To Re-Publish are exactly those with the develop setting UprightVersion ~= nil:

        
        
----------------------------------------------------------------------

Examining the before and after develop settings of a photo, it's exactly the settings added for the Guided Upright tool that appear after the 1:1 zoom but not before:

PerspectiveX = 0, 
PerspectiveY = 0,
UprightCenterMode = 0, 
UprightCenterNormX = 0.5, 
UprightCenterNormY = 0.5, 
UprightFocalLength35mm = 35, 
UprightFocalMode = 0, 
UprightPreview = false, 
UprightTransformCount = 6, 
UprightVersion = 151388160, 

It appears that when
a photo imported into the catalog by CC 2015.5 is zoomed 1:1 for the first time
in CC 2015.7, LR adds the Upright settings with default values to the develop
settings, then it renders the photo.  It
then compares the new develop settings with the old, notices that the Upright
settings are different, and marks the photo for republishing.

 The fix is to treat
a nil value for those settings the same as the default settings, thus avoiding the spurious conclusion that the photo has changed.

[johnrellis]

I just posted a precise recipe for reproducing the bug and updated the original post with a link to the recipe. This is often required to get Adobe to add the bug to their bug-tracking, but it doesn't guarantee a fix.

[flingo]

Thanks a lot for your support!

[Fredy Benavides]

I found a way to fix this issue in all the photos that I have in a Lightroom Catalog. My Lightroom version is CC 2015.7

Follow these steps:

1) Select all your Pictures in your Catalog.
2) Save metadata to file
3) Read metadata from file 
4) Go to your published services Folders and select all the pictures
5) Mark all the pictures as up-to-date.

After doing these steps published folders work properly. No more random pictures appear under "Modified Photos to Re-Publish". 


Enjoy Lightroom again!

[johnrellis]

Excellent!

I verified that your steps work around the particular instance of the problem described here: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis...

For others reading this who aren't as familiar with LR commands, the steps are:

1. In the Catalog panel on the left, click on All Photographs.

2. Do Edit > Select All.

3. Do Metadata > Save Metadata To File.

4. Do Metadata > Read Metadata From File.

5. In the Publish Services panel, click on a published collection that contains Modified Photos To Re-Publish.

6 Right-click one of the select pics and do Mark As Up-To-Date.

7. Repeat steps 5-6 for every published collection affected by the bug.

[p0gue]

Well I was quite happy about this workaround until I realized that writing/reading the metadata breaks the Develop module before/after functionality. Not so good.

Writing/reading the metadata apparently resets the develop module before state to when the metadata file was read into the catalog. Previous edit states still exist and can be accessed, but 'y' and'\' no longer show the original imported image.

Images in published collections jumping to Republish was driving me insane, but now I have 1000 images that I can't use the '/' and 'y' key to check before and after. I'm not sure I understand what the read/write workaround is doing, but is there a way to correct the metadata without breaking before/after? And is there a way to repair my metadata to restore correct before/after?

[p0gue]

OK, if anyone is in the same position, there is a way to repair the before state. Right-click the 1st 'import' history step, and select 'Copy history step settings to before.' That resets the before state to the original import. Of course, it's only one image at a time, but better than nothing.

[p0gue]

Since it appears that the bug is triggered by new versions of LR viewing images published by older LR versions, and re-importing the metadata from file messes with the Develop module 'before' state (and creates many unnecessary files), I think I've found a solution that works better for me - just edit all published images in the new LR version. This will update the metadata for all published images, preventing the Republish bug from being triggered.

This is what I did. One caution... Since this procedure temporarily modifies develop settings of all of your published images, the safest plan is to back up your catalog before starting. If LR crashes while applying edits in Autosync, you could end up with unwanted changes to all your published images. 'Undo' will reverse Autosync changes, but not after LR quits. So...

1. Select all the published collections in Publish Services.
2. Select all the images in grid view.
3. Go to Develop module
4. Engage Autosync (Be very careful with this setting: With Autosync engaged, every edit you make is applied to all selected image at once)
5. Carefully make one small change to a setting not used for any published image. (I chose Grain) If you are editing many published images, you may have to wait while the edit is applied to all the images. Look for the progress bar, upper left.
6. Immediately reverse the edit you just made, leaving the setting as it was before. Wait for change to apply to all the selected images. So now all published images have been updated to the new LR version.
7. Disengage Autosync! (very important. If you forget, future edits could be accidentally applied to all your published images)
8. Return to Library module
9. Go through each published collection and mark all images as up to date.

This seems to have solved the problem for me.

Incidentally, in my case (Smugmug LR plugin) published images were jumping up to Republish without even viewing them 1:1. All I had to do was scroll through the collections in grid view and images would be marked to republish and jump to the top. I couldn't even reliably select images in the collections, because the grid was always changing so quickly. Really frustrating! Nice work by the folks posting to this thread in diagnosing the problem. Adobe really needs to fix this republish bug ASAP.

[Samoreen]

Hi,

Upon installation, Lightroom creates a set of sample smart collections. Among them the "Last 7 days edited images" and the "Last month edited images".

I'm wondering what "edited" means in that context since I can see in these collections thousands of images that I have not touched since years.

[johnrellis]

See the previous post by Paige Miller with very similar symptoms: https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis...

It's very possible that the issue with Guided Upright settings added in 2015.6, described here:  

https://feedback.photoshop.com/photoshop_family/topics/modified-to-republish-problem?topic-reply-lis...

is also causing LR to mark the photo as "edited" when it adds the new develop settings to the photo.  Or it could be a separate cause.

[Samoreen]

I further investigated this case. I looked at the data related to an image unduly added to the "Last 7 days edited images". This is what I found :

Catalog
Adobe_Images table  / touchTime  field :  2017/02/12    17:32:41
AgLibraryFile table     / modTime field :      2015/05/16     11:57:04

XMP file
xmp:ModifyDate="2015-05-16T13:57:04.46"
xmp:CreateDate="2015-05-16T13:57:04.46"
xmp:MetadataDate="2017-02-16T09:48:06+01:00"

Only 2 time stamps could justify the addition of this image to the collection. So I have successively reset these fields to an older date (this requires conversion to the Cocoa date format). This had absolutely no effect. The image was still added to the collection.

Now I'm wondering which data LR is looking at when determining the last "edit" date.

I also tried to change the "last modified" Windows time stamp of the XMP file itself, resetting it to an older date. No change. The image is still in the collection.

Really strange.

PS : Those interested in playing with the time stamps contained in the LR database will have to convert to and from the Cocoa date format. You'll find an online converter here : https://blog.paddlefish.net/?page_id=90 .

[Samoreen]

This appears to be a very old bug, indeed :

https://forums.adobe.com/thread/1399404

[johnrellis]

"Now I'm wondering which data LR is looking at when determining the last "edit" date. "

The SDK API provides access to a per-photo field "lastEditTime".  This appears to correspond to the Adobe-images.touchTime field in the database.  To test that, I took a several values of "lastEditTime" and searched for them in a text SQL dump of the database, and they showed up in the "touchTime" field.

You might find this plugin handy for seeing all the photo fields that are available to plugins: https://dl.dropboxusercontent.com/u/21811200/showmetadata.1.1.zip

[Samoreen]

Thanks, John. I'll have a look at this plugin. However, as explained above, I already tried to set the touchTime field to a date value that should exclude the corresponding image from the smart collection. This had no effect.

Maybe I made a mistake. I'll try again and let you know.

[Samoreen]

Maybe I made a mistake. I'll try again and let you know.

No mistake. Changing the touchTime field has no effect on the membership to the smart collection.

There's another table, AgLibraryImageChangeCounter,  that has a date field : changedAtTime. It is not stored in the Cocoa format as the other time stamps in the database but as a string.

Not many images in that table and I can't make any connection with the image I'm testing. There's an image field that is obviously there to identify the image but I can't connect it to the rootFile field in Adobe_images or to the id_local field in AgLibraryFile.

[johnrellis]

Mysterious.  "touchTime" maps to "lastEditTime" in the API but not "Edit Date" in smart collections. We're clearly missing something.

[johnrellis]

My curiosity got the better of me and did some tests. After first, I observed the same thing you did -- changing "touchTime" of a photo didn't change the contents of an "Edited Today" smart collection.

I was making the change to "touchTime" using Database Browser For SQLite (Mac, 3.3.1). But when I started making the change with "sqlite3", it worked exactly as expected, with the value of "touchTime" corresponding exactly to "Edit Date" in smart collections.

Then I noticed that when I changed "touchTime" with Database Browser and then dumped the database with sqlite3's .dump command, the value would be quoted, e.g.

'509059250.086592'

But when I changed "touchTime" with "sqlite3" and dumped the database with .dump, the value was not quoted, e.g.

509059250.086592

My understanding was that SQLite represented everything internally as strings, so I'm confused as to what's happening with Database Browser.  But using "sqlite3" confirms that "touchTime" really is "Edit Date" in smart collections and "lastEditTime" in the SDK API.

[Samoreen]

Hi John,

I did all the data manipulations described above with SQLite Expert Personal 4. So, it may have yet another behavior when manipulating the touchTime field. If I look at the Design tab for the Adobe_images table in SQLite Expert, the touchTime field has no specified type. So I guess it's a string.

Anyway, this morning, my "Last 7 days edited images" collection has much less images in it. So I guess that something happened during the past week that caused a lot of images to be incorporated. This event probably spanned over several days, otherwise I wouldn't still have thousands of images in the collection.

[johnrellis]

I refreshed my understanding of SQLite's types. Columns are dynamically typed, and any type can be stored in any column.  The type declarations of columns are hints ("affinities") about how values should preferably be stored. Adobe_images.touchTime is missing a type declaration:

CREATE TABLE Adobe_images (...    touchTime NOT NULL DEFAULT 0)

so according to the documentation it should have type NUMERIC, where values are stored as integer or real if possible. 

DB Browser for SQLite 3.9.1 is buggy and by default stores values in "touchTime" as text rather than numeric values.  According to the documentation, that shouldn't matter because the text value will be converted to numeric in expressions that require numeric values.  But likely there's some subtlety here that causes LR to treat text values in "touchTime" representing numbers differently than numeric values.

Most likely, SQLite Expert Personal 4 is doing something similar, storing the values you enter for "touchTime" as texts rather than numbers.

So I think any further experiments need to be done with "sqlite3".