Lightroom creates duplicates when republishing to Flickr

[Benito Aramando]

When you make any changes to a photo that is included in a set that has been published to Flickr it moves from the "Published Photos" section to the "Modified Photos to Republish" section. When the set is published these are supposed to update the existing image on Flickr (as per this page: How to publish photos from Photoshop Lightroom Classic CC to Adobe Stock and social media), but Lightroom just re-uploads it, creating a duplicate on Flickr (albeit with the updated version of the image). I have just updated to the latest version of LR Classic, 7.2, to no avail.

[dj_paige · Accepted solution]

When the set is published these are supposed to update the existing image on Flickr

I don't really see where it says that. But you have to have a Flickr Pro account to be allowed to replace existing photos.

[Benito Aramando]

It isn't explicit, but under the heading "Upload to Flickr", part 2, it says "If prompted, click Replace to update published photos with newer versions", which clearly implies that photos which are being re-published should replace their existing counterparts on Flickr rather than being added again.

However, I think you're right about the Pro account level requirement. This is not well-documented on Flickr's end, and you can replace photos on a free account via the website which is why I thought this should be working, but the API does appear to restrict this to Pro accounts; error code 1 for the photo replacement API is "Not a Pro account": https://www.flickr.com/services/api/replace.api.html

[DClark064]

This is marked as the "Correct answer" but I think this answer is not correct. I have a PRO account and I am finding that edits to photos that get re-published create duplicates, and triplicates, etc, etc.  It's a mess!  Flickr support says they are working with Adobe to correct this problem, and they say the Flickr publish module is Adobe's responsibility.   Why is this not a BUG instead of an item in Discussions?

[dj_paige]

Are you using the Lightroom Classic built-in Flickr publisher? I use the Jeffrey Friedl Lightroom plug-in, and I don't have problems with duplicates being created; in fact it worked correctly yesterday. Lightroom Classic 13.2, Windows 11.

Please provide similar information for your situation.

[DClark064]

Yes, I am using the Adobe Flickr plugin.  Flickr support has verified that the Adobe Flickr publish plugin has problems and Flickr engineers are attempting to work with Adobe.  I say attempting since this seems to be a problem that has been around for a long time and there is no commitment to a fix anytime soon.  I have been trying to find any indication that Adobe acknowledges there is a BUG that needs to be fixed.  I can find reports of duplication of photos that go back a few years but no indication they recognize something is wrong and I see no indication any action is being taken.  For example, the post that started this thread is from 2018!

[alexr15]

This issue has bothered me for YEARS and I've always worked around it by using the "Mark as Up-to-date" action from the plugin menu. Finally I got fed up and spent a few hours troubleshooting this over a period of a few days, thinking there must be some specific reason this happens, considering it doesn't happen all the time.

In my experiments I observed that the problem seems to be specific to individual photos, and it only seems to happen to a small percentage of photos of specific file types:

- JPEGS from 2010-era cameras, including iPhone 4s, Nikon D7000, Canon S100

- ORF (Olympus raw files) from E-PM2, but not from E-M1

I also noticed that including specific stuff in the Title (long titles or special characters) can either cause or trigger the problem. So I fixed that first before proceeding to additional experiments.

In the end I was able to de-duplicate a lot of images and re-point the plugin to the original upload by manually updating the AgRemotePhoto table in the LRCAT file (using sqlite DB Browser).

[alexr15]

There seems to be something very intrinsic to the image file itself that causes it to have this problem.

In one experiment, I took one of the problematic ORF files, removed it from the catalog while keeping the published photo on Flickr, re-imported it as a DNG, then manipulated the AgRemotePhoto table to re-point the newly published (DNG) photo to the old instance on Flickr. This worked successfully -- subsequent edits to the photo (DNG version) replace it correctly.  

In a second experiment, I took another problematic ORF file, removed it from the catalog while keeping the published photo on Flickr, removed the XMP sidecar, re-imported as an ORF (not DNG!), then did all the same steps around manipulation of AgRemotePhoto. The result was unsuccessful: the first edit/republish results in a duplicate photo getting uploaded to Flickr.

[alexr15]

Final update...

I finally got to the bottom of this issue!

Many thanks to Adobe for publishing the source code of an older version of the Flickr plug-in, as part of the SDK bundle. This source code enabled a detailed investigation.

It turns out, observations about file types and old cameras were coincidental. The real issue is special characters in Flickr titles. I had mentioned my suspicions of special characters in my previous post, but dismissed it as my testing did not initially confirm those suspicions. It turns out, special characters are the cause of the problem, but only when entered directly into Flickr.

Here are the details of my investigation...

Bug mechanism:

1. In order for the Lightroom-Flickr plugin to decide  whether to replace or upload a fresh copy of a modified photo, it first retrieves all photos from the specified photoset on Flickr. It scans the set 500 photos at a time (a "page"), in reverse chronological order.

2. There is a section of the plugin code where the Flickr API Response is first transformed using XSLT, and then parsed using Lua syntax. The XSLT does not have any code to escape special characters. As a result, double-quotes in Flickr photo titles break the parsing. 
2. Due to improper error handling, plugin then ignores the affected photo and all photos in same 500-item result page. 

3. Lightroom uploads new duplicates instead of updating existing photos. 

Example: Photo 1234 has double-quote in title, photos 1000-1500 not recognized for updates. Any pending updates in this range uploaded as new photos. This doesn't mean 500 photos will be re-uploaded. Only the set intersection of these 500 with the set of photos that were marked for re-publishing.

If you are Adobe reading this:
The issue is in function FlickrAPI.listPhotosFromPhotoset in FlickrAPI.lua.
Easy Fix: Add quote escaping in XSLT:

Original code: title = "<xsl:value-of select="@title"/>",
Fixed code: title = "<xsl:value-of select="replace(@title, '&quot;', '\&quot;')"/>",

If you are an end-user just looking to revert the duplicate uploads and prevent the problem from re-ocurring:
1. Go to your Flickr account and check the Title field of EVERY PHOTO in your LR-managed photo set. Remove any double-quotes found in the Title. It is critical that you check every single photo in the set.
2. (Optional) Reassociate your plugin collection photos to the original photo on Flickr: Open your LRCAT file in Sqlite DB Browser, go to the AgRemotePhoto table. Find the matching photos and replace the duplicative Flickr ID with the original Flickr ID in both the remoteId and the url columns. Then manually delete the duplicate from Flickr.

3. Republish from Lightroom.

Note: Latest version of the plugin replaces the quote characters with dashes on publish, so the problem can only happen again if you directly edit the Flickr photos to add titles that contain quotes.

[DClark064]

Wow!!  Amazing investigation. 

Is the only character that causes a problem double quotes (i.e. ")?

I searched through more than a thousand photos and found only 1 with double quotes.  I am sure I did not enter the double quotes by editing the title on the Flickr site.  That title was entered in LrC and uploaded to Flickr using the plugin.  Would the presence of that file cause the problems I have been experiencing?

Thanks for your work on this.  I will watch for any future probems to see if double quotes in the title is causing the problem.  It seems to me that someone at Adobe should have done what you have done years ago.

[alexr15]

Thanks for your response.
I wasn't sure whether my post was really a "note to future self" or anyone else would read it 🙂

To your questions:

• I suspect that other special characters can also cause the problem, but I only tested with double-quotes. In theory, any character that is not valid unescaped inside a Lua string, will cause the same issue.
• One photo is enough to cause up to 500 other photos to become duplicated on each upload batch. I also only had one in the set at the time I discovered this.
• According to my testing, the current version of the plugin will not send a double-quote into the title, HOWEVER, my plugin configuration has "Set Flickr Title Using: Filename" with "Rename To: Custom Settings" where I interpolate the Title along with other info. I suspect, but I have not tested it, that if you set "Set Flickr Title Using: IPTC Title", that the plugin may still obliviously send the double-quotes into Flickr. Is this your configuration?

Thanks

[DClark064]

"any character that is not valid unescaped inside a Lua string", what characters are these?

My plugin is set to use the IPTC Title, and when updating to replace the existing title.

[alexr15]

"any character that is not valid unescaped inside a Lua string", what characters are these?

Potentially non-printables such as tab, newline, etc, and malformed utf-8 encodings. I have not tested any of these.

I also tested backslash (\) but it does not appear to cause problems, which is surprising.

My plugin is set to use the IPTC Title, and when updating to replace the existing title.

I just did a quick test with your settings. Indeed, the double-quote is transmitted intact to Flickr!

So it is pretty easy now to replicate the duplication problem starting with an empty set, without leaving Lightroom:

1. Create a new Album in the Flickr Publish Service

2. Add a single photo to the Album

3. Set the title to any text containing double-quotes (")

4. Hit Publish

5. Tweak the photo and Hit Publish again.

6. Check Flickr account, you're guaranteed to see a duplicate.

7. Repeat Step 5, now you have 2 duplicates. Rinse and repeat for as many duplicates as desired.

If the Album has less than 500 photos, any other photos will also get duplicated, even if they don't contain a double-quote in the title.

[alexr15]

Just to clarify the last statement: If the Album has less than 500 photos, any other photos will also get duplicated every time they are updated, even if they don't contain a double-quote in the title. If the Album has more than 500 photos, other photos will get duplicated every time they are updated and they fall into the same 500-item page as the problematic photo. Even if the problematic photo itself is not Updated! 

[DClark064]

A double quote prints.

I will use the method you descrive to experiment with characters.  Thanks.

I will also experiment with your "optional" step to reassociate the photos.  That will require some learning with the SQLite browser.

[alexr15]

An unescaped double quote is not valid in a double-quoted Lua string - that is the root of the problem.

Any other stuff mentioned such as non-printables, malformed utf-8, etc, are just hypotheticals which I have not tested.

[alexr15]

I also tested backslash (\) but it does not appear to cause problems, which is surprising.

Alas, the backslash is not completely safe. It'll cause the same cascading failure, if it's the last character in the title.

I realized this while thinking about the plugin code, and confirmed it by testing with a fresh set, a single photo, and "Set Flickr Title Using: IPTC Title."

There could be additional characters that are problematic. At this point my patience has run out, so I'm not going to try to identify all the different failure triggers. I just wrote a script to nuke the title on every photo in the LR-managed Flickr Album, and I changed my plugin settings to overwrite the title with a simple timestamp on future exports.

I'm finally able to sync and re-sync a set of 2,500 photos without triggering this problem.

[stewartr43936066]

I have now started to get this issue. I have a Pro account and have been able to update images with no issues but I am now getting dupes every time I republish. I don't think it started right after the Mojave upgrade but I do't recall it before wither. Anyone have any thoughts as I am at a standstill ATM.

Additional - I have just been back to take  look at some earlier uploads and it was happening before the Mojave update. It seems that any image that is updated creates a new version.

S

[Rogerhollingsworth]

I have the same issue.

[b9dazzle@gmail.com]

I am getting the photos to be republished in Flickr.  There are 55 photos I have not touched in years.  I have a pro account and contacted them, they said it was a bug in adobe.   I talked to adobe tech support  they told tme to edit settings then leave as is.   so this is the second time in a couple months,  I am running the current version of  Lightroom classic and Windows did an update right before this happened.

[alexr15]

In the process of investigating and fixing this issue, I created several python scripts.

I've shared them in a github repo:

https://github.com/photography-tools/lightroom-flickr-duplication-issue