• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
0

Smart previews for cloud sync recreated after metadata edits?

Explorer ,
May 31, 2021 May 31, 2021

Copy link to clipboard

Copied

Hi,

(Lightroom Classic 10.2 on Windows 10 19043)

I imported 3000 raws in Lightroom Classic, smart previews for cloud sync were generated in the background as expected.

Days later I added metadata info (Exif location, keyword... not written to the file) and Lightroom has been working (CPU and upload) like crazy.

 

Therefore my question: does adding metadata trigger the re-generation of smart previews for cloud sync? (which should be avoided in my opinion)

 

Thanks,

Olivier

TOPICS
Windows

Views

1.4K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
May 31, 2021 May 31, 2021

Copy link to clipboard

Copied

It shouldn't, because these extra data are not synced.

 

-- Johan W. Elzenga

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
May 31, 2021 May 31, 2021

Copy link to clipboard

Copied

More testing:
- It's Exif edition that sparks this (not keywords)

- Lightroom will use the CPU, access/read the raw files (on my D drive), read/write its' catalog (on C drive) and resync those images on the cloud.

 

To put in perspective: when adding one Exif word on 3000 images, Lightroom will read several gigabytes on drive, use 3 minutes of CPU and upload hundreds of megabytes to the cloud (see attached performance monitoring). I don't see why... and it does slow down Lightroom quite a bit.


PS: and "Automatically write changes to XMP" is disabled.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
May 31, 2021 May 31, 2021

Copy link to clipboard

Copied

"It's Exif edition that sparks this (not keywords)"

 

Can you clarify what you mean by "Exif edition"?  In English, there is no field in the Metadata EXIF panel called "edition".  A screenshot would help, especially if you're using LR in a language other than English.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
May 31, 2021 May 31, 2021

Copy link to clipboard

Copied

Sorry. I mean editing/modifying IPTC fields (I tried Location ones and Title). See attached screenshot (in French).

Lightroom_IPTC.png

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
May 31, 2021 May 31, 2021

Copy link to clipboard

Copied

Your performance monitor only shows a few megabytes uploaded through your wifi card but a ton of disk access. Are you sure it is uploading much? I have no clue why it would do so much disk access for just a title or caption update if you haven't enabled the automatic writing of xmp. 

 

P.S. Do you really need smart previews? They are only useful if you often edit while not having access to your originals. For any other scenario, they are just dead weight and take up space. It is usually best to never create smart previews except if you do some offline editing. If you sync images to the cloud the needed smart previews are created on the fly only once and you never need them again so for syncing to the cloud you also do not need to generate smart previews.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
May 31, 2021 May 31, 2021

Copy link to clipboard

Copied

I don't activate/use Smart Previews, but as you mentioned they are created on the fly for synchronizing Collections (which I use on my iPad).

 

The perf graph is rather long. The strong CPU activity last about 6 minutes. The network upload is therefore about 100 seconds at 2MB/s... so it seems those "on the fly" smart previews are recreated and uploaded again after an IPTC modification 😕

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
May 31, 2021 May 31, 2021

Copy link to clipboard

Copied

100 seconds at 2 MB/s is only 200 MB. That is not enough for smart previews to be reuploaded for 3000 images. A smart preview is typically a few MB for each image. I am pretty sure Classic never reuploads any smart previews after the first sync of images. It will sync all kinds of other stuff like custom camera profiles, presets, etc. but not the actual image after the first sync. All subsequent syncing is done by xml. It is likely not very optimized as Adobe hasn't worked on Classic syncing for a long time as they declared a few years ago that they would not add any cloud syncing features to Classic which apart from some minor bug fixes is all they did.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
May 31, 2021 May 31, 2021

Copy link to clipboard

Copied

"100 seconds at 2 MB/s is only 200 MB. That is not enough for smart previews to be reuploaded for 3000 images."

 

That's about 67 KB / photo. I'm reasonably familiar with the HTTP REST protocol used for syncing with the LR Cloud, and that sounds plausible for syncing a change to metadata.  As Jao said, it's not at all "optimized".

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
May 31, 2021 May 31, 2021

Copy link to clipboard

Copied

I was curious about disk activity when syncing, so I used the Windows Sysinternals Process Monitor to monitor LR 10.2's activity. When initially syncing raws, LR created temp files of 1 - 2 MBs each containing the smart previews.  But when I updated the titles in LR, those temp files for the smart previews weren't created again.  LR did write about 3 MB to the local sync database when I changed the titles of 17 synced raws (about 175 KB / photo).

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jun 01, 2021 Jun 01, 2021

Copy link to clipboard

Copied

Another test, this time with 2091 JPEG from my old Canon 300D. As in the previous one, Lightroom showed a notice about updating Metadata (about 30s) then proceeded for a long synchronization (about 8 minutes) as you can see here:

Lightroom_IPTC_Perf2_notes.png

I noticed this graph was impacted by other programs (ex: BackBlaze backup after files changes, browser...) but I also took notes of Lightroom specific statistics which gave me this:
- CPU time : 6mn 26s
- Disk write = 2.18GB | read = 178MB

- Network receive = 24MB | send = 7MB

The issue seems to be the writing to disk (and CPU usage)... therefore I used Resource Monitor to see what was happening and found that Lightroom was writing all this to the folder [my catalog name] Sync.lrdata.

 

Any idea about that? Or some confirmation?
I'm also starting to think wether my Jeffrey Friedl's plugin "Smart-Collection Sync" could interfere with all this...

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jun 01, 2021 Jun 01, 2021

Copy link to clipboard

Copied

Update:
I deactivated all plugins, restarted Lightroom... and got the same results: plugins are not the issue.

The two files with most disk writing activity in Sync.lrdata folder have .docstore and .docstore-wal extensions.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jun 01, 2021 Jun 01, 2021

Copy link to clipboard

Copied

"Disk write = 2.18GB ... Sync.lrdata"

 

That about 726 KB / photo written to the local sync database, whereas in my smaller experiment, I observed 1/4 as much, 175 KB / photo. They are SQLite databases, but I haven't poked into them to see why so many bytes are getting written into them.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jun 02, 2021 Jun 02, 2021

Copy link to clipboard

Copied

I monitored again my first test: an IPTC modification (Title) on 3120 raws (Sony A7RIII) wich sparked Metadata update and 3120 files cloud sync:

Lightroom_IPTC-Perf3.png

Stats for Lightroom only:

- CPU time = 14mn 14s

- Disk write = 3.3GB (1083kB per file, on Sync.lrdata folder)

- Disk read = 2.1GB (690kB per file, mostly on raw files)

- Network receive = 32MB

- Network send = 8MB

 

Disk read is very different from my JPEG test. Lightroom is accessing all original files (on D drive), probably to read/check metadata, and maybe it needs to read larger chunks with raw files.

Disk write is very high in both cases. Could Lightroom be writing all Metadata and creating some thumbnail/preview for each file again in its' DB sync?

I don't see the need for reading/checking metadata from original files (it's already in Lightroom DBs) nor for recreating 1MB of metadata per file for a few octet IPTC single modification.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jun 02, 2021 Jun 02, 2021

Copy link to clipboard

Copied

It does look quite excessive. I poked in the local sync database when implementing my Any Comment plugin some years ago, and my impression at the time was that it was designed for a much more featureful cloud sync than actually got implemented.  The sync database appears to maintain a complete copy of all the metadata of each synced photo.  That could conceivably be as much as a 1 MB, if you have huge amounts of local-adjustment brush strokes, but I'd be surprised if your photos all had that much brushing?

 

You could post a bug report on the official Adobe feedback forum:

https://feedback.photoshop.com/photoshop_family/categories/photoshop_family_photoshop_lightroom 

 

But as Jao indicated above, Adobe explicitly communicated that they won't be adding any more features to LR Classic - LR Cloud sync, and they've been very slow in fixing bugs (perhaps slower than they are with other bugs).   So it might do a little good to post a bug report, but it might also be mostly ignored.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Jun 02, 2021 Jun 02, 2021

Copy link to clipboard

Copied

Are you using any custom profiles in your database? There is a long standing bug with custom profiles with LUTs embedded get copied over and over into the catalog and perhaps also into the sync data. It makes catalogs balloon to crazy sizes but yeah I would not be surprised if this is just a very inefficient implementation and it just constantly writes and rewrites the metadata every time.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jun 02, 2021 Jun 02, 2021

Copy link to clipboard

Copied

Images were imported with Lightroom's Auto tone, a yellow label, IPTC copyright and the location. Very simple indeed. No local adjustment, no custom profile.

 

PS: even deleting a single image seems to trigger a large metadata writing, taking 5 seconds of CPU instead of being instant. Quite frustrating...

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jun 02, 2021 Jun 02, 2021

Copy link to clipboard

Copied

Bug reported here: https://feedback.photoshop.com/conversations/lightroom-classic/60b7cb7b8f793f7cedf81fd8

Thanks for the advice and all answers provided here 🙂

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jun 02, 2021 Jun 02, 2021

Copy link to clipboard

Copied

Check the files in WIndows Explorer ans see if XMP files have been created. You mentioned 'Automatically Wriyte Changes to XMP' is not checked, but a Preferences file corruption could enable it. If you see XMP files try resetting LrC Preferences file. Place a copy of the Preferences file on your desktop so it can be easily restored if of no help.

 

https://www.lightroomqueen.com/how-do-i-reset-lightrooms-preferences/

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jun 02, 2021 Jun 02, 2021

Copy link to clipboard

Copied

No XMP file found here 😕

Should I delete sync.lrdata folder completely to restore... errr.... hope? 😉

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jun 02, 2021 Jun 02, 2021

Copy link to clipboard

Copied

You can try that, but I'm not sure it will change anything. Go to LrC Preferences> Lightroom Sync and click on 'Delete All Synced Data.' It will take some time to resync the 3,000 image files.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jun 05, 2021 Jun 05, 2021

Copy link to clipboard

Copied

Thanks for the tip.

I have about 40.000 images in sync and a not-so-fast network at home. I can't afford to delete all those from the cloud and reupload all of them again.

Is there a way to just make Lightroom Classic recreate again the cache/DB on my PC without losing what is already in the cloud?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Jun 05, 2021 Jun 05, 2021

Copy link to clipboard

Copied

quote

Is there a way to just make Lightroom Classic recreate again the cache/DB on my PC without losing what is already in the cloud?

 

On the LR Sync tab of Preferences, hold down the Alt/Opt key and a Rebuild Sync Data button appears. Or just delete the Sync.lrdata file that's in the same folder as your catalogue.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jun 06, 2021 Jun 06, 2021

Copy link to clipboard

Copied

So I rebuilt the local sync data, it took quite some time... and I still have the same issue.

 

I can't reinstall Lightroom or reset the preferences right now... but I'll see if I can do it later on (thanks todd@craftedreality for the explanation).

@john beardsworth : originals were not changed (nor their EXIF) and they are the same unmodified raws.

 

I sum-up my observations:
- Lightroom insists on reading original files' metadata for every IPTC change although unnecessary (ie: already in Lightroom's DB).
- Lightroom writes 1000x data than necessary for every IPTC change to update its' own DB for sync.

=> What should be a few megabytes I/O on disk becomes several gigabytes (and a lot of useless CPU time)... 😕

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jun 14, 2021 Jun 14, 2021

Copy link to clipboard

Copied

LATEST

I updated to Lightroom 10.3 (June 2021 release), read "With this release, you will experience performance improvements while selecting and updating metadata for multiple images." with some excitation... did the same test and got the same results: Lightroom Classic brought to its' knees for a simple IPTC change (ie: 1000 times more disk usage than necessary).

I guess I'll have to reinstall LrC and reset prefs to check this out.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines