Data Usage over LAN/Internet/VPN Help

[Dirk5D13]

My 35000 photos are stored on a NAS, the catalog and the previews on the local Laptop. The data consumption over Network when scrolling through the photos is enormous: each photo is literally downloaded. It doesn't matter whether I'm on the LAN at home or dialing in from outside via VPN. With RAW, 25MB is added with each view and with jpgs 5MB correspondingly.
This quickly adds up to several GB and you soon reach the mobilephone data limit when you access them via mobile hotspot.
The preview folder contains about 35000 files, the catalog also contains about 35000 photos, 5GB are reserved for the camera-raw cache, but contains only 120 files.
LR uses 1,8GB RAM at the moment with 16GB installed.
By the way, I'm only talking about scrolling through the library module in grid view, without developing or using the magnifying glass.
How can I limit this data consumption? Thank you in advance!

[DS256]

FWIW, IMHO, you are asking LrC to do something it wasn't designed for.

That being said, here are a couple of thoughts:

• Check to see what you PREVIEW setting is. A lower quality preview may help with bandwith consumption.
• How are you are running LrC from a mobilephone? You may want to investigate Adobe mobile solutions which are designed to run over the Internet.
• If could be tricky, but investigate local cache software solutions that would at least prevent new downloads 

[JohanElzenga]

Generating smart previews may help.

[Dirk5D13]

Thanks for your reply!

I just want to make it clear that it's not about Adobe Cloud or synced folders or LrC on mobile. The mobile phone is only used for internet access, since the problem with the data volumes first appeared.

It's about the same laptop with the same LrC installation and catalog.
As far as the preview size is concerned, you can see how the data usage over the network increases by 25MB with each view of a 25MB raw photo. However, the corresponding preview only has 5MB. But why, that's exactly the question.
Maybe I should have all the previews recreated overnight.

[ouiouiphoto]

Hello

If you have preview of the size of your display, 1:1 preview and Smart preview, the network consumption is supposed to be low. In fact you can even stop you NAS and it will work. My best guess is probably because you don't have the good previews

[Jao vdL]

This is very similar to my setup. If you are scrolling through images in library, the original raw will get fetched every time only if the requested preview resolution is bigger than what you are displaying. So what matters enormously for this type of scrolling is the standard preview size you have set up. If it is too small (and you don't have 1:1 previews or are using embedded previews) yeah, it will hit your raw files every time. If the preview size is set correctly, it won't hit the raw files (I tested this extensively). So check this in the catalog settings menu item - File handling tab. Usually Auto is the right setting there but not necessary if you have dual displays as it might choose the lowest resolution screen for the auto setting. For my Mac Book Pro with retina XDR screen Auto defaults to 3456 pixels which should indeed be enough for the 3456x2234 screen. This does not help if you like to zoom in 1:1 when just going through images in library. If you do and don't have 1:1 previews, it will hit the raw file. 

Because of the above, another really good trick to avoid hitting the raw files as much as possible is to import using the "embedded and sidecar" preview setting. This will create the initial preview database for newly imported files from their embedded jpeg previews and is lightning fast. This works especially well when you combine this with a raw default setting (preferences->Presets) of "Camera Settings". It will enormously speed up workflow and prevent your computer from needing to access the raw files when just culling through even if you are in the habit of zooming 1:1 when doing culling. 

Another thing: Smart previews do not matter for this type of scrolling (they are never used for this except if your originals are offline) and are actually detrimental to speed on the modern high resolution screen as they are simply too small resolution (they are 2560 pixels on the long end and often too small nowadays) and they just fill up your SSD without any benefit and therefore can slow down performance by making your internal SSD too full. Smart previews are highly counteradviced nowadays except if you often want to edit without your originals available. 

[ouiouiphoto]

Hello Jao

I will recheck but If you select  "embedded and sidecar" it is not supposed to create previews. It will use the one embedded in the RAW and then it will hit the RAW more frequently. That's why you have in the preference a setup to replace the Embended preview by the real ones if your computer is inactive 

[Jao vdL]

If you do embedded it will only hit the raw when you go into develop and change settings. At that point it will then create a standard preview. It will not hit the raw (except for loading the original jpeg preview from it) until then. So this will prevent you from hitting the raw for as long as you refrain from changing any develop settings. If like me, you only develop 25% of images, this prevents a lot of raw file access because of this. It really is by far the most efficient way to go through lots of files. Also if you have the "camera setting" raw default set, the embedded preview is extremely close to the Lightroom rendering anyway so no point to have Lightroom replace it.

[johnrellis]

"If you do embedded it will only hit the raw when you go into develop and change settings. "

Only if you've disabled Preferences > General > Replaced Embedded Previews With Standard Previews During Idle Time, which I'm sure you've done.

[Jao vdL]

Isn't it disabled by default? I never changed that setting that I can remember and it really should be disabled by default as it defeats the purpose of using embedded previews to replace the 1:1 previews you get for free from the camera by lower resolution previews that take cycles to generate. But yeah it should be pointed out for this workflow to work this setting needs to be disabled.

[Dirk5D13]

So, I've now recreated the previews of a large part at the recommended resolution. Unfortunately, Lightroom obviously doesn't use these local previews, but always downloads the original ones over the network. If the original is 2MB in size, 2 MB will be downloaded when clicked, if the original is 5 or 25 MB in size, 5 or 25 MB will be downloaded.

I don't make any changes to e.g. the metadata and I don't perform any file operations. Just looking at the images in the library module generates the traffic.

By the way, if the NAS with the Originals is disconnected, LR tries to reload Data (Screenshot attached).

What surprises me is that the documentation recommends >2048 px for a 1920px monitor. I'm going to render the default previews at the new resolution again.

[Jao vdL]

When it is trying to load the image over the network is the preview black like your screenshot 5.png seems to suggest? That seems to imply there is no correctly sized preview for this image somehow and that is why it is fetching (or trying to fetch) the original over the network. It is possible that your preview database is corrupt somehow as it definitely shouldn't do that when the preview is available. You might want to optimize the catalog and delete the Lightroom preferences (often helps resolve weird behavior like this). 

[johnrellis]

@Dirk5D13: "I've now recreated the previews of a large part at the recommended resolution. Unfortunately, Lightroom obviously doesn't use these local previews, but always downloads the original ones over the network"

@Jao vdL: "It is possible that your preview database is corrupt somehow as it definitely shouldn't do that when the preview is available. You might want to optimize the catalog and delete the Lightroom preferences (often helps resolve weird behavior like this). "

I agree that this smells like the previews database is corrupted. Assuming resetting preferences doesn't fix the issue, I recommend cutting to the chase and simply deleting the entire preview database, following these instructions:

https://helpx.adobe.com/lightroom-classic/kb/lightroom-gives-error-preview-cache.html

(An easier way to ensure you get to the correct catalog folder is to do Catalog Settings > General > Show.)

After restarting LR, enable the option Preferences > Performance > Generate Previews In Parallel, select all photos, and do Library > Previews > Build Standard-Sized Previews.  The rebuilding should go  at least one photo/second, so at 35,000 photos, should complete overnight.

[Dirk5D13]

Thanks again for the support!
Now I've created all the previews and the database from scratch. With a direct LAN connection instead of WLAN and parallel generation it only took 165min, so about 3 photos/s. Unfortunately, it didn't make any difference: The traffic when simply viewing some folders quickly reached 1GB.

Attached is a screenshot of Lightroom's network traffic.

I'll take care of all these unclear metadata statuses first (Screenshot).
Two weeks ago I switched from V6 to the subscription version and I now suspect that not everything went so smoothly after all...

[Jao vdL]

Have you tried resetting Lightroom Classic's preferences? https://helpx.adobe.com/lightroom-classic/help/setting-preferences-lightroom.html

[Dirk5D13]

Just noticed that .xmp's were created for 8851 elements, although it was never selected that way... It's getting crazy

[johnrellis]

Building on Jao's comment, I can think of two more steps to narrow down the problem:

1. Reset preferences if you haven't already.

2. Make a new test catalog with a couple thousand of the photos added into it, and see if the problem occurs with the test catalog.

[Dirk5D13]

Thanks for all the hints... I could have come up with the idea of creating a new catalogue 👍

So now I have created a new catalog with 3000 photos and new previews. Unfortunately, even that didn't help: Attached is a screenshot of the traffic while scrolling through. I'm at a loss, why doesn't Lightroom use the local previews, but the originals from the NAS over the net?

[Jao vdL]

How does it behave when you don't have the NAS mounted and you go back and forth in Library? Does it not show anything? Throw an error? Show very low res previews? It should be very fast and not throw any errors.

[Dirk5D13]

The previews are always displayed in good quality, even when the NAS is disconnected. Sometimes, for a brief moment, the message comes up that data is being loaded.

It doesn't matter now, I give up. It doesn't work the way I want it to, so I'm going to move all the originals back to the laptop and only work with local data.

Thanks again for your help!

[Jao vdL]

I should have done this earlier but I did the same thing to test on my setup with LR 13 on a MBP M1 Max using my synology NAS as source. I made sure standard previews existed for a folder of about 200 raw and derived tiffs. Scrolling through the  images is very fast and I see no delays. However I was surprised to see that indeed the program seems to be fetching original images in the background just looking at network activity. It doesn't seem to affect anything but it is loading 100's of megabytes over the network connection to my NAS. If I step through images quickly, it doesn't appear to haul all originals over but it seems to get the one that I stop on. This might be to speed up jumping into Develop because indeed if I jump into develop from the last image, there is zero delay for it to be available for editing. It seems to not need to be fetched anymore. So what might be going on is that Lightroom is trying to be ahead of you by prefetching the raw images into the camera raw cache that you're likely to want to jump into develop for. It doesn't seem to affect speed at all but yeah it seems to be loading raw images from the NAS constantly when just scrolling in Library even if you have all your previews created.

[Dirk5D13]

That sounds plausible, in my opinion you have provided the most conclusive explanation for this behavior and thank you for recreating it, this is not a mistake but intentional: The originals should be kept in advance in case you want to develop. That's what happens when developers think they know what I'm going to do next, it's a shame that it's not communicated and that you can't turn it off.

[johnrellis]

"If I step through images quickly, it doesn't appear to haul all originals over but it seems to get the one that I stop on."

Good observation. I used Microsoft's Process Monitor to monitor file accesses in Windows LR 13.0.1 in Library Grid view, with thumbnails built to Auto size.  In my tests, whenever the most-selected photo changes, LR reads the original but doesn't write the previews database or the Camera Raw cache.  

This happens when I use the arrow key to move quickly -- each photo that gets the selection, even very briefly, is read. This occurs with originals on the local disk and on a network volume.  

If I scroll using the scroll bar or the mouse scroll wheel to scroll through the grid, which doesn't change the selection, original photos are not read from disk.

@Dirk5D13, if you scroll using the scroll bar rather than the arrow keys, not changing the selected photo, do you observe large amounts of data read from your NAS?

"This might be to speed up jumping into Develop"

That sounds right to me. I vaguely recall a discussion years ago, perhaps at LIghtroom Queen, about LR pre-loading photos to make going into Develop faster.

Supporting this hypothesis, I observe slightly different behavior in Develop. When you move to a photo in Develop, LR immediately reads the next photo shown in the filmstrip.

So the general rule is that LR pre-loads the photo you're most likely to go to next in Develop.

[Jao vdL]

Thinking about it I remember a discussion around this too with developers about them implementing some intelligence when prefetching images. That must be what we're seeing happening here. I never put these two data points together.