P: Disable smart collection

Hi,

I have logged a feature request internally so that we can track this idea.

For now users can do the following workaround (not great but still it works) to get the same effect. Create a Collection set called "Disabled smart collections" or any name you like. Move all the smart collections you want disabled into it. Collapse the collection set and restart Lightroom. As long as this collection set remains collapsed they won't be evaluated. One drawback of this approach is that users need to remember to keep the collection set collapsed. If they open it they need to remember to close it and restart Lightroom to get the benefit.

Just to wrap up on this: It appears that your current catalog had gone "stale" with respect to the speed of smart collections -- when you exported it to a new catalog, it was much faster.   This is good to know for the future, thanks.

John,

I opened LR this morning and saved a catalogue as requested with no options

Switched off PC and restart PC.

Open the new catalogue and it took 20 seconds to fill the one set of SCs.

Using FILE > OPEN RECENT switched back to the original catalogue and this took 75 seconds to fill the same SCs.

Using FILE > OPEN RECENT switched back to the new catalogue and this took 12 seconds to fill the same SCs.

Using FILE > OPEN RECENT switched back to the orginal catalogue and this took 8 seconds to fill the same SCs.

All timings start once it has got all windows in the Browse module open and I can see the list of SCs

BTW did you see my post where I thanked you for all the work and help you have done because I can't see my post in the above thread. I hadn't thought about using a combination of Filter Preset and SC. This makes great sense. It did improve the performance and would make future changes a little easier. I have now gotten all the stats I need for my work so probably won't need this set of SCs again. But I have learned a lot from you should I need to do anything similar. Thank you.

So your catalog has is actually smaller in terms of bytes / photo: Yours is 15 KB / photo, mine is 26 KB / photo -- that doesn't explain the slowth you're seeing on cold starts.

The camera serial number, lens id, apertural, and focal length are all stored in a few small SQL tables, and each of those fields have indexes. But the tables are small enough such that a linear scan through 90K in-memory entries should be extremely fast even if it doesn't use the index (e.g. for the "contains" operators), and if the tables don't happen to be in memory, they could be read in a small fraction of a second.   That's all demonstrated by timings on my catalog.

So clearly something bad and out of the ordinary is happening with your catalog, but I don't see any clues as to what.  

In addition to using filter presets as suggested before, a couple more workaround suggestions:

- Use "is" rather than "contains" for serial number and lens id for any remaining smart collections using those fields. That will allow the indexes to be used (but as discussed before, the tables are so small that  a linear scan without the indexes should be plenty fast enough, unless the SQLite query optimizer has gone off the rails with your particular queries).

- Try exporting a copy of the catalog with File > Export As Catalog, with all of the options unchecked, and open the copy in LR. That will export a "clean" SQL database, perhaps "cleaner" than what the Optimize command does -- I believe the Export essentially rebuilds the tables from scratch.  So maybe there are some rotted bits that are embedded in the old catalog that exporting will remove.  Just an educated guess, probably only a 15% chance of having an effect.

lrcat is 1,392,516 KB

Lightroom Classic version: 7.0.1 [ 1142117 ]

Correction: The SQL query execution time is directly proportional to the total number of photos in the catalog, since SQL indexes can't be used for these queries.

As a workaround, you could significantly reduce the number of smart collections by removing the camera and lens from the criteria.  Then define a filter preset for each camera/lens combination:



So you'd first select the desired smart collection and then the appropriate camera/lens filter.

"Each group takes over 2 minutes to process before the numbers appear. If I close LR and re-open it immediately it takes 20 seconds to refill them. But by the next day when I start LR it is back to over 2 mins per group."

This strongly indicates that unusually slow or excessive disk i/o is slowing down the "cold starts" compared to the "warm starts" by a factor of 6.

With a cold start, the operating system's in-memory disk cache is empty, so LR has to read the relevant parts of the catalog file from disk. But with a warm start (restarting LR immediately after you exited), the catalog file is already in the disk cache, so little extra disk i/o is required. 

There are two possibilities: Your disk is responding to read requests unusually slowly, or your catalog file is excessively large for some reason.  Once you post the size of your .lrcat catalog file, we can compare the size / photo with my catalog and others to see if yours is excessively large (per photo).

 Before someone shouts "Optimize!", consider: LR Classic optimizes the catalog when it converted from an earlier version, so your catalog was optimized in the last week or so. So while you could try File > Optimize Catalog, it's unlikely it will have any effect. (It also compresses the history field of each photo, resulting in catalogs that can be 50% or more smaller than LR 6.)

 ------------------------

I did some quickie timings with a plugin script to verify that on warm starts, your LR is executing smart collections roughly as fast as mine, further pointing the finger at cold-start disk i/o.

 I timed the execution of a smart collection very similar to the ones you posted.  For warm starts, I observed times similar to what you (roughly) observed: 0.9 x 10^-5 sec/photo on my catalog, versus 1.8 x 10^-5 sec/photo on your catalog.  But for cold starts, I observed just slightly slower times of 1.1 x 10^-5 sec/photo, versus order-of-magnitude larger times of 11 x 10^-5 sec/photo on your catalog.

 (My timing was done on a somewhat smaller catalog stored on a reasonably fast external hard drive, with a Macbook Pro mid-2015 with 16 GB memory. The test smart collection returned 14K photos. The SQL query execution time is directly proportional to the number of photos, since SQL indexes can't be used for these queries.)

Below is the timing script I used. You can run this script in your LR by doing Preferences > Presets, clicking Show Lightroom Presets Folder, and creating a subfolder "Scripts" under the "Lightroom" folder that's selected in Mac Finder or Windows Explorer.  Then copy the script into a file "timesmartcollection.lua" in that folder and restart LR.  You'll see a Scripts menu appear on the far right of LR's menu bar, and the script "timesmartcollection" will appear in that dropdown.

 If you want to measure cold and warm starts, you'll have to make a copy of your catalog folder and measure with the copy.  Delete the smart collections from that copy so they don't affect the timing.  To measure cold starts, you can either reboot your computer each time; or on Mac, use the "purge" command in Terminal. On Windows, the following recipe supposedly works, though I haven't tried it: https://stackoverflow.com/questions/478340/clear-file-cache-to-repeat-performance-testing

If you want to time a different smart collection, in the LR's Collections panel, right-click the smart collection and do Export Smart Collection Settings.  Open the exported .lrsmcol file in a text editor and copy and paste into the "timesmartcolleciton.lua" script where indicated.

local LrApplication = import "LrApplication"
local LrDate = import "LrDate"
local LrDialogs = import "LrDialogs"
local LrTasks = import "LrTasks"
local catalog = LrApplication.activeCatalog ()
local s
--[[**** Insert the contents of the .lrsmcol file here ***************]]
s = {
    id = "CC90F266-2911-427C-8753-1469B13A76AD",
    internalName = "Focal 28-105mm 24-105 lens",
    title = "Focal 28-105mm 24-105 lens",
    type = "LibrarySmartCollection",
    value = {
        {
            criteria = "lens",
            operation = "all",
            value = "24-105",
            value2 = "",
        },
        {
            criteria = "cameraSN",
            operation = "==",
            value = "1831227178",
            value2 = "",
        },
        {
            criteria = "focalLength",
            operation = "in",
            value = 28,
            value2 = 105,
        },
        combine = "intersect",
    },
    version = 0,
}
--[[******************************************************************]]
LrTasks.startAsyncTask (function ()
    local t = LrDate.currentTime ()
    local photos = catalog:findPhotos {searchDesc = s.value}
    t = LrDate.currentTime () - t
    LrDialogs.message (s.title,
        string.format ("%d photos\n%g seconds", #photos, t))
    end)

 

 

Also, please do Help > System Info and report the exact version number. (LR's unreliable update often fools people.)

How big (in MB) is your catalog file (the .lrcat file only)?  And are you on Windows or Mac? (On Mac 1 MB = 1000 x 1000, while on Windows 1 MB = 1024 x 1024.)