P: Disable smart collection

[ianwhiting]

Large or multiple smart collections can seriously impact performance.

It would be useful to disable a smart collection or, even better, disable a set of smart collections, so they are no longer processed until re-enabled.

As it is we can only delete the smart collections and recreate them when we need them again next month.

A selective export and import option of smart collections would serve a similar purpose and would add functionality to LR but would not be as simple to use just for this disable/enable purpose.

Whilst here I would also like to add my vote to the many people who have asked for a smart collection A only to select images from within smart collection B. Though I can see some concerns: how many levels down does it allow should B only select from smart collection C, etc. That is maybe why they have not implemented it. Also there would be a possibility of creating an infinite loop should smart collection A only select from B but B is set up to only select from A. Perhaps it could be limited to only allow selection from smart collection B provided that B does not have use any sub-selection of its own.

[JohanElzenga]

Yes, I believe that a possible never ending loop is indeed one of the reasons why Adobe did not implement searching for images in another smart collection. The other reason is that this is functionality that already exists anyway. You can simply include the criteria of smart collection B in the criteria of smart collection A. It may be a little more work, but in the end it's exactly the same result.

The problem with feature suggestions is that people can suggest anything they like. Thy don't have to check how difficult it is to add this to the existing code. They don't have to check wether Lightroom would become even more bloatware with all these features. They don't have to check whether more than a handful of people might ever use this. And of course they always think that their feature request is the most important request in the history of Lightroom and so it is unbelievable that it is still not implemented...

[JohanElzenga]

Coming back to your initial question: I doubt that turning off smart collections would make Lightroom significantly faster. When Lightroom starts, it needs to run the query for each smart collection. That is why you see the image count behind your smart collections appear not instantly, but (depending on the size of the catalog) after a few seconds. But after that, when you change the criteria of an image, Lightroom only needs to check that one particular image against the smart collections. When I change an image from 4 stars to 5 starts, I see my four star and five star smart collections update instantly. BTW, my catalog contains 170,000 images.

[ianwhiting]

Hi Johan, you are quite right. But I find that my smart collections, I have about 12, are rebuilt every time LR starts. This background process takes about 5 minutes and slows down LR. I have read that some users have a lot more smart collections. I imagine they are just simple SQL statements and thus selecting on an index-able criteria like the "star rating number column" then they will be very quick. But if using a CONTAINS or LIKE search (e.g. lens criteria text column contains "24-105") they take a lot longer as they need to read all rows in the database rather than using a pre-built index.

[johnrellis]

"This background process takes about 5 minutes and slows down LR."

There have been very few posts in the last many years about very slow smart collections.  It would be good to narrow down what it is about your situation causing this.

- How many photos in your catalog?

- Has your catalog been upgraded from previous major versions, and if so, which ones?

- Can you right-click each smart collection and do Export Smart Collection Settings.  Zip up the 12 .lrsmcol files, upload to Dropbox or similar, and post the sharing link here.  I'm intimately familiar with LR's search capabilities (see my Any Filter plugin), so I might be able to identify a culprit.

[ianwhiting]

Thanks John,

90,000 images and Lightroom Classic is fully up-to-date (7.0.1) and catalogue converted. It is on a reasonably fast Windows 10 PC with 16GB. Not much else runs concurrently. LR is on a HD, not a SSD.

Right now I have 12 SC (Smart Collections) similar to the two in the attached zip file in one set, then the group is repeated for each camera and lens combination. Previously they were a little more complex

https://www.dropbox.com/s/kh80r9f0gs7v4g2/LR.zip?dl=0

BTW The lens description in the image file is "EF24-105mm f/4L IS USM", it might be faster if I searched for the full description but I can imagine instances where limiting it to some embedded text is preferable, eg to include different versions of a 50mm lens.

Each group takes over 2 minutes to process before the numbers appear. If I close LR and re-open it immediately it takes 20 seconds to refill them. But by the next day when I start LR it is back to over 2 mins per group.

I would like to keep many SC sets active (or disabled if that was an option) so I didn't have to edit and re-edit each time for each new criteria, e.g. date ranges, lens type, camera number etc. but as they are so slow I have now deleted all but one set.

(I do have some other SCs but they are quick to resolve)

It would be even better if a group could select from another smart collection, eg SC 2 selects all images for a camera and lens type, then each SC in a group pull from SC 2 and refine that selection by, say aperture value range. That would make setting up groups much easier and I would only need to change SC 2 to get the next batch of figures.

[johnrellis]

Thanks, it will probably take me a day or so to look at this in detail.

[johnrellis]

How big (in MB) is your catalog file (the .lrcat file only)?  And are you on Windows or Mac? (On Mac 1 MB = 1000 x 1000, while on Windows 1 MB = 1024 x 1024.)

[johnrellis]

Also, please do Help > System Info and report the exact version number. (LR's unreliable update often fools people.)

[johnrellis]

"Each group takes over 2 minutes to process before the numbers appear. If I close LR and re-open it immediately it takes 20 seconds to refill them. But by the next day when I start LR it is back to over 2 mins per group."

This strongly indicates that unusually slow or excessive disk i/o is slowing down the "cold starts" compared to the "warm starts" by a factor of 6.

With a cold start, the operating system's in-memory disk cache is empty, so LR has to read the relevant parts of the catalog file from disk. But with a warm start (restarting LR immediately after you exited), the catalog file is already in the disk cache, so little extra disk i/o is required. 

There are two possibilities: Your disk is responding to read requests unusually slowly, or your catalog file is excessively large for some reason.  Once you post the size of your .lrcat catalog file, we can compare the size / photo with my catalog and others to see if yours is excessively large (per photo).

 Before someone shouts "Optimize!", consider: LR Classic optimizes the catalog when it converted from an earlier version, so your catalog was optimized in the last week or so. So while you could try File > Optimize Catalog, it's unlikely it will have any effect. (It also compresses the history field of each photo, resulting in catalogs that can be 50% or more smaller than LR 6.)

 ------------------------

I did some quickie timings with a plugin script to verify that on warm starts, your LR is executing smart collections roughly as fast as mine, further pointing the finger at cold-start disk i/o.

 I timed the execution of a smart collection very similar to the ones you posted.  For warm starts, I observed times similar to what you (roughly) observed: 0.9 x 10^-5 sec/photo on my catalog, versus 1.8 x 10^-5 sec/photo on your catalog.  But for cold starts, I observed just slightly slower times of 1.1 x 10^-5 sec/photo, versus order-of-magnitude larger times of 11 x 10^-5 sec/photo on your catalog.

 (My timing was done on a somewhat smaller catalog stored on a reasonably fast external hard drive, with a Macbook Pro mid-2015 with 16 GB memory. The test smart collection returned 14K photos. The SQL query execution time is directly proportional to the number of photos, since SQL indexes can't be used for these queries.)

Below is the timing script I used. You can run this script in your LR by doing Preferences > Presets, clicking Show Lightroom Presets Folder, and creating a subfolder "Scripts" under the "Lightroom" folder that's selected in Mac Finder or Windows Explorer.  Then copy the script into a file "timesmartcollection.lua" in that folder and restart LR.  You'll see a Scripts menu appear on the far right of LR's menu bar, and the script "timesmartcollection" will appear in that dropdown.

 If you want to measure cold and warm starts, you'll have to make a copy of your catalog folder and measure with the copy.  Delete the smart collections from that copy so they don't affect the timing.  To measure cold starts, you can either reboot your computer each time; or on Mac, use the "purge" command in Terminal. On Windows, the following recipe supposedly works, though I haven't tried it: https://stackoverflow.com/questions/478340/clear-file-cache-to-repeat-performance-testing

If you want to time a different smart collection, in the LR's Collections panel, right-click the smart collection and do Export Smart Collection Settings.  Open the exported .lrsmcol file in a text editor and copy and paste into the "timesmartcolleciton.lua" script where indicated.

local LrApplication = import "LrApplication"
local LrDate = import "LrDate"
local LrDialogs = import "LrDialogs"
local LrTasks = import "LrTasks"
local catalog = LrApplication.activeCatalog ()
local s
--[[**** Insert the contents of the .lrsmcol file here ***************]]
s = {
    id = "CC90F266-2911-427C-8753-1469B13A76AD",
    internalName = "Focal 28-105mm 24-105 lens",
    title = "Focal 28-105mm 24-105 lens",
    type = "LibrarySmartCollection",
    value = {
        {
            criteria = "lens",
            operation = "all",
            value = "24-105",
            value2 = "",
        },
        {
            criteria = "cameraSN",
            operation = "==",
            value = "1831227178",
            value2 = "",
        },
        {
            criteria = "focalLength",
            operation = "in",
            value = 28,
            value2 = 105,
        },
        combine = "intersect",
    },
    version = 0,
}
--[[******************************************************************]]
LrTasks.startAsyncTask (function ()
    local t = LrDate.currentTime ()
    local photos = catalog:findPhotos {searchDesc = s.value}
    t = LrDate.currentTime () - t
    LrDialogs.message (s.title,
        string.format ("%d photos\n%g seconds", #photos, t))
    end)

[johnrellis]

As a workaround, you could significantly reduce the number of smart collections by removing the camera and lens from the criteria.  Then define a filter preset for each camera/lens combination:



So you'd first select the desired smart collection and then the appropriate camera/lens filter.

[johnrellis]

Correction: The SQL query execution time is directly proportional to the total number of photos in the catalog, since SQL indexes can't be used for these queries.

[ianwhiting]

lrcat is 1,392,516 KB

Lightroom Classic version: 7.0.1 [ 1142117 ]

[johnrellis]

So your catalog has is actually smaller in terms of bytes / photo: Yours is 15 KB / photo, mine is 26 KB / photo -- that doesn't explain the slowth you're seeing on cold starts.

The camera serial number, lens id, apertural, and focal length are all stored in a few small SQL tables, and each of those fields have indexes. But the tables are small enough such that a linear scan through 90K in-memory entries should be extremely fast even if it doesn't use the index (e.g. for the "contains" operators), and if the tables don't happen to be in memory, they could be read in a small fraction of a second.   That's all demonstrated by timings on my catalog.

So clearly something bad and out of the ordinary is happening with your catalog, but I don't see any clues as to what.  

In addition to using filter presets as suggested before, a couple more workaround suggestions:

- Use "is" rather than "contains" for serial number and lens id for any remaining smart collections using those fields. That will allow the indexes to be used (but as discussed before, the tables are so small that  a linear scan without the indexes should be plenty fast enough, unless the SQLite query optimizer has gone off the rails with your particular queries).

- Try exporting a copy of the catalog with File > Export As Catalog, with all of the options unchecked, and open the copy in LR. That will export a "clean" SQL database, perhaps "cleaner" than what the Optimize command does -- I believe the Export essentially rebuilds the tables from scratch.  So maybe there are some rotted bits that are embedded in the old catalog that exporting will remove.  Just an educated guess, probably only a 15% chance of having an effect.

[ianwhiting]

John,

I opened LR this morning and saved a catalogue as requested with no options

Switched off PC and restart PC.

Open the new catalogue and it took 20 seconds to fill the one set of SCs.

Using FILE > OPEN RECENT switched back to the original catalogue and this took 75 seconds to fill the same SCs.

Using FILE > OPEN RECENT switched back to the new catalogue and this took 12 seconds to fill the same SCs.

Using FILE > OPEN RECENT switched back to the orginal catalogue and this took 8 seconds to fill the same SCs.

All timings start once it has got all windows in the Browse module open and I can see the list of SCs

BTW did you see my post where I thanked you for all the work and help you have done because I can't see my post in the above thread. I hadn't thought about using a combination of Filter Preset and SC. This makes great sense. It did improve the performance and would make future changes a little easier. I have now gotten all the stats I need for my work so probably won't need this set of SCs again. But I have learned a lot from you should I need to do anything similar. Thank you.

[johnrellis]

Just to wrap up on this: It appears that your current catalog had gone "stale" with respect to the speed of smart collections -- when you exported it to a new catalog, it was much faster.   This is good to know for the future, thanks.

[Sreenivas R]

Hi,

I have logged a feature request internally so that we can track this idea.

For now users can do the following workaround (not great but still it works) to get the same effect. Create a Collection set called "Disabled smart collections" or any name you like. Move all the smart collections you want disabled into it. Collapse the collection set and restart Lightroom. As long as this collection set remains collapsed they won't be evaluated. One drawback of this approach is that users need to remember to keep the collection set collapsed. If they open it they need to remember to close it and restart Lightroom to get the benefit.

[johnrellis]

That's a good tip, thanks!

[FeedbackCommunityMember]

When you export a catalog this way and you want to keep using sync to LR Mobile and Web, I guess you need to re-sync everything when taking that exported new catalog as the primary catalog used for syncing? And then LR Mobile and Web will have to re-sync everything for the "new" catalog, even though nothing has changed with the photos?

Just speculation, but will be useful if I need to do this to my own catalog sometime. I have >18.000 photos synced in LR CC 2015.

[FeedbackCommunityMember]

...and it's enough to collapse any Collection Sets containing Smart Collections, for the same effect, right? (SC don't actually have to be in just one Collection Set)

[Sreenivas R]

Yes. It can be any collection set. But if you know some of the smart collections take a long time to update leading to problems and you don't plan to use them for sometime it is always better to collect them in a separate collection set so that they don't come in the way of your day to day work.

[FeedbackCommunityMember]

Thank you for confirming this. Yes, I've done just that – I keep my Smart Collections for recent work (by dates) separate from many other Smart Collections in separate Collection Sets.

[ianwhiting]

Many thanks, that is good to know

[axmelissarios1]

Hi all,

This feature should be present in today's Lightroom Classic CC 7.2 version. Please update your application and let us know what you think!

Lightroom Classic CC 7.2 is now available!

Thanks,
Melissa

[Califdan2]

I have over 50 smart collections, maybe closer to 75 but I don't need them all to be active all the time.  Some I only use infrequently. But I don't want to delete them as they are at times quite complex.  I still want to see them in the "collections" panel so I don't want to Export the Smart Collection to a file (for future use) and delete them from LR.  I also don't need LR to continuously keep their content current which sucks up a fair amount of processing resources. 

My request is to add a "Status" toggle on the Smart Collection Edit screen and/or in the Collections Panel that will allow me to enable or disable (activate or deactivate) individual smart collections.  When "enabled they would behave as they do now.  When disabled LR would gray them out in the Collections Panel, and would not keep their image content up to date.

Shouldn't be that hard to implement.

Thanks -- Dan 

[johnrellis]

As a less-convenient workaround, try Sreenivas' tip: 

"Create a Collection set called "Disabled smart collections" or any name you like. Move all the smart collections you want disabled into it. Collapse the collection set and restart Lightroom. As long as this collection set remains collapsed they won't be evaluated. One drawback of this approach is that users need to remember to keep the collection set collapsed. If they open it they need to remember to close it and restart Lightroom to get the benefit."