Found a significant LR4 speedup - regenerate ACR cache

[jfriend0]

Like many others here, after I upgraded to LR4, my performance went completely to hell.  It would literally take 9-10 seconds just to advance to the next image in the Develop module.  Marking an image for deletion would take 15 seconds.  I'm running a Q6600 - quadcore processor at 2.4GHz with 4GB RAM on Vista 32-bit.  By today's standards, it's not a terrific speed demon, but no slouch either and it worked fine for several years with LR3.  I mostly process D300 RAW files.

After reading a lot of posts here about crummy performance, I wasn't able to find any tips that might help so I started poking around in the preferences/settings.  One thing I discovered was that my ACR cache was perhaps getting starved a bit for size.  It was set to 50GB, but it was on a drive that might not have that much free space.  I reasoned that if the cache couldn't quite be large enough to hold my working directory of images, then it might be thrashing and pretty much never loading from the cache.

So, I moved my ACR cache to a new drive with lots of free space, increased the cache size to 100GB and then proceeded to regenerate the cache for the directory of 500 images I was working on by making a 2 point change in sharpening on all images and then regenerating all previews.  It took awhile to make all new previews for all the images, but after doing so - WOW my old performance was back again, even running on the new 2012 process.  I could move from one image to the next in the develop module in under a second.

So, I don't know if it was low disk space, some sort of general caching problem, a corrupted cache or what, but after making those ACR cache changes, my LR4 performance is back neaer where the LR3 performance was.

[ashleykaryl]

Last night I created a brand new catalogue in Lightroom 4 to get away from any legacy from 3 that might be causing problems and then imported the whole lot again from their existing locations rather than an existing catalogue. I previously had my previews set to high quality 1680 pixels and to never discard but this time I tried 1440 pixels and it hasn't made a bit of difference. I've also just tried changing the previews on a small selection to medium quality but that is no faster either.

When I select an image in Develop mode it says Loading at the bottom and a wheel spins for a couple of seconds. Then when I try to enlarge it sometimes says Loading again and generally takes 2-3 seconds before it changes from a pixelated blurred image into something sharp. It just shouldn't be like this on a machine with plenty of horsepower and masses of drive space where the previews have already been saved to cache. 

[Lou Dina]

Ashleykaryl,

Our systems seem somewhat similar, and my results are pretty much the same as yours. The software has problems and I think many of these problems will not be truly solved until the offending code is corrected, rewritten, made more efficient, addresses common conflicts, etc. There is no reason that our fairly robust systems should not work great with LR4. The suggestion that we upgrade is avoidance of the real problem, plus it's expensive. I can have Photoshop, InDesign, Illustrator, Excel, Word, Firefox, LR3 and 3 or 4 other programs open simultaneously and they all work great—better than LR4  open all by itself!

In the meantime, I continue to try different things to see if I can make the develop module a bit faster. Nothing seems to help very much so far. I've tried many of the suggestions recommended on this forum and elsewhere.

Lou

Mac Pro, dual 2.26 GHz processors, 8 cores, 64 bit, 16 GB RAM, 240 GB SSD, (3) 1TB HDDs @ 7200 rpm, Nvidia GeForce GT 120 video card with 512 MB VRAM, dual monitors (it doesn't help if one is turned off), OSX 10.6.8

[ashleykaryl]

Hi Lou,

I actually had the same model of computer as you up until about a month ago before Apple replaced it under warranty due to some ongoing problems. They replaced it with the 8 Core 2.4Ghz and in terms of clock speed there isn't much difference as confirmed by some bench tests I've seen, however the graphics card is supposed to be considerably faster and yet we have the same problem, so I certainly wouldn't rush out to spend money on a new graphics card.

A few minutes ago I loaded up 4.1 RC1 on a fairly low end Windows laptop. It was a budget priced machine bought almost 3 years ago with 4 gigs of Ram and I imported 4 DNG files from the 1DsII as a test. It's clearly faster to preview images than the Mac and I'm not even sure the laptop has a dedicated graphics card plus it will be running a slow hard drive. This is ridiculous if you think about it.

Furthermore, why was everything faster last night after purging the cache but slow again after starting again with a completely fresh catalogue? I'm inclined to think you are right that this is all down to poor coding, especially given that enlarged preview images snap into sharp focus instantly in Bridge on my Mac. I only hope Adobe are reading this and working on a solution.

Ashley

PS I just tried purging the cache again and the images are suddenly fast but how long will it last?

[areohbee]

Lou,

I confess, with what seems like should be vanilla hardware (Mac), it's hard to understand why there would be problems not found by Adobe.

Once upon a time, when I was having a bad day with Lightroom, I tried Aperture on a pretty vanilla Mac and could not understand why it was an order of magnitude slower than it should have been. Had it performed well, I may be posting on an entirely different forum right now.

Anyway, at the risk of stating the obvious, if it ain't the hardware, then it's the software. (and by software I mean bits on disk, be they program or data).

I'm sure both you and Adobe want to solve this problem, but it may be that you can solve it quicker in your case.

I mean, if you pull the disk drive out and all other non-essential hardware, then put in a new disk drive and reformat it / re-install the OS, then install Lightroom. Will it work well with a new catalog?

I'm not saying this is a viable solution, although it may be. Simply knowing that it's possible for it to work well on your hardware/drivers may give hope. And knowing that it is *not* possible even on a squeaky clean system would mean you need a new computer if you want to run Lightroom before Adobe fixes it (or the problem lies with the core drivers).

I don't mean that the hardware or drivers are bad, just that they are not playing nice with Lightroom, or if you prefer - vice versa.

Rob

[ashleykaryl]

I've just had a thought here. In my case all the Lightroom preview data files and the catalogue etc are stored on a drive that also contains all the images but not the OS because I read somewhere that this would deliver faster performance and yet the Lightroom default is to store all that info on the same drive as the main OS. I'm wondering if this may be part of the problem. 

[areohbee]

I dunno. But the saying: "if it ain't working the way it is, then change something" springs to mind...

[web-weaver]

ashleykaryl,

It should not make a difference in the speed of Lr if you have the catalog on the same drive where your photos are. But if this is by any chance an external USB drive, then you'd better put the catalog on an internal drive even if you have only one.

The USB connection is too slow and the catalog is constantly written to and read from, so there's quite a bit of traffic.

Thus, while is helps to put the catalog on a different drive than where the OS and Lr reside, it is more important that the catalog is on a drive with a fast connection. For external drives that means only eSata drive would be fast enough for the catalog.

[ashleykaryl]

No it was on an internal Sata drive alongside all the images. I've just copied the catalogue to the main drive and tried it from there but it's just as bad as before. I can't see any difference really.

[SistersCountry]

@ashleykaryl: " I've just copied the catalogue to the main drive and tried it from there but it's just as bad as before. I can't see any difference really."

That last step, separating the catalog and the images, seemed to help me a bit when mine was having big problems. In your troubleshooting steps have you yet exported everything to a new catalog? This had a positive impact for me after originally having done a straight conversion from 3.6. Another thing for the check list is to make sure you are not automatically writing to .xmp. Mine has cleared up and is running very well now, even better with RC1. I don't think it was any one magic bullet that did it, rather covering a lot of different bases including video driver updates.

I have just been building 1:1 previews for my folders as needed, one reason being it helps me benchmark how performance may be changing.

[ashleykaryl]

@SistersCountry: Thanks for your suggestions. I feel like I've basically tried it all at this point. I started by importing the catalogue from LR3, then I tried updating all the previews and even took the step of converting every single image from PV2010 to PV2012.

After this I even scrapped the catalogue entirely and started from scratch with a brand new one in LR4 by importing the images from the current position. In all of this the only thing that seems to work is purging the cache, which sees an instant improvement where everything happens in the blink of an eye but only for a short period before the slow down kicks in. I've always built the 1:1 previews at time of import partly becuase it's always been so slow otherwise and maybe that's the problem, though it absolutely shouldn't be and certainly Bridge is fine.

[ashleykaryl]

I'm also on 4.1 RC1 now and it hasn't really made any difference. I've just taken the rather drastic move after backing up my data to convert the entire Raw catalogue to the 2012 processing version but it hasn't helped.

[areohbee]

Lou Dina wrote:

Whenever I display ANY DNG image in my main import folder (this is the folder into which I always import my images for temporary evaluation), I do NOT get any writes to cache.

This may be because fast-load data is enabled. See if you get writes to the cache when fast-load data is disabled - you may need to try it with virginal photos, since there is a chance Lr4 will read it when available and still fresh, but not write it, even when fast-load data is disabled.

[Lou Dina]

Rob,

Yup....that's it. I decided to purge my cache and turn of fast load data. Then I imported an image into my standard Import folder. When I open the develop module, the other images (which apparently have fast load data embedded....even though some were imported into LR3 before this feature existed) do not write to cache. The new image (with fast load data disabled) definitely did write to cache. These are all DNG files.

My only complaint is that fast load data ain't fast!!!

Lou

[areohbee]

Lou Dina wrote:

My only complaint is that fast load data ain't fast!!!

It's entirely possible that the underlying ACR caching technology hasn't changed in any revolutionary way. It's performance benefit used to be less than 15% max, perhaps the fast-load data gives similar improvement?

Although I am curious, I probably won't be checking it, since a small performance boost is not sufficient reason for me to switch to DNG. A big performance boost would be enough reason...

Rob

[areohbee]

Lou Dina wrote:

Thanks, Rob. I was afraid that would be your answer.

"How much of a speed-up are you getting from the fast-load data? Anybody else noted a difference with/without fast-load data?"

I'm not sure how to answer your question. I am assuming, (perhaps incorrectly), that only the DNG images I have imported since the fast load data checkbox was checked use the fast load data and have the slightly larger file size. I am guessing that all my previously imported/converted CR2 > DNG files are slightly smaller size and do not have the fast load data built into them. How can one tell?

At any rate, I just imported three raw images from my 5Dmk2, and told LR4 to convert to DNG and create 1:1 previews upon import. (The import was pretty slow too). When moving between images displayed at 1:1 maginification in the Develop module, the new files (imported with fast load data and 1:1 previews) take about 1-1/2 to 2 seconds to show up sharply. Some older DNG images imported with LR3 take 3-4 seconds to show up sharply. Not very scientific. All these DNG files have a file size in the 20-25 MB range.

Hi Lou,

Since I don't use DNG, I haven't done my homework, but one way to check:

1. Enable fast-load data.

2. Make a copy of a virginal photo file (outside of Lightroom).

2. Import as new photo.

3. Check time to switch to photo after it's already been selected once in develop view, and therefore the fast-load data has been written, but select a half dozen other photos first, to assure develop module is not fetching develop view from ram.

4. check file size.

5. Disable fast-load data.

6. Re-import virginal copy of same photo again, but with different filename.

7. Check time to switch to photo after it's already been selected once in develop view, but select a half dozen other photos first...

8. check file size - it should be considerably smaller than in the previous test, since it definitely won't have the fast-load data.

The difference between times is the performance advantage of using fast-load data.

Rob

[Scooby007]

Adobe recommends ACR cache of "10gb or more" for Lightroom

http://helpx.adobe.com/lightroom/kb/optimize-performance-lightroom.html

I think the norm for the ACR cache is around 50GB just to make sure it's big enough for a large shoot.

That would be one heck of a shoot - about 140,000 images! (with the cache per image for a 21mp capture averaging 350k)

[jfriend0]

B r e t t wrote:

Adobe recommends ACR cache of "10gb or more"

http://helpx.adobe.com/lightroom/kb/optimize-performance-lightroom.htm l

In practice how large an ACR cache you need would depend upon whether you work with RAW images, how big your RAW images are and what a typical working set of RAW images that you work with are (200, 500, 1000, etc...).  The more you process together, the larger a cache you would need.  I figure that 2x the max amount of images you ever import at a time is probably a good safe minimum to work on that particular shoot.  So, since I sometimes come near to filling up two 16GB cards in a big sports shoot, a safe min for me would be 2x16 x 2 = 64GB.  I set mine to 100GB.

[Scooby007]

I was simply passing along what Adobe - the author of the software - recommends.

I figure that 2x the max amount of images you ever import at a time is probably a good safe minimum to work on that particular shoot.  So, since I sometimes come near to filling up two 16GB cards in a big sports shoot, a safe min for me would be 2x16 x 2 = 64GB.  I set mine to 100GB.

The cache is not the same size as the capture. It's about 2 or 3% of the size (e.g. 97% smaller). Your 100gb is enough for almost 350,000 photos.

[areohbee]

In my experience, when Lightroom is operating normally, ACR cache hits generally have only a very minor effect on performance. If it has a big effect, I think you've got some wonky cachin' goin' on, like jfriendl0 had...

[Scooby007]

I have cache going way back and the max is 3.9mb, with the average being 350k. In any case it is somehting like 75-95% smaller than the file, not 3 to 4 times bigger.

[SeanMcCormack]

13 sounds big. 500k sounds about right for a standard preview. Have you been doing a lot of 1:1 preview recently? They should be bigger.

I'm moving to a DNG workflow for the Fast Load Data. Older non selects will go to Lossy DNG. They're already backed up as Raw and right now, I've probably mere thousands of shots I'd be lost without.

[areohbee]

Pre Lr4, all cache entries were exactly the same size (and big), given the same preview settings (I can't remember if it was quality or resolution that affected it, but it was not what one might think), and the same model camera. Now, the sizes are image dependent, and small. Something has definitely changed, ACR-cache-wise, a lot, in my copy/system. Not using DNG.

[SeanMcCormack]

Just had a look. I've a mix of 500k and 8.8Mb files recently.

[areohbee]

All small for me since purging recently (minutes ago) - regardless of process version.

[SeanMcCormack]

Purged.. some of them were well old and despite the 50 gig limit I entered, it was 56Gb..