Found a significant LR4 speedup - regenerate ACR cache

[jfriend0]

Like many others here, after I upgraded to LR4, my performance went completely to hell.  It would literally take 9-10 seconds just to advance to the next image in the Develop module.  Marking an image for deletion would take 15 seconds.  I'm running a Q6600 - quadcore processor at 2.4GHz with 4GB RAM on Vista 32-bit.  By today's standards, it's not a terrific speed demon, but no slouch either and it worked fine for several years with LR3.  I mostly process D300 RAW files.

After reading a lot of posts here about crummy performance, I wasn't able to find any tips that might help so I started poking around in the preferences/settings.  One thing I discovered was that my ACR cache was perhaps getting starved a bit for size.  It was set to 50GB, but it was on a drive that might not have that much free space.  I reasoned that if the cache couldn't quite be large enough to hold my working directory of images, then it might be thrashing and pretty much never loading from the cache.

So, I moved my ACR cache to a new drive with lots of free space, increased the cache size to 100GB and then proceeded to regenerate the cache for the directory of 500 images I was working on by making a 2 point change in sharpening on all images and then regenerating all previews.  It took awhile to make all new previews for all the images, but after doing so - WOW my old performance was back again, even running on the new 2012 process.  I could move from one image to the next in the develop module in under a second.

So, I don't know if it was low disk space, some sort of general caching problem, a corrupted cache or what, but after making those ACR cache changes, my LR4 performance is back neaer where the LR3 performance was.

[Hal P Anderson]

Lou,

PC, and I did it with NEF files. If you have DNG and created them with fast load data, I think the ACR cache won't be used.

Hal

[areohbee]

Lou,

I'd be very interested in the results if you benchmark improvements resulting from fast-load data.

Rob

[Guest]

Just a reminder that the ACR cache is used in a few places, mostly for read only. All you have to do is scroll through grid mode in the Library against some photos you haven't looked at in awhile, and you will see new entries. If you are like me and have a 4x4 grid in the library as a default, you will see 16 new cache files. You will know it is doing this if the grid previews are greyed-out at first.

[areohbee]

Sounds like there has been some significant new handling of ACR cache. It used to only be used in develop module (or exports) - big entries written (or read) when switching to develop view, or creating 1:1 previews using Library menu, but otherwise never created in Lib module.

Now it's creating much smaller files, when browsing in Lib module too...

How about one of you go-getters mastermind, benchmark, and report, eh?

[Lou Dina]

Hey all....thanks for the responses.

First, I knew nothing about "fast load data" for DNG files. I looked in my LR4 Prefs and found that I do have fast load data checked for DNG files, but I am not sure when that happened. I assume it is checked by default when you install LR4. However, most of my DNG files were created when importing raw files using LR2 and LR3. I typically import all my images into my "5D Images" folder for initial review, and after playing with these images for a few days or weeks, I move them to other folders for final storage and organization. 

I can click view any image in my 5D Images folder, in either the Library or Develop module, and no entries are written to my cache file—not a single one.

Next, I went to Library mode and navigated to different folder, then clicked through some images, and every DNG file I clicked on did then write to the cache file. I'm not sure why that doesn't happen in my 5D Import folder—maybe because the images are more current, have had previews recently created, etc? Beats me. Anyway, the average size of the cache entries seems to be about 400kb in size, plus or minus about 50kb.

If I set my magnification level to 100% view (1:1) and move from one DNG image to another, it seems to take 2-3 seconds for a sharp final image to appear. I'm using a 240 GB SSD drive for programs, OS, LR catalog & previews and my cache at the moment. All my 40,000 images are on a different 7200 RPM, 1TB HDD. I'm running OSX 10.6.8, 16 GB RAM, (two) 2.26 4-core GHz processor (total of 8 cores). Sometimes when I navigate back to a previously cached image and click on it again, it comes up sharp instantaneously, but sometimes it takes another 2-3 seconds to display sharp (all this is at 1:1 preview).

In the Develop module, a previously cached image (or one where I created 1:1 preview manually) will immediately come up sharp at 1:1 magnification, but then it gets fuzzy  for 2-3 seconds while the little hour glass spins, and returns to being sharp again. That's crazy. I am not sure what is going on, but wanted to report back on what I am finding.

Lou

[areohbee]

Thanks for the update Lou.

How long does it take to get a finished develop view with fast-load data disabled?

Lou Dina wrote:

1: Sometimes when I navigate back to a previously cached image and click on it again, it comes up sharp instantaneously...

2: In the Develop module, a previously cached image (or one where I created 1:1 preview manually) will immediately come up sharp at 1:1 magnification, but then it gets fuzzy  for 2-3 seconds while the little hour glass spins, and returns to being sharp again. That's crazy. I am not sure what is going on, but wanted to report back on what I am finding.

See 8 posts up for the answer to these questions (1: its cached in ram, 2: lib preview - crude dev rendering - final dev rendering).

Rob

[Lou Dina]

Thanks, Rob.

Is there any way to keep it cached in RAM so it is faster and skips that slow, fuzzy 2nd stage? I have 16 GB RAM, 35 GB dedicated to ACR Cache (most of which is empty since I recently purged the cache) and a 240 GB SSD drive that has about 130 GB free.

[areohbee]

Hi Lou - no way to speed it up that I know of, and thus I complain too...

How much of a speed-up are you getting from the fast-load data? Anybody else noted a difference with/without fast-load data?

Rob

[Lou Dina]

Thanks, Rob. I was afraid that would be your answer.

"How much of a speed-up are you getting from the fast-load data? Anybody else noted a difference with/without fast-load data?"

I'm not sure how to answer your question. I am assuming, (perhaps incorrectly), that only the DNG images I have imported since the fast load data checkbox was checked use the fast load data and have the slightly larger file size. I am guessing that all my previously imported/converted CR2 > DNG files are slightly smaller size and do not have the fast load data built into them. How can one tell?

At any rate, I just imported three raw images from my 5Dmk2, and told LR4 to convert to DNG and create 1:1 previews upon import. (The import was pretty slow too). When moving between images displayed at 1:1 maginification in the Develop module, the new files (imported with fast load data and 1:1 previews) take about 1-1/2 to 2 seconds to show up sharply. Some older DNG images imported with LR3 take 3-4 seconds to show up sharply. Not very scientific. All these DNG files have a file size in the 20-25 MB range.

[paul-w]

Hi All,

I have exactly the same problem that Lou had . NO writes to the cache at all. I've purged the cache, set it to 50Gb, chosen a location on a different local drive to the catalog and images, made changes to 150 DNGs with fast load data, re-written 1:1 previews for those 150 files and still no writes to the cache.

This has only been happening since I installed the trial, prior to that, with the same set-up, I had LR4beta installed and it was writing to the cache as it always had done with 1,2 and 3! I didn't uninstall the beta before installing the trial.

Nex step going to try to re-install and see what happens!

Intel dual processor 3GHz, 8Gb RAM (max available on this motherboard) + 16Gb readyboost RAM, Win7 64bit, OS,LR and catalog on 7200rpm hdd with 30% space, cache on separate 320Gb drive 7200 rpm, DNGs on 640Gb 7200rpm hdd.

Message was edited by: paul-w - added extra info

[paul-w]

Instead of re-installing I dismounted the ReadyBoost RAM and again purged the cache, made a change to about 150 7.5Mb DNGs and again created 1:1 previews. Let that process finish and then checked the cache. Still no writes.

Next stop re-install and see what happens

[areohbee]

I apologize in advance if this has already been answered, but has anybody confirmed that the cache is still supposed to be used for DNGs when fast-load data is enabled?

[paul-w]

In post 37 above Hal said he didn't think that the cache would be used with DNGs and Fast Load Data. But nobody has confirmed this.

Still reinstalling and generating 1:1s of 200+ files. Will get back as soon as I can.

[Lou Dina]

Paul-w....

Whenever I display ANY DNG image in my main import folder (this is the folder into which I always import my images for temporary evaluation), I do NOT get any writes to cache. Some of these images were imported after LR4 was installed with fast load data enabled. Some of these images were imported months ago using LR3, so fast load data was not enabled. All are DNG files (no proprietary raw files....ie, CR2). There are some PSD, TIf and JPG files, but they do not trigger a write to cache. Even the DNG files do not trigger a write to cache.

Since LR4 was performing so sluggishly, I did at one time create 1:1 previews for my entire import folder. Perhaps this is the reason nothing is written to cache? I'm not sure. Like I said, some were imported and converted to DNG by LR4 with fast load enabled, and some were imported and converted to DNG by LR3. But no files in this folder trigger a write to cache, not even DNG files.

If I switch to another folder, clicking on a DNG file DOES trigger a write to cache. In these other folders did not manually create 1:1 previews. I am guessing, perhaps mistakenly, that manually creating 1:1 previews writes this data to the LR image preview file and perhaps eliminates the need to use cache. PSD, TIF, JPG and other formats don't seem to trigger a write to cache no matter where they are located. I don't have any native CR2 files on my machine, so cannot comment on them.

Just wanted to be clear what my situation is, since it may be different from yours. Even with 1:1 previews manually generated and previously cached images, I still find the display at 100% magnification to be fairly slow. If an image was just viewed (within the last half dozen images), it pops up fast and in focus. But, even if I created 1:1 previews and it was cached, it comes up in 2-4 seconds if viewed awhile back.

Lou

[ashleykaryl]

I think I've been seeing the same sort of problem as Lou for a considerable time. In fact it was slow with LR3 as well and never gave the impression that data was being saved to the cache, even though the designated folder was full of files. Curiously if I try to view the same images large inside Bridge they pop into sharp focus at large size in a fraction of a second but in Lightroom (now version 4) it takes 2-3 seconds. I'm on a new 8 Core Mac Pro with 32 Gigs of Ram and it's not really any better than on my 5 year old iMac.

I've just tried purging the cache, which I've never tried before and I've allowed up to 200 gigs of space but the drive itself has over 600 gigs of space left and the cache was only 8 gigs.

All files from my 1DsII are converted to DNG with 1:1 previews created at time of import but it just doesn't feel like that data is being properly saved. This is a catlogue that was upgraded from LR3 and I just selected all the images with the idea of rendering 1:1 previews again. For some reason it picked out 639 images from over 11,000 that needed rendering. Once that's finsihed I'll try to update DNG Previews & Metadata. I'm clutching straws here because I just don't know why it's like this.

[ashleykaryl]

It's really strange but after clearing the cache most images are popping into sharp focus quickly now but some still take a few seconds. Do I try re-rending previews for the whole lot?

[Lou Dina]

Ashley,

I'm not finding much improvement no matter what I do. The best improvement I got was creating a new catalog from my files. Rendering new previews, clearing cache, etc, sometimes helps and sometimes it doesn't. I think most of the problem is poor coding, poor caching, poor usage of computer power and cores, probably some inefficient algorithms, etc. Why it takes 3-5 seconds for sharpening to show up when toggling the sharpening on and off is a mystery to me. I'm sure some tweaking will improve performance somewhat, but Adobe needs to address the bottlenecks at the coding level. It's sad that paying consumers have to jump through hoops to get LR4 to perform even acceptably. I have loaded LR4.1 RC1, which is also a beta release, but don't see much difference. It's much slower than LR3.6. Hopefully, they will get it right soon and address these bottlenecks.

I did a paid job today and uploaded about 75 images from my 5Dmk2. I used LR4 to upload, convert to DNG, had fast load data turned on, and had it create 1:1 previews. In the develop module, there is still an unacceptable delay when moving from image to image. Library module is fast, but develop module is slow. I have plenty of HDD space, reasonably fast drives, SSD for OS, programs and my cache, 16 GB RAM and 8 cores. It's not the fastest computer on the block, but it's no dog either. I pity the poor people with older computers.

Given some time, I hope Adobe finds a way to process these images efficiently.

Lou

[DdeGannes]

Quote "In the develop module, there is still an unacceptable delay"

Unlike the Library module that uses the previews for displaying image the Develop Module uses the Camera Raw cache and rerenders to display the original image with adjustments. Make sure you have allocated sufficient size relative to your Catalog. I have a catalog of approx 18,000 raw files and I have 40 GB for the cache and when I checked it today it had used 39.9 GB. 

[areohbee]

Cache entries are so small now I could use a cache less than 1/10 of the size I needed in Lr3.

[areohbee]

Lou,

Lr4.0 performs much better for me (e.g. sharpening shows up in a small fraction of a second when toggling sharpening on and off) - like Lr2.0/1 & Lr3.0/1 did, Lr4.0/1 obviously still has system dependent performance problems.

My point is that it may be worth doing a little Sherlock Holmes-ing on your system to fix the problem. I realize you'd rather Adobe did it, and they might, but it may be a year from now in Lr4.6 before the fix for your particular case is handled...

My apology in advance if this effort is already well under way...

Rob

[Lou Dina]

Rob,

You have zero to apologize for. I appreciate your knowledge, feedback and assistance, as always.

I've been screwing around with lots of settings, caches, configurations, new catalogs, rendering 1:1 previews, turning sharpening on and off, etc. My computer, like lots of other people out there has plenty of power and available resources. This thing is a pig so far. I've also loaded LR4.1 RC-1. I'm getting burnt out on doing Adobe's job for them after paying for their "final" release (LR4.0, which is supposed to be the post-beta release). It is absolutely IMPOSSIBLE that they didn't know about many of these problems and the poor performance before releasing this product—they have a LOT of beta testers out there. Having spent many years in the corporate world, I am sure it was a management decision based on revenue generation. I'll continue to chip away, but decent baseline performance is their responsibility. Sorry....done venting.

I can use LR4, but I'm not happy at all with the performance. Thankfully, I am not under a lot of pressure to perform, because this is NOT a productivity tool. LR3 was. I like some of the new features and prefer the new controls in the basic tab. Hopefully, they can get this pig to sing.

Best, Lou

[jfriend0]

Lou Dina wrote:

I can use LR4, but I'm not happy at all with the performance. Thankfully, I am not under a lot of pressure to perform, because this is NOT a productivity tool. LR3 was. I like some of the new features and prefer the new controls in the basic tab. Hopefully, they can get this pig to sing.

LR 4.0 is frustrating with a quad-core on 12 megapixel images.  I can't imagine how this would perform is I had one of the 24 or 36 megapixel cameras - eeck.  I hope they fix it too in 4.1.

[ashleykaryl]

I tried playing with some sample Raw files earlier from the new Nikon D800 and it was painful. Right now I'm starting with a brand new catalogue and creating 1:1 previews from scratch. Let's see if that helps, though it didn't in the past. This took all night the last time I tried it.

[areohbee]

There is one bug still in Lr4.1 RC that has been identified for fix in Lr4.1 final:

Go look at this comment

[areohbee]

Lou Dina wrote:

My computer, like lots of other people out there has plenty of power and available resources.

Indeed, assuming you've got at least a reasonable modicum of ram, these types of performance problems are due to things not playing nice with other things.

Once you've exhausted the Lightroom-specific things, like preferences, caches, plugins, catalog... it's time to look at the non-Lightroom-specific things:

Reboot with all non-essential services disabled..., or try a new graphics card...

Rob