Lightroom 3.3 Performance Feedback

[Tom Hogarty]

Please use this discussion topic for your feedback on Lightroom 3.3 RC and the final Lightroom 3.3 release when it becomes available.  The Lightroom team has tried very hard to extract useful feedback from the following discussion topic but due to the length and amount of chatter we need to start a new, more focused thread.  Please post specifics about your experience and be sure to include information about your hardware configuration.

Regards,

Tom Hogarty

Lightroom Product Manager

[Sean McCormack]

TK, actually it does make a difference if you render 1:1 previews, because along with creating the Library Previews, it's also filling the Camera Raw Cache, so when you do go to Develop, you've cut the render time down. Yes, if you do a lot of edits, especially spotting, it's proportionally less, but responsiveness is a feel thing, and shaving a second off can help it feel faster. You may be working on 50 images, but I'm shooting a lot more than that, so yes, it does make a difference.

[areohbee]

Seán McCormack wrote:

... if you render 1:1 previews, along with creating the Library Previews, it's also filling the Camera Raw Cache ...

Thanks for the tip - I did not know that / did not expect that. That makes a difference in how I'll be handling 1:1 preview rendering in the future 🙂

PS - I've followed TK's detailed ACR Cache analysis in other threads, and I don't think he's under any illusions - he just thought it was being updated with full renderings when he first started out, as opposed to just the one-time demosaic baseline, and so it wasn't as much of a savings as he thought it would be...

I think TK was mostly trying to point out that the more edits you've applied, the less the ACR cache helps. And it may help less in Lr3 than it did in Lr2, due to the extra processing required for lens corrections, sharpening, and NR - the result of which is not saved in the (ACR) cache.

whitedog:  ACR = (Adobe) Camera Raw.  So ACR Cache = Camera Raw Cache.

TK: Please forgive if I've overly softened your points about the ACR cache while venturing to speak for you ;-}

Everyone: the proof is in the pudding - its easy enough to verify exactly how much the cached entries help - just try it with, then without (or vice versa) - the "it" being the measurement of develop-mode rendering (loading) time.

_R

[Sean McCormack]

I did agree with that Rob, in my post.

From both my own experience and that of my blog readers, it has helped others. It's certainly not a magic cure all.

If Dan Tull says he's working on specific areas that are showing issues that vary widely for different users, then I absolutely trust him on solving it.

[areohbee]

Seán McCormack wrote:

I did agree with that Rob, in my post.

Check.

I do think that the points both you and TK have made about optimal ACR cache size are relevant, to paraphrase:

     "The biggest improvement will come by making sure its big enough to include all the photos you'll work on in a shoot (or edit session). Diminishing gains by trying to make it big enough to span multiple shoots / multiple editing sessions.".

Quantifying: my cached entries are about 5MB, so if I were to edit 1000 photos in a session, I'd need 5GB cache to cover them all. I have it set to 50GB, which covers 10 such editing sessions - probably way overkill for me, but I've got the space... The 1GB default cache (if that's really what it is) would only cover a 200 photo edit session - so a pro(lific) shooter/editer could easily notice improved develop performance by increasing the default cache size. But, as TK has pointed out, possibly not as much improvement as you might think.

This aimed more at other readers than you Sean.

_R

[JayS_In_CT]

areohbee wrote:

Seán McCormack wrote:

I did agree with that Rob, in my post.

Check.

I do think that the points both you and TK have made about optimal ACR cache size are relevant, to paraphrase:

     "The biggest improvement will come by making sure its big enough to include all the photos you'll work on in a shoot (or edit session). Diminishing gains by trying to make it big enough to span multiple shoots / multiple editing sessions.".

Quantifying: my cached entries are about 5MB, so if I were to edit 1000 photos in a session, I'd need 5GB cache to cover them all. I have it set to 50GB, which covers 10 such editing sessions - probably way overkill for me, but I've got the space... The 1GB default cache (if that's really what it is) would only cover a 200 photo edit session - so a pro(lific) shooter/editer could easily notice improved develop performance by increasing the default cache size. But, as TK has pointed out, possibly not as much improvement as you might think.

This aimed more at other readers than you Sean.

_R

You know guys... I was blissfully thinking I had this all straight in my head, and now I find myself questioning what does what where and how??  🙂  Did I miss a point somewhere that 1:1 generated Previews on import play a role in the Camera Raw Cache file(s) and has some bearing on then Develop performance?  I just poked inside the Cache folder I have set up, and even though I've been working on a number of images, I'm not seeing any current dates??  I see a whole slew of cache entries in the file though on the day I swtiched from 1440 to 1680 on my Preview size by deleting the 1440 Preview file and generated a new set of Standard Previews at 1680..  so obvisouly, that effort loaded the Camera Raw Cache with some baseline?

I think I'm confused again...  🙂

Jay S.

[areohbee]

JayS In CT wrote:

Did I miss a point somewhere that 1:1 generated Previews on import play a role in the Camera Raw Cache file(s) and has some bearing on then Develop performance?

Yes - Sean pointed this out, I did not previously know this either. Delete your ACR cache then render 1:1 previews and watch your ACR cache refill...

I just poked inside the Cache folder I have set up, and even though I've been working on a number of images, I'm not seeing any current dates??

This is what TK was talking about - ACR cache entries are never updated, since they are just a "bare minimum" conversion of raw data to rgb. No develop edits will ever be reflected in the cached entries.

Clear as mud?

_R

[JayS_In_CT]

areohbee wrote:

JayS In CT wrote:

Did I miss a point somewhere that 1:1 generated Previews on import play a role in the Camera Raw Cache file(s) and has some bearing on then Develop performance?

Yes - Sean pointed this out, I did not previously know this either. Delete your ACR cache then render 1:1 previews and watch your ACR cache refill...

So, that also explains the large number of entries in the cache from when I reloaded at 1680.. so if this is true and they aren't updated, what is the purpose of the "purge cache" function?  If it's purged, is the only thing in there then anything new that is imported?  Does that mean that anything that "was" in there no longer has the baseline entry?

I just poked inside the Cache folder I have set up, and even though I've been working on a number of images, I'm not seeing any current dates??

This is what TK was talking about - ACR cache entries are never updated, since they are just a "bare minimum" conversion of raw data to rgb. No develop edits will ever be reflected in the cached entries.

Clear as mud?

Yeah.. now I've no clue why we've been saying to have a large cache if Develop never reads it, or does it read the baseline then add the edits..  Head spinning... lights growing dim....     🙂   I mean the whole purpose of a cache (as I've ever known them) is to able to stage something commonly used and to speed up access to it again.. 

_R

[Sean McCormack]

and whatever you look at in develop

[areohbee]

JayS In CT wrote:

.. so if this is true and they aren't updated, what is the purpose of the "purge cache" function?  If it's purged, is the only thing in there then anything new that is imported?  Does that mean that anything that "was" in there no longer has the baseline entry?

Under normal circumstances, there is no reason to purge the cache, since as Sean pointed out, the oldest entries will be overwritten to make way for new ones...

Similarly, under normal circumstances, there is no reason to purge the library previews.

My guess is that purging is more for testing and problem solving and for recovering storage when you need it for something else... i.e. not beneficial to do as a regular maintenance chore.

Entries are added to the ACR Cache whenever you edit a photo if an entry is not already there.

JayS In CT wrote:

.. now I've no clue why we've been saying to have a large cache if Develop never reads it, or does it read the baseline then add the edits..

The latter is precisely correct - the develop module looks for a cache entry, and if found reads it and then combines the edits from the catalog database for the final image presentation. If the cache entry does not exist, then the raw data is read from the original photo file, and converted to rgb, then written back to the cache for next time.

PS - And, its because the time to convert from raw data to rgb is only a "small-ish" part of the total time to render a photo, that the usefulness of the cache is limited.

Gettin' better or worse?

_R

[JayS_In_CT]

areohbee wrote:

JayS In CT wrote:

.. so if this is true and they aren't updated, what is the purpose of the "purge cache" function?  If it's purged, is the only thing in there then anything new that is imported?  Does that mean that anything that "was" in there no longer has the baseline entry?

Under normal circumstances, there is no reason to purge the cache, since as Sean pointed out, the oldest entries will be overwritten to make way for new ones...

O.K., so then if we have large cache sizes, are we running some risk of added time to fetch the baseline image?  Would it be better to have a somewhat smaller cache (maybe not 1GB), but something that is refreshed more frequently with items currently being worked on vs. having a cache large enough to hold thousands of images, but takes longer to look up?

Similarly, under normal circumstances, there is no reason to purge the library previews.

My guess is that purging is more for testing and problem solving and for recovering storage when you need it for something else... i.e. not beneficial to do as a regular maintenance chore.

I have no idea Rob.. It's pretty conspicuous to be there only for problem solving.

Entries are added to the ACR Cache whenever you edit a photo if an entry is not already there.

It seems like it would be added on import, unless we're talking about an image loaded back as the result of a plug-in or if it rolled off I guess.

JayS In CT wrote:

.. now I've no clue why we've been saying to have a large cache if Develop never reads it, or does it read the baseline then add the edits..

The latter is precisely correct - the develop module looks for a cache entry, and if found reads it and then combines the edits from the catalog database for the final image presentation. If the cache entry does not exist, then the raw data is read from the original photo file, and converted to rgb, then written back to the cache for next time.

PS - And, its because the time to convert from raw data to rgb is only a "small-ish" part of the total time to render a photo, that the usefulness of the cache is limited.

Gettin' better or worse?

I understand what you're saying here.. just not sure it is the most efficient use of a cache, but then what is the purpose of the cache in ACR which is not used normally as a repository.  Normally, ACR is get in, do the edits and move it out as some processed file.  So we're up to clear as.. watered down mud.  🙂

_R

Jay S.

[ChBr02]

I've been with Lr since 1.0.  Suffred a real set back going from 2.7 to 3.0.  It was totally unusable for me.  3.2RC/3.2 made some improvement.  Haven't fully wrung out 3.3 yet, but it shows some promise.  Sliders are almost as fast as 2.7.  I have two very different platforms.  I won't bore you with the details at this point.

I do have a couple of questions:

1) I mostly shoot in Hi-Res jpg's. Please don't give me a lecture on RAW (NEF) versus compressed.  What I need to know is whether or not the Camera Raw Cache is of any use whatsoever when you are only editing jpg's?

2) What element (catalog, original photos, virtual memory, programs, etc.) would benefit the most from the fastest DASD available.  On one platform, I have no choice, but on the other, I could upgrade to SSD, use available 10K WD drive mirror, etc.  I can always add an eSATA 2 drive to ether platform.

3) What would be the benefit to ugrading from 6GB to 8GB RAM running Windows 7 Ultimate?

[TK2142]

ChBr02 wrote:

What I need to know is whether or not the Camera Raw Cache is of any use whatsoever when you are only editing jpg's?

No use whatsoever. You can easily verify this by looking into the cache directory. If you never deal with RAW files it will be empty and will stay that way.

In my book, there should be a limited cache for the fully rendered versions, i.e., a temporary cache for 1:1 previews. Not everyone wants to dedicate the harddisk space to have all images available through 1:1 previews but for a small working set of images it would make 100% sense to cache the most recent versions. This would allow quick back- and forth flipping between two images in the Develop module, something I like to do when doing comparisons.

[Sean McCormack]

You do have the option to set the 1:1 discard time down to one day, making it practically a temporary cache.

[thewhitedog]

Since the discussion seems to be drifting back and forth between the subjects of the Camera Raw Cache and previews, I'll take this opportunity to ask for clarification about the difference between the Lightroom 3 Catalog Previews.lrdata file, which is stored on my (Mac) system in the ~/Pictures/Lightroom/Backups folder in a dated folder and the Lightroom 3 Catalog.lrcat file in the same folder? Even though these two files are in the oldest folder, they are dated yesterday in the Date Modified column - the last time I used Lightroom. The Previews file is approximately seven times larger than the Catalog file.

In addition to being curious about the nature of the data in these two files, the reason this seems important is that it is the proliferation of subsequent backups of the Lightroom 3 Catalog.lrcat file that is really eating up space on my hard drive. When is it safe to delete files in the Backups folder and which ones should not be touched?

[JayS_In_CT]

thewhitedog wrote:

Since the discussion seems to be drifting back and forth between the subjects of the Camera Raw Cache and previews, I'll take this opportunity to ask for clarification about the difference between the Lightroom 3 Catalog Previews.lrdata file, which is stored on my (Mac) system in the ~/Pictures/Lightroom/Backups folder in a dated folder and the Lightroom 3 Catalog.lrcat file in the same folder? Even though these two files are in the oldest folder, they are dated yesterday in the Date Modified column - the last time I used Lightroom. The Previews file is approximately seven times larger than the Catalog file.

In addition to being curious about the nature of the data in these two files, the reason this seems important is that it is the proliferation of subsequent backups of the Lightroom 3 Catalog.lrcat file that is really eating up space on my hard drive. When is it safe to delete files in the Backups folder and which ones should not be touched?

thewhitedog,

I don't have a Previews lrdata file in the Backups folder, just the Catalog lrcat file dated with when I did the back-up.  I'm assuming once you do a back-up you can delete the prior entries, unless you want to keep an extra one or so in case the current should fail for some reason..

Jay S.

[thewhitedog]

Wow, what happened to your Previews file? Did you move it somewhere else?

And "I'm assuming" is not good enough for me, no offense intended. I assume the same thing, but until I have a definitive answer I'm not acting on my assumption or yours.

[JayS_In_CT]

thewhitedog wrote:

Wow, what happened to your Previews file? Did you move it somewhere else?

And "I'm assuming" is not good enough for me, no offense intended. I assume the same thing, but until I have a definitive answer I'm not acting on my assumption or yours.

thewhitedog..

I'm not sure if you're replying to my answer about the backup folder or not???  My Previews lrcat is there with my standard catalog in the Pictures folder.  There is no Previews Lrcat in the Backup folder. I thought I read in your message that space was getting eaten up and that you have Preview Lrcats in the Backup folder.  I don't.. just a backup of the Catalog lrcat..

Jay S.

[JayS_In_CT]

Seán McCormack wrote:

You do have the option to set the 1:1 discard time down to one day, making it practically a temporary cache.

Sean,

O.K., so I just did a "build all" standard size preview and the scan found 10 or so images that needed a 1620 size build.  Now when I go to the cache, I see the index was updated, but there is no new cache file entry dated today..

Are you saying that a 1:1 build on previews will somehow build a cache file set of entries that would help performance even after the 1:1 files are gone?  If I do build 1:1 previews on top of the 1620 size ones, and then the 1:1 are deleted at the end of the day, am I without any preview size being available for Library?

Jay S.

[JayS_In_CT]

All,

O.K., let's leave cache alone for a minute.  🙂  Opinions desired please.  I'm upgrading my internal Macbook Pro drive from 200GB to 500GB with a new drive, better performing, gives me some additional room to work with, etc.  Right now I've got LR Catalog and previews lrdata on Boot drive, Cache on an external eSATA, images on eSATA RAID 0.  Is there any benefit to carving out a separate partition on the new internal drive and putting the cache there instead (guess I wasn't leaving cache alone)  🙂   ..   Really a question of heads still moving on a single internal drive vs. coming through eSATA link...  Just curious if anything thinks one way is better (or worse) than another.  Thanks!

Jay S.

[Jim Stamates]

Jay, I have same MBP and upgraded to a 500gb a few months ago. I didn't partition and I have the ACR cache on that drive along with cat and previews. all images reside on external drive except when I'm in the field. When I return I move images to external. So far I'm pretty happy with the performance of LR 3.3 an haven't noticed too much lag time. Way better than LR 3.

As another note about ACR cache, I set mine to 150gigs just thinking that it would only use what was necessary to adjust images. I recently found the folder and it was 133 gigs in size with over 10,000 files some dating back to July.  Then after reading these posts I reduced it to 50 gigs. I didn't purge the cache but it is now about 53gigs in size with about 4000 files and still has some files from July. Odd.  Each file is around 12mb.

The biggest plus changing the cache size is it freed up hard drive space. I have no experience yet on if it slowed down develop on older images.

[JayS_In_CT]

Jim Stamates wrote:

Jay, I have same MBP and upgraded to a 500gb a few months ago. I didn't partition and I have the ACR cache on that drive along with cat and previews. all images reside on external drive except when I'm in the field. When I return I move images to external. So far I'm pretty happy with the performance of LR 3.3 an haven't noticed too much lag time. Way better than LR 3.

As another note about ACR cache, I set mine to 150gigs just thinking that it would only use what was necessary to adjust images. I recently found the folder and it was 133 gigs in size with over 10,000 files some dating back to July.  Then after reading these posts I reduced it to 50 gigs. I didn't purge the cache but it is now about 53gigs in size with about 4000 files and still has some files from July. Odd.  Each file is around 12mb.

The biggest plus changing the cache size is it freed up hard drive space. I have no experience yet on if it slowed down develop on older images.

Jim,

Thanks.. I know I can always go back and try something different if I want.  I've had my ACR cache set to 60GB.  I've done similar with images while away from home (e.g. vacation or a shoot that I need to show someone immediately) ..  Just an easy task to relocate the files in the catalog once they're moved.  I guess I'll try with the cache in both locations.  I just didn't know if there was some inherent additional value by paritioning that section off in terms of how the OS treats accessing the data (and the application I suppose).

Jay S.

[thewhitedog]

My advice is to leave your new boot drive at one partition. I used to partition my drives; I no longer do so - one partition or another was always filling up and I'd have to back up, reformat and start over. Another reason for partitioning was that maintenance on smaller partitions ran faster; this is no longer a significant factor, with faster drives, faster CPUs and system buses and faster software.

On the other hand, putting your Camera Raw cache on a second partition or even on the same partition will hammer your drive all the more. If, for instance, Lightroom is jumping between your Lightroom 3 Catalog.lrcat or Lightroom 3 Catalog Previews.lrdata files and the Camera Raw Cache it will have to stop one operation to complete the other; if the Camera Raw Cache remains on your external drive those operations can be done simultaneously, improving performance and reducing wear and tear on your boot drive. Your external drive setup sounds plenty fast as well, so there is, in my opinion, nothing to be gained, and much to lose, in moving your cache to your new internal drive, partitioned or not.

[Jim Stamates]

I did not know that. But how do you address 'in the field' time when the external hard drive is not with you. Do you create another cache folder then copy it to the external when you return? Seems like more stuff to remember.

Enlighten, please.

[JayS_In_CT]

Sean..

You looked like you were going to post something in response, but it came in blank..

Jay S.