Experiencing performance related issues in Lightroom 3.x

[laurencec]

Hi

I just upgraded from lightroom 2.7 to lightroom 3. I then proceeded to import my old catalog. this all went fine but lightroom is so slow, the thumbnail previews take forever to load if I manage to have the patience to wait  for them.

is there a quick solution?? How can it be sped up?

thanks

Laurence

Message title was edited by: Brett N

[JayS_In_CT]

areohbee wrote:

Dan,

I assume you realize that there are two kinds of problems in Lightroom 3.0:

1. Those making performance sub-optimal / noticeably slower than the betas or than 2.X - disappointing but still useable.

2. Those that are so bad it makes the product practically unusable. e.g. dust-spot tool that's so slow to respond to the mouse that you can't even get it positioned over the dust spot...

I hope Adobe is concentrating their efforts on solving type 2 problems first.

Rob

Rob,

+1 for the above.

Jay S.

[JayS_In_CT]

DanTull wrote:

> I think you're trying to say that in your last paragraph when talking about floodgates.

Yes. LR 2.x had some of the same sensitivities, but it had some bottlenecks that tended to throttle everything back. On one hand, that meant it went slower than it could on some systems, but in trade it tended not to flood itself under so severely because artificial internal contention restricted other kinds of resource contention between subsystems. Actually we had to work through analagous issues of throttling back certain behaviors to avoid paralyzing feedback effects even in LR 1 and 2, but the revamp in LR 3 opened up some new ones by being more distributed across threads.

There are some cases where things were accepted as being slower in order to make them more responsive (and provide progress indicators). Some people will say that "feels" faster due to the responsiveness, but some notice the walltime slowdown and don't like it. The develop rendering is shunted off to a background thread so the sliders aren't as "chunky" but it adds a little bit of extra latency to the adjustment rendering. Which is better is subjective at that point.

More tradeoffs:

Cache things to avoid needing to go back to disk or re-doing work you've already done, but not too much or you'll swap (swap is the performance kiss of death). Pre-compute some things so they'll come up faster when the user asks for them, but not at the expense of what they're asking for right now. Check that things on disk haven't changed externally so there's not a long lag before the changes show up in LR, but not so much that we never quiesce or rebuild things unnecessarily.

Anyway, now I'm just rambling, but you get the idea.

DT

Dan,

Kinda yes and no...  Also, I (we collectively) hope that you are not the single conduit for sorting through everything in this thread and trying to feed it back to the Adobe team..  Clearly some of these issues are already well known by now in Adobe and that we're not just now starting a process of trying to, as I said, single thread the things in this (and other LR 3 issue threads) through you..  You will be a very busy and tired person very quickly.

From your above note though, I'll just pull one example... where you talk about the sliders being "chunky" in LR 2.  That was true VERY early on, but the sliders and imaging rendering in 2.7 were fine..  With that as an expectation, I would expect no degradation in the timing of an adjustment slider to seeing it happen real time in LR3.  Remember, Adobe strongly pushed LR 3 as rich in new function and superior in performance.  What you are indicating, and I apologize if I'm picking up the wrong signals, are compromises in some places, not balancing.  I think the general gist of your note is talking about the internal structural balances Adobe is looking at e.g. balance the new function with the performance while supporting an incredible matrix of hardware, OSs, 3rd party software, etc.

All I wanted to clarify was that there are obvious "bugs" in LR3..  We all expected that (to more or less extent).. some of those may be lending themselves to the performance issues, but again, using your adjustment rendering example.. we know there are issues there.  Too many times the rendering just gets "stuck" and doesn't complete.  Multiple people have reported it in this thread.  When that one gets fixed, will the user "performance" experience of adjustment to seeing it rendered be fixed as well?  We don't know... and that is as much a part of the frustration as anything.  At this point we don't know what Adobe is recognizing as bugs vs. impact on performance, etc.

Again, appreciate your being here, and appreciate continued information.

Jay S.

[DanTull]

> ... you are not the single conduit ...

Well, not single, but you might be shocked to know how small the LR team really is...

> Clearly some of these issues are already well known by now...

We have cornered a few issues (slight mistakes in cache sizing that caused memory load to be higher than was beneficial, some extraneous rendering in some interactive adjustment -- mostly painting, especially at 1:1 view -- scenarios, etc), but I don't believe that they fully explain the menagerie of issues described in this thread. It's really hard to say for sure. For those that were seeing swapping and excessive memory use, they may find their woes mostly resolved if they tried the updated version, or it may reveal new (hopefully less severe) pain points hiding beneath.

> ... e.g. balance the new function with the performance ...

An example is the new process version. It is slower. It takes more CPU horsepower, especially if you're applying noise reduction. That cost drove changes elsewhere to try to "make room" for that extra work and mitigate the costs it incurs. Some people find the result smoother (they might even say "Hey, Lightroom is faster!"), but while there were aspects that were tuned, there are aspects that are objectively slower.

> Too many times the rendering just gets "stuck" and doesn't complete.

Hmm. That sounds like worker pool saturation. I've seen stalls and (long) hitches (especially on low end hardware), but I haven't been able to reproduce an out and out "stuck" condition that never resolves. Resolving such a bug would certainly help, though I expect the time it takes from adjustment to final rendering may still be longer than one might hope.

DT

[goodlux7]

This is the issue I'm having now...I get the jaggies, the the "loading..."

freeze, then LR hangs on that particular image for an random amount of time.

If I click another image in the filmstrip, the image does not change. This

could last 2 seconds or 30 seconds....or longer.

It seems like lightroom is first "looking" at the catalog preview, getting a

rough image, then "looking" for data from the DNG file. Maybe the worker

processes are hanging because data isn't returned fast enough over the

firewire connection?

The way to replicate this would be to set up LR with a relatively fast

machine (a fairly new iMac). Then put a bunch of DNG files on a USB or

Firewire drive. Import them into a catalog, keeping the files in their

location, and you should have the same problem when scrolling through the

catalog.

Too many times the rendering just gets "stuck" and doesn't complete.

Hmm. That sounds like worker pool saturation. I've seen stalls and (long)

hitches (especially on low end hardware), but I haven't been able to

reproduce an out and out "stuck" condition that never resolves. Resolving

such a bug would certainly help, though I expect the time it takes from

adjustment to final rendering may still be longer than one might hope.

[JayS_In_CT]

DanTull wrote:

> ... you are not the single conduit ...

Well, not single, but you might be shocked to know how small the LR team really is...

Well, not surprised but not totally comforting either..  There are a lot of things people are experiencing, so I don't envy your role right now.  If there are things and ways that can help, let the communicity know.

> Clearly some of these issues are already well known by now...

We have cornered a few issues (slight mistakes in cache sizing that caused memory load to be higher than was beneficial, some extraneous rendering in some interactive adjustment -- mostly painting, especially at 1:1 view -- scenarios, etc), but I don't believe that they fully explain the menagerie of issues described in this thread. It's really hard to say for sure. For those that were seeing swapping and excessive memory use, they may find their woes mostly resolved if they tried the updated version, or it may reveal new (hopefully less severe) pain points hiding beneath.

Good to know that some things are found already, with hopefully more to come.  I did report this morning an issue where LR 3 in 64 bit more (on Macbook Pro) is not releasing memory after an operation, whereas the same operation done in 32 bit mode shows a decline in the memory footprint as soon as it is completed.  In 64 bit mode, the app has to be restarted to get back to the base load level.

> ... e.g. balance the new function with the performance ...

An example is the new process version. It is slower. It takes more CPU horsepower, especially if you're applying noise reduction. That cost drove changes elsewhere to try to "make room" for that extra work and mitigate the costs it incurs. Some people find the result smoother (they might even say "Hey, Lightroom is faster!"), but while there were aspects that were tuned, there are aspects that are objectively slower.

Well you can readily see the impact there by turning off the "Detail" and seeing how much more responsive LR 3 is.  It is definately a contributor.  That said, it also needs to be a prime area for Adobe to look at trying to get a better balance of the impact it has on the rest of the product.  Not everyone is running 8 or 16 cores.  It may mean reworking some areas of the new process to "skinny" down the overhead.  Since it is the newest addition, I imagine it is also a target for optimization.

> Too many times the rendering just gets "stuck" and doesn't complete.

Hmm. That sounds like worker pool saturation. I've seen stalls and (long) hitches (especially on low end hardware), but I haven't been able to reproduce an out and out "stuck" condition that never resolves. Resolving such a bug would certainly help, though I expect the time it takes from adjustment to final rendering may still be longer than one might hope.

Fully admit that it doesn't happen all the time, but enough folks have hit it.  Again, just from the randomn threads here about running second monitors, higher resoluitions, etc. as Rob said in his post so well, there are 2 types of issues.  This is not a time to final rendering issue.. the image just doesn't resolve.

DT

Again.. thanks Dan...

Jay S.

[DanTull]

One other note -- I'm going to be offline tomorrow (Friday) and most of the weekend. So if there's a reply lag, that's what's up.

If there are things and ways that can help, let the community know.

Seems like it might be worth spinning off one new forum thread for each issue that has distinct symptoms. Some that come to mind are:

- The slow loading of grid thumbnails issue (I have at least some ability to reproduce this one, so I'm less concerned about it).

- Excessive memory consumption (to the point of swapping) issues, especially on 64-bit

- Issues where the cursors for tools (spot and localized correction brush, IIRC) are slow even if no adjustment is being applied.

- General Develop slowdowns since Beta2 (especially crop and localized corrections).

- Excessive CPU usage (preview building?) that doesn't readily idle down.

Posts to those threads should essentially be of the form:

I see this issue, here's the contents of my "System Info" dialog (in the help menu) in Lightroom. In addition, information about the kinds of photos (DNG, Raw, JPEG, Camera Make/Model, # of MP), whether they're on internal or external drives, resolution and number of monitors wouldn't hurt to include. Bonus points if there are videos of the issue occurring, along with monitoring tools (task manager, activity monitor, or event loggers) in view of the capture. It's amazing how often a video provides a clue that doesn't come out otherwise.

For those not seeing a given issue that are brave and motivated to help, feel free to _try_ to replicate the results of the unfortunate ones. Scientific rigor is really useful here. Change one thing at time. If it makes the problem happen change it back and see if it goes away. Repeat  until you're sure that it's not a fluke.

Ideally, these threads should have a bit less chatter, but obviously there's only so much that can be done to control that.

In all these cases, some of the things to try:

- Reducing/increasing monitor resolution or window size.

- Switching off panels in Develop or using the 2003 process version.

- Smaller/larger images (also JPEG versus Raw or different raws).

- Automatically write XMP on/off.

- Anti-virus and disk indexing software on/off.

- Digitizer tablets enabled/disabled.

- Reproduce the issue in the most minimal catalog you can.

DT

[Nick Walker]

Dan,

Your input is appreciated, for me good customer relations is as important as the product. I have been a professional sports photographer for 13 years and during this time owned various computers and cataloguing/software including news media targetted cataloguing software that some photographer's may not have heard of or used. I currently use 12 and 24 megapixel cameras. I only capture RAW (NEF) files and upgraded to V3 for its superior 2010 demosaicing algorithms, noise reduction and Adobe's claims of dramatically accelerated performance. I have been using Lightroom from V1, also the betas.

On my machine* Lightoom V3 is snappy enough in the grid mode and general UI but after a few edits in the develop module it crawls along and stops - especially the spot removal and brush tool. I was recently repairing a RAW image (14 bit capture - NEFF recorded as 48.44MB), backlit scene with many midges to remove when the spot tool repeatedly froze after half a dozen repairs or so. As I progressed onto the brush tool matters got worse; freezing (beech ball) for several seconds and then minutes before I could apply further edits - I had to quit the programme, it was that painful.

*Mac OS X 10.6.4 Mac Pro 2 X 2.8 GHz Quad-Core Intel with 14 GB of RAM. Stock Radeon HD ATI 2600 XT graphics card updated with latest firmware.

No other programmes running in the background. Three MacPro internal hard drives and plenty of space. Only use external rack system to back up data, not Lightroom tasks.

Lightroom V3 64 bit mode.

Only one catalogue (regularly optimised) all images at 1:1 previews.

One folder in this catalogue, containing mainly TIFF files, has 19677 images and another folder containing only RAW images has 15365 images (lrdata file = 66.27GB) - The catalogue is 1.4 GB.

Cache pref at 50 GB - raising this from the default setting has made no noticeable difference.

Lightroom V2.7 was responsive in the develop module with Leopard (not Snow) with only 8 GB RAM - upgrading to Snow Leopard and 14 GB RAM has not impinged on V2.7's behaviour. V2.7 is using a similar sized catalogue and lrdata.

Since upgrading to Snow Leopard and moving from 8 - 14 GB RAM I can confirm (tested and timed) speed improvements for various Photoshop CS5 tasks, also some non Adobe software - this indicates that my hardware is performing within spec.

Lightroom V3 is a great product expect for productivity. I should be able to take Lightroom out of the box and use it to its full potential with my hardware spec.

[JayS_In_CT]

Nick Walker wrote:

Dan,

Your input is appreciated, for me good customer relations is as important as the product. I have been a professional sports photographer for 13 years and during this time owned various computers and cataloguing/software including news media targetted cataloguing software that some photographer's may not have heard of or used. I currently use 12 and 24 megapixel cameras. I only capture RAW (NEF) files and upgraded to V3 for its superior 2010 demosaicing algorithms, noise reduction and Adobe's claims of dramatically accelerated performance. I have been using Lightroom from V1, also the betas.

.........

Nick,

Good post.  A couple of things to try.  Flip LR 3 to 32 bit mode (from the Information panel) and see if that helps some. I, and others, are getting better RAM behavior in LR3 (counter to what it should be I know).  Also, several folks, including myself, have noted issues with 10.6.4.  I had a call with an Apple tech who admitted it was causing grief in a number of areas and help me roll back to 10.6.3.  Bandaids I realize, but perhaps it will help till Adobe pins some of this down.  Also, Dan asked we start up more specialized threads, but not sure how to classify yours..

Jay S.

[Nick Walker]

Jay,

Ta!

I haven't tried 32 bit mode and will give it a whirl.

[calm_Light157F]

Dan,

Rendering +1400 images from a Canon 7D takes in LR2.7 30 secs on 10 images. With 2003 process in LR3 it takes 33 secs on the same images. The 2010 process takes 50 secs on these 10 images.  The scanning done before starting the build of previews (1680 pixels, hight quality) takes 1-2 secs on  1400 images in LR2.7 and  30 secs on LR3!  That is strange, isn't it?

I have one image that raises my memory use up to 3.8Gb when continuing editing it. It is a bit dark 7D images at iso 1600.  I have notices that at iso 100 LR3 works a bit better. I can send this "bad" image to you if you want it.

I am running win 7 home premium 64 bit. 4 GB mem, Q6600 cpu clocked at 3.2GHz, internal disk at 7200 rpm, virus checker is microsofts, graphic card in ASUS ENGTX275.

I have reduced screen size to work faster so I can work somehow fast now. Worst thing now (very annoying too) is the filters setting being changed from LR2.7 to LR3 but that another thread about sticky filters.

- Terje

[JayS_In_CT]

teho59 wrote:

Dan,

Rendering +1400 images from a Canon 7D takes in LR2.7 30 secs on 10 images. With 2003 process in LR3 it takes 33 secs on the same images. The 2010 process takes 50 secs on these 10 images.  The scanning done before starting the build of previews (1680 pixels, hight quality) takes 1-2 secs on  1400 images in LR2.7 and  30 secs on LR3!  That is strange, isn't it?

I have one image that raises my memory use up to 3.8Gb when continuing editing it. It is a bit dark 7D images at iso 1600.  I have notices that at iso 100 LR3 works a bit better. I can send this "bad" image to you if you want it.

I am running win 7 home premium 64 bit. 4 GB mem, Q6600 cpu clocked at 3.2GHz, internal disk at 7200 rpm, virus checker is microsofts, graphic card in ASUS ENGTX275.

I have reduced screen size to work faster so I can work somehow fast now. Worst thing now (very annoying too) is the filters setting being changed from LR2.7 to LR3 but that another thread about sticky filters.

- Terje

Terje,

Sounds like my experiences with 2.7 and 3.0. The increase in load time even with 2003 settings is indicative that it isn't all a 2010 process issue I think.  Maybe this should be a specific Load/Render time thread like Dan suggests?

Jay S.

[imajez]

DanTull wrote:

Seems like it might be worth spinning off one new forum thread for each issue that has distinct symptoms.

Dan, maybe you could start each of the threads you require, with pointers of how to give feedback and do testing, as then they will have a bit more authority and credence and hopefully keep them more focused than if we were to do it.

Bonus points if there are videos of the issue occurring, along with monitoring tools (task manager, activity monitor, or event loggers) in view of the capture. It's amazing how often a video provides a clue that doesn't come out otherwise.

When Beta testing CS5 I [and many other testers] found iShowU HD to be extremely useful in video screen capturing and clearly demoing some issues, that are hard to describe in writing.  This is a Mac programme only. Never tried any video capture on PC, so cannot recommend anything.

http://store.shinywhitebox.com/ishowuhd/main.html

[Butch_M]

Imajez

On Mac, you don't need extra software ... there is screen capture recording built within QuickTime X ... though you need to be using 10.6.3 ... as earlier versions did not recognize some custom monitor color profiles ....

[imajez]

Butch_M wrote:

Imajez

On Mac, you don't need extra software ... there is screen capture recording built within QuickTime X ... though you need to be using 10.6.3 ... as earlier versions did not recognize some custom monitor color profiles ....

Macs also come with iPhoto. I don't use that either as I prefer to use more advanced tools!

[Butch_M]

imajez wrote:

Butch_M wrote:

Imajez

On Mac, you don't need extra software ... there is screen capture recording built within QuickTime X ... though you need to be using 10.6.3 ... as earlier versions did not recognize some custom monitor color profiles ....

Macs also come with iPhoto. I don't use that either as I prefer to use more advanced tools!

That's all quite cute ... I also don't use iPhoto ... don't think I have ever started it up even .... but why should someone have to buy software just to show you or anyone else their LR problems when they have a built-in function that serves the purpose?

Having discriminating taste is one thing ... insulting others offering a possible no-cost solution is another ....

[imajez]

Butch_M wrote:

imajez wrote:

Butch_M wrote:

Imajez

On Mac, you don't need extra software ... there is screen capture recording built within QuickTime X ... though you need to be using 10.6.3 ... as earlier versions did not recognize some custom monitor color profiles ....

Macs also come with iPhoto. I don't use that either as I prefer to use more advanced tools!

That's all quite cute ... I also don't use iPhoto ... don't think I have ever started it up even .... but why should someone have to buy software just to show you or anyone else their LR problems when they have a built-in function that serves the purpose?

Having discriminating taste is one thing ... insulting others offering a possible no-cost solution is another ....

There was an emoticon at end of my sentence for a reason, duh!

I wasn't suggesting anyone buy IShowU if they think Quicktime is fine, but I chose to do so as it is much better than Quicktime for demonstrating software problems. Just like I use LR for my images over free software like Picassa.

[Butch_M]

imajez wrote:

There was an emoticon at end of my sentence for a reason, duh!

Oh ... I forgot .... emoticons give you a free pass ... sorry my bad ....

[imajez]

Butch_M wrote:

imajez wrote:

There was an emoticon at end of my sentence for a reason, duh!

Oh ... I forgot .... emoticons give you a free pass ... sorry my bad ....

No they indicate that I'm not being entirely serious and are often used so as to try and avoid people pointlessly overreacting just like you did.

Don't be so oversensitive and remember that text written in forums can seem a bit harsher than they would in real life without the smile and apposite intonation you would have when speaking in jest. So, if emoticons are being used, pay attention, they are not random graphics.

[Jeremy 2]

Using the trial version of LR3 on XP with pentium 4 & 3GB RAM. (pentium 4 is specified as the minimum supported processor)

I downloaded about 30 photos from CF card (RAW & JPG).

the CPU ran at 99 or 100% for about 30 minutes & the fan was going crazy when it was trying to generate the thumbnails......

Once they were processed, it seemed to be fairly quick to use.

The same files imported into PSE6 in a frction of the time.

[DanTull]

What kind of camera were the photos from and what level of previews were you building as part of import? (note for reasons that I won't go into in detail that grid thumbnails will effectively drive standard previews to be built for all visible photos before it fully quiesces).

Also, were you importing with the option to treat JPEGs as separate files or were they just sidecars? (in the latter case, you've effectively doubled the rendering work, which is why I ask)

DT

[JayS_In_CT]

DanTull wrote:

What kind of camera were the photos from and what level of previews were you building as part of import? (note for reasons that I won't go into in detail that grid thumbnails will effectively drive standard previews to be built for all visible photos before it fully quiesces).

Also, were you importing with the option to treat JPEGs as separate files or were they just sidecars? (in the latter case, you've effectively doubled the rendering work, which is why I ask)

DT

Dan,

Can you cover that in more detail, the double rendering with sidecars vs. not with separate JPEGs?  Not sure I understand why sidecars would double the rendering.

Jay S.

[DanTull]

...were you importing with the option to treat JPEGs as separate files...

Can you cover that in more detail, the double rendering with sidecars vs. not with separate JPEGs?  Not sure I understand why sidecars would double the rendering.

When the "treat JPEGs as separate files option is enabled, an import of 30 RAW+JPEG pairs would instead import 60 total photos. Since the JPEGs get rendered through ACR as well and are generally the same resolution, it would (roughly) double the work involved in building out 1:1 previews.

DT

[JayS_In_CT]

DanTull wrote:

...were you importing with the option to treat JPEGs as separate files...

Can you cover that in more detail, the double rendering with sidecars vs. not with separate JPEGs?  Not sure I understand why sidecars would double the rendering.

When the "treat JPEGs as separate files option is enabled, an import of 30 RAW+JPEG pairs would instead import 60 total photos. Since the JPEGs get rendered through ACR as well and are generally the same resolution, it would (roughly) double the work involved in building out 1:1 previews.

DT

Dan,

O.K.. the reason I asked is cause I think you got it backwards when you replied earlier..   You said:

"Also, were you importing with the option to treat JPEGs as separate  files or were they just sidecars? (in the latter case, you've  effectively doubled the rendering work, which is why I ask)"

You said in the latter case (and sidecars was the latter case) you doubled the rendering work. 

So.. net RAW+JPEG double rendering, RAW with Sidecar faster.

Jay S.

[Jeremy 2]

DanTull wrote:

What kind of camera were the photos from and what level of previews were you building as part of import? (note for reasons that I won't go into in detail that grid thumbnails will effectively drive standard previews to be built for all visible photos before it fully quiesces).

Also, were you importing with the option to treat JPEGs as separate files or were they just sidecars? (in the latter case, you've effectively doubled the rendering work, which is why I ask)

DT

Sony A700 . files were cRAW & JPEG.

Don't know the answers to thte other 2 questions - I didn't specifically select anything other than the default. The only thing I selected was the directory name to save the photos in.

[Ian Lyons]

imajez wrote:

When Beta testing CS5 I [and many other testers] found iShowU HD to be extremely useful in video screen capturing and clearly demoing some issues, that are hard to describe in writing.  This is a Mac programme only. Never tried any video capture on PC, so cannot recommend anything.

http://store.shinywhitebox.com/ishowuhd/main.html

On Mac OS X 10.6.x you can use QuickTime X in screen capture mode.

BTW: you might want to review the NDA you signed for the CS5 program, and don't think your alias means that Adobe (and some others) don't know who you really are.