• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers

P: Better Develop Module Performance

45 Votes
LEGEND ,
May 22, 2011 May 22, 2011

Copy link to clipboard

Copied

On my machine (DELL E6400) adjusting the exposure of an image in the Develop module is more difficult than it should be because of the time lag between moving the slider and the image update.

I suggest a different strategy for generating visual feedback that yields better immediate performance, possibly trading in slight inaccuracies at first, but within a scheme that resolves the latter as soon as possible. Please see the end of this FR for the actual suggestion; I'd like to motivate it first.

It appears that the fixed order in which image operations are applied in the LR pipeline causes some adjustments to result in worse interactivity than necessary. Adjusting the exposure with no other (not even default) image adjustment settings is quick.

With some other image adjustments in place, however -- in particular when using the "Details" panel -- adjusting exposure results in jerky image updates that make it hard to find the correct level accurately and quickly.

I'm assuming that the LR image pipeline mandates that exposure updates happen relatively early, so that sharpening etc. always has to be recomputed after an exposure update. This is why updating the exposure is quick when no other adjustments have to be performed but slow otherwise.

Just to make sure that I'm making the correct assumptions, here's how I arrived at my conclusions:

My initial issue was that updates to the image in the Develop modules seemed to be slow and jerky in full screen mode whereas they were quick and led to smooth updates when I used LR in window mode. It turned out that the deciding factor is the size of the image. Images in portrait orientation are naturally displayed smaller and are less affected by the performance problems. Even images in landscape orientation can show smooth updates in full screen mode, if one artificially decreases their size by making the left hand side panels as wide as possible.

So it seems to me that:
a) LR struggles to deliver a decent image update frequency in the Develop module when the image is above a certain size (at least on my hardware).
b) This performance issue is a problem not only for computation intensive operations, like lens correction, but for simple ones like "exposure" because apparently potentially computation intensive operations, e.g., sharpening/noise reduction, come after exposure in the pipeline and have to be redone after each exposure update.
c) This is a big problem since I'd like to be able to apply some default sharpening on import (or even lens correction for some lenses) and not delay such settings after I have done other image adjustments in the Develop module.

It is clear to me that the user should not be forced to perform edits in a certain order (in addition to the aforementioned problems there is also a common advice to put off spot removals or lens corrections till the end) in order to address LR performance issues.

Here's the actual suggestion:

Could LR not cache all previous adjustments and apply the current image adjustment (e.g., exposure) as the last update in a relative manner?

I realise that this goes against the fixed ordering of the pipeline, but the latter is the problem causing the performance issues. When image operations are commutative they should be applied in the order which provides the best performance (with the use of caching, of course).

I also realise that some image operations (e.g., sharpening and exposure) only seem commutative but may not be completely commutative. It may matter (slightly) whether sharpening or exposure adjustments is performed first. Nevertheless, for the initial visual feedback, it should suffice in most cases to perform the current adjustment as the last and thus be able to capitalise on prior image caching.

Once the user stopped doing the adjustment, i.e., once there is no need to provide immediate visual feedback anymore, LR could recompute the current image using the standard image pipeline ordering. This may in some cases cause a subsequent, subtle change to the image after a short while, but I strongly believe I would prefer that if it meant that my current adjustments would just be a small delta to previous adjustments causing smooth updates rather than causing (almost?) the whole pipeline to be re-triggered.

Note that the proposed scheme would address the concerns regarding inaccurate image renderings in the Develop module. Noise reduction could always be performed, at all magnification levels, if it only impeded the speed of the final rendering -- after direct interactivity is no longer needed -- but would not impact on the interactivity of image adjustments that become before noise reduction in the image pipeline.

The proposed approach may even mean that using the adjustment brush on images with a high number of edits (including a multitude of previous brush strokes) may become more interactive. Currently, the lag between moving the mouse cursor and seeing the effect of the brush, becomes larger the more edits an image has. This may just point to caching potential regarding brushes (I hope not all strokes are always redrawn), but may also be related to the pipeline issue.

Idea Released
TOPICS
macOS , Windows

Views

65

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
45 Comments
LEGEND ,
Oct 09, 2012 Oct 09, 2012

Copy link to clipboard

Copied

|> " Most likely it is grounded in a (somewhat puritanical) mindset of ensuring Develop view only shows 100% genuine pixels or somesuch."

I can't help but think a part of the reason is that Lr's dev module is essentially a thin veneer over the same ACR used in Photoshop (which does not share the code to access Lr's previews...). So, there may be a ACR code-sharing thing too... - just guessing.

|> " I do know that DNG files created with more recent versions of Lightroom embed a proxy preview (I don't recall the name) that can accelerate Develop view loading significantly."

Could the name be: "fast-load data"? My experience: marginal improvement. about the same amount as ACR cache (~15% faster), making me think it is just housing the ACR-cache data internally. Again, this is speculation, albeit based on experience...

Votes

Translate

Translate

Report

Report
LEGEND ,
Oct 09, 2012 Oct 09, 2012

Copy link to clipboard

Copied

|> " I'll give the DNG a try..."

My experience: raw DNGs (w/fast-load data) load for develop no faster than raw NEFs (with ACR cache entry). But, that's just one data point. Please do report back after you try it.

|> " I had tried it in prior versions but found a ton of DNG corruption across a year's worth of work files..."

That's disturbing. Do you have any idea what caused this? Were they stored on optical media or a wonky disk or something? (it's very rare for corruption to be caused by Lr writing the DNGs).

Rob

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 09, 2012 Oct 09, 2012

Copy link to clipboard

Copied

The DNGs were stored on a Drobo so stored redundantly on many disks at once.
// Sean //

Mac Studio Ultra • 128GB RAM • 48-core GPU • 2TB SSD • 32" LG OLED • Nikon

Votes

Translate

Translate

Report

Report
LEGEND ,
Oct 09, 2012 Oct 09, 2012

Copy link to clipboard

Copied

So, any idea what caused the corruption? - any reason to believe if you try again the same thing won't happen again?...

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 09, 2012 Oct 09, 2012

Copy link to clipboard

Copied

I'm going to try DNG again. I back up my 'virgin' NEF files to backblaze cloud and blu-ray before touching them in Lightroom.
// Sean //

Mac Studio Ultra • 128GB RAM • 48-core GPU • 2TB SSD • 32" LG OLED • Nikon

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 10, 2012 Oct 10, 2012

Copy link to clipboard

Copied

I concur with Rob-- the DNG development responsiveness was no faster than NEF. In fact, I am seeing a similar delay with JPG development...

Camera: 16MP Nikon D4
Computer: Mac Pro 2010, 3.33GHz 6-core, 24GB RAM, Catalog on 240 OWC Extreme SSD (550 read/write uncompressed sequential on PCIe 6G SATA), images on 3TB Seagate Barracuda (150 read/write)

Full screen 2560x1440, just dev panel open about 350px wide on right, same 11 horizontal images
NEF = 1.3, 1.5, 1.3, 1.4, 1.3. 1.6, 1.4, 1.4, 1.3, 1.5, 1.5
JPG = 1.2, 2.3, 1.6, 2.0, 1.6, 1.9, 1.9, 1.8, 1.9, 1.8, 1.8
DNG = 1.4, 1.7, 1.9, 2.0, 2.3, 3.1, 2.2, 4.2, 2.3, 1.9, 1.9

So, DNG is actually a bit slower on my system (some cases much slower). DNG also continues to render the image while the sliders are available contributing to feedback lag between slider adjustment and image adjustment.

Now, compare that to a 1:1 cache pull in the Library module which is pretty much 0.1 seconds from sharp render to the next sharp render. Dan, I'm not seeing why a 1:1 jpg render in cache isn't the same as the Development Module render that's presented. What's the difference? I don't see any compression artifacts in either. I really, really want a preferences option to allow me to bypass that re-render. What about an option for those of us that could stuff 64-128GB of ram in our system to preload all of the NEF/DNG/JPG files into RAM? Wouldn't that make Lightroom fly?
// Sean //

Mac Studio Ultra • 128GB RAM • 48-core GPU • 2TB SSD • 32" LG OLED • Nikon

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 10, 2012 Oct 10, 2012

Copy link to clipboard

Copied

The Library case just has to scoop the bits off disk, decompress the JPEG, and blit assuming the preview is up to date.

The Develop case is actually grabbing the raw data (possibly short circuiting the early stages of the pipeline via the ACR cache, though that's not usually a huge savings, honestly) and performing the render right at that moment.

At your screen resolution, it's probably using the 1:1 pyramid level, too, which means any sharpening and noise reduction settings are more expensive to boot. Having now seen your resolution info now, I'm not too surprised that the DNG fast load isn't helping (sorry I didn't reply soon enough to save you the DNG experiment). That preview is capped at 2K on the long edge, IIRC, so in your case it's probably falling back to doing a full render anyway.

There's another DNG load speed boost, but it only applies if you're reading them from a really fast disk (striped 10K raptors fast) and have lots of cores, but I got some mixed results if I varied the setup at all (it didn't work very well on Windows, either for whatever reason) and ran that experiment quite some time ago, so I wouldn't bank on that one, either.

The scenarios that might be interesting to compare would be:
- what's the load time if the images are on a faster disk?
- what's the load time if the images have default settings (no sharpening or noise reduction, especially)?
- what's the load time if you reduce your window size so that the on screen display is less than 1/4 of the total resolution of the image?

The latter two would indicate the problem is bound by the rendering, the former indicates that I/O is holding you up.

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 10, 2012 Oct 10, 2012

Copy link to clipboard

Copied

I moved 10 NEF files to my SSD which has a sequential, uncompressed read of about 500MB/s. Extremely fast drive, same or faster than a striped array of raptors especially with non-sequential stuff. No difference in the delay. This is all CPU bound. It does render faster with smaller screen sizes in both cases (SSD & 3TB barracuda spinning disk)

But I had another question before: What's the difference between the ACR cache version of the raw file that I made in the Library module and the render version that gets made when I choose the raw file in the Develop module? Is it a DNG versus a JPG? more data?

I'm basically asking this stuff because I do about 30 jobs a year that require 2,000-3,000 photos each so probably 100,000 images per year when you add in odd jobs. If there could be a way (LR 5) for me to queue up some job to build out these renders while I sleep so I can skip them while I edit, that would be HUGE time savings. That's what I do now for the 1:1 in the Library module for pruning, but you are alluding that this 1:1 cache build is ignored for some good reason when I hit the Develop module. I don't see why but I'm ignorant on the mechanics here.
// Sean //

Mac Studio Ultra • 128GB RAM • 48-core GPU • 2TB SSD • 32" LG OLED • Nikon

Votes

Translate

Translate

Report

Report
LEGEND ,
Oct 10, 2012 Oct 10, 2012

Copy link to clipboard

Copied

Thanks Dan, for participating and informing...

I have found that the time from selecting the next image (when not in negative/ram cache) until sliders enabled depends more on resolution than format, whereas time from sliders enabled until loading indicator extinguished depends more on format. This is in agreement with Sean's report above.

After my initial experiments with "fast-loading" DNG, (when I didn't know it was possible to reduce the resolution and still maintain raw editing characteristics), I abandoned the EditInLightroom project, since it wasn't helping with the most critical lag (time until sliders enabled). Moral of the story - it was the resolution drop that helps enable those sliders more quickly.

|> " If there could be a way for me to queue up some job to build out these renders while I sleep so I can skip them while I edit, that would be HUGE time savings."

This is precisely what EditInLightroom allows you to do (granted, you lose the ability to zoom 1:1). Is it not helping? Or, are you simply asking Adobe for similar help, built-in & smooth...

Rob

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 10, 2012 Oct 10, 2012

Copy link to clipboard

Copied

Adobe should have something similar, built-in & transparent to the user. At normal screen res, show me a proxy that's optimized for my workspace. If I zoom in, pull up the RAW and give me more res. By the time I'm in the Dev module, I've already made my picks so I don't actually do much pixel peeping at that stage. Others might, but that might be a flaw in their workflow. I've advocated face detection for intelligent 1:1 zooms, too. If this was applied, Lightroom could zoom in on the faces (not ID the faces, just ID that they are faces) and show me that subsample'd region of the raw, not the full raw with panning. Right?

Your software is brilliant. I used it on about 50 images, but the conversion back to hi-res didn't load the data. I had to manually load it. Also, I don't see that flags sync meaning that I can't do further filtering while developing my files which is quite frequent. Also, on the Hi->Lo, I get a LR dialog for each selected file saying "Lightroom is concerned". So, for a 2k set, you can imagine that not working out too well. I'm probably not doing it right, though.
// Sean //

Mac Studio Ultra • 128GB RAM • 48-core GPU • 2TB SSD • 32" LG OLED • Nikon

Votes

Translate

Translate

Report

Report
LEGEND ,
Oct 10, 2012 Oct 10, 2012

Copy link to clipboard

Copied

Sean,

I couldn't agree more - improvement should be built-in. But, seeing as we've now gone over half a decade without such improvement, it seems worthwhile to consider workarounds for the mean time. It might be another half decade before Lr has such improvement... Anyway, many plugins that push the envelope require some reading of "the fine print" too. But if your originals were DNG, then 'Update DNG Previews & Metadata" should trigger Lightroom to re-apply the lo-rez settings to the hi-rez version (along with flags...), after invoking the 'Edit Hi-Rez' plugin command.

Anyway, feel free to contact me outside the forum for plugin support.

Rob

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 10, 2012 Oct 10, 2012

Copy link to clipboard

Copied

I'm not certain about that timeframe.... At it's present incarnation, I could use Lightroom for 20 years. CPUs will become more powerful and the image files will become more detailed, but the functionality is pretty much there barring some cool trick tools. So, this is a perfect time to focus purely on optimizations and rethink and improve the engine.
// Sean //

Mac Studio Ultra • 128GB RAM • 48-core GPU • 2TB SSD • 32" LG OLED • Nikon

Votes

Translate

Translate

Report

Report
LEGEND ,
Oct 10, 2012 Oct 10, 2012

Copy link to clipboard

Copied

You're preaching to the choir (although I have a voracious appetite for some cool trick tools too). And, Adobe has heard you too, even if they don't respond. But it's going to be at least a year or so, at the earliest, and maybe longer, so it may be worth considering a strategy for the mean time. PS - I'm now guessing you were using EditInLightroom with proprietary raws (right?), - try with DNGs - it's smoother. PPS - Don't forget to check that "Don't show again" box when you've seen a prompt as much as you care to ;-}.

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 11, 2012 Oct 11, 2012

Copy link to clipboard

Copied

Well, how about the strategy of me driving down the street and knocking on their door? I live about 2 miles away from their office....
// Sean //

Mac Studio Ultra • 128GB RAM • 48-core GPU • 2TB SSD • 32" LG OLED • Nikon

Votes

Translate

Translate

Report

Report
LEGEND ,
Oct 11, 2012 Oct 11, 2012

Copy link to clipboard

Copied

I doubt they will be more likely to accelerate development in response to the personal nature of your visit - could be wrong. Got a plan B?

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 11, 2012 Oct 11, 2012

Copy link to clipboard

Copied

It's Seattle, I'd bring coffee and Top Pot Donuts for the whole office.
// Sean //

Mac Studio Ultra • 128GB RAM • 48-core GPU • 2TB SSD • 32" LG OLED • Nikon

Votes

Translate

Translate

Report

Report
LEGEND ,
Oct 11, 2012 Oct 11, 2012

Copy link to clipboard

Copied

Those must be some killer donuts ;-}.

I'd still like to talk with you about your experience using EditInLightroom. Even if you've given up already, it may help to improve it so others can use it successfully.

http://forums.adobe.com/people/areohbee

Cheers,
Rob

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 15, 2012 Oct 15, 2012

Copy link to clipboard

Copied

Almost lost the other question in all the noise (though I will note there aren't many folks in Seattle in a position to affect Lightroom Develop module performance regardless of bribery or threats ;o).

Anyway, getting back to the question for me before this thread ran off: "What's the difference between the ACR cache version of the raw file that I made in the Library module and the render version that gets made when I choose the raw file in the Develop module?" The difference is that the Library module is loading a fully rendered JPEG of the raw with settings applied. The ACR Cache doesn't store fully rendered previews so it still has to do the actual rendering work.

The cache does allow ACR to skip some expensive work toward the beginning of its pipeline, but the rest is done when the image is opened. Note also that previews are rendered into a color space that may not produce precisely the same results as you'll see on screen in Develop (which works in a wider gamut space, IIRC), so those previews aren't suitable for use in Develop. Lastly, the moment you actually started to move a control, ACR would have to do the setup and rendering anyway, so using a fast preview would just kick the can a bit further down the road, so to speak. It was viewed as better to delay enabling the controls and be responsive when they were used than to enable immediately and have them be initially sluggish.

That's not to say it couldn't be made faster, only that there are reasons for why it works as it does at the moment, so it's probably not simple or trivial to make it faster to load. Hopefully that's illuminating or at least helpful.

Votes

Translate

Translate

Report

Report
LEGEND ,
Oct 15, 2012 Oct 15, 2012

Copy link to clipboard

Copied

Dan, you write "...so using a fast preview would just kick the can a bit further down the road, so to speak. It was viewed as better to delay enabling the controls".

The advantage of kicking the can further down the road would be that one could stay in the Develop module when navigating between images.

Partly due to preview caches not being handled properly, navigating in the Develop module can be painful to the extent that I sometimes switch to the library module to locate an image (or force an update to the Develop module previews) and then switch back again.

AFAIC, it should never be necessary to leave the Develop module just because one wants to quickly flick through some images, to compare them or chose the next one for editing.

Votes

Translate

Translate

Report

Report
Contributor ,
Oct 15, 2012 Oct 15, 2012

Copy link to clipboard

Copied

LATEST
Dan, thanks for clearing that up. I really do appreciate the time you are taking to clue us in.

For those of us that prioritize performance over precision [a huge group I'd imagine], I would love the 1:1 previews to be initially in a color space appropriate for the Develop module. Why not? It's simple matrix math, eh?

TK is spot on, too. The reason I desire the quick preview and slider access is for those images that I don't even want to adjust. I've already applied the adjustments upon ingestion (sharpening/contrast/fill/highlight) and the 1:1 previews are either good or very close.

For one job, I prune down to 1000 images in the Library module. When I step into the Develop module, that's 1500-2000 seconds of pure render delay. That's 30 minutes per job times 30 jobs = 900 minutes per year [15 hours] (on the low end) I'm taping my finger while LR is rendering before I can determine I I simply want to move to the next photo. Let me state that again: that's 15 hours regardless if I want to make an adjustment or just move to the next photo. And, that's with my 6-core 3.33GHz machine. Most of my peers are using much slower machines.

So, pulling from the 1:1 versus the re-render would be one potent kicking of the can in my case.
// Sean //

Mac Studio Ultra • 128GB RAM • 48-core GPU • 2TB SSD • 32" LG OLED • Nikon

Votes

Translate

Translate

Report

Report