Lightroom 4.2 very poor CPU usage

[Sebasvideo]

Lightroom 4.2 seems reasonably fast when I work with it, whether it's browsing photos or adjusting sliders, although it takes several seconds to go into develop mode after launching it for the first time.

But now I'm exporting 1498 photos that are 5184 by 3456 and it's taking quite a while, I would say about an hour or more. This is on a brand new system I just assembled consisting of an i7 3930k with 32 GB of RAM that flies with every other program. While exporting this batch I opened Task Manager and I noticed that CPU usage never goes to 100%, not even close. There are peaks of 50%, but on average it must be in the 20s:

This is very disappointing on a CPU that has 6 physical cores and 12 logical cores with hyperthreading at 3.2 with turbo at 3.8 Ghz. The batch is exporting these photos from one SATA 6 drive to another SATA 6 drive, and the HD LED barely lights up, so I know the hard drives are not the bottleneck. So I'm wondering, is Lightroom 4.2 really that bad when it comes to taking advantage of the CPU cores? Is there anything I can do to make it use the CPU more?

Thanks,

Sebastian

[Todd Shaner]

Sounds like it's time to buy a new system.

I wish I could give you solid configuration guidance, but there are some users with six-core Xeon processor systems, 32GB memory, and multiple SSDs who are still experiencing LR performance issues.

My first LR system was a single core Pentium 4 with hyper-threading running on XP 32bit with 2GB memory and two 500GB hard drives. I used it with LR 1, 2 & 3 processing Canon 300D 6.3 mp raw image files. It wasn't "speedy," but it was very usable. When I purchased a 5D MKII in 2010 the 21 mp raw files brought LR to its knees! I purchased an HP Pavilion HPE-170t in 2010, which has run flawlessly with LR3 and all versions of LR4.

Camera image resolution (Megapixels) has a definite impact on LR's performance, as well as monitor resolution and use of dual monitors. For the latter getting a high-performance graphics card won't help since LR doesn't support Open GL or any other type of graphics acceleration, at least not yet.....

I have no doubt my current system will run out steam with the addition of a higher resolution monitor (>1920 x 1080), higher resolution camera (>21 mp), or a future LR upgrade. Welcome to the sometimes wonderful and sometimes wacky world of LR performance!

[BillAnderson1]

Yeah I didn't mention that I have a pair of HD 1920X1080 24 inch monitors and I consistently use both at once. Plus I zoom constantly trying to get that last little bit of clarity.

Not all of LRs problems are hardware and they can't be fixed with HW. And, not all hardware is all it's cracked up to be. SSDs are fairly new and have some issues, especially if they are installed on older SATA 2 controllers with out of date firmware.

Some issues are caused by the use of a rather weak and poor performing relational database system called SQLite. I don't know for sure but I think it may new to LR 4 and I believe it is not well suited to seriously high thread count system, there many poor rankings of it. I've noticed a file with the name "lock" making me suspicious that they queue on it to control simultaneous updates. Adobe could add some value to the equation by hiring a professional database analyst to recommend a better product and design a better structure to accommodate high usage situations.

Anyway, it's been a treat to kick this around with you folks. It reminds me of my very, very early days as an IBM Operating Systems Programmer. Alas, that was when the S/360 Model 30 was powered by steam and our 36-hour work shifts were fueled by candy bars, coffee and cigarettes, followed, of course by long sessions at the tavern to tell each other how totallly cool we were!!!

[Todd Shaner]

BillAnderson1 wrote:

It reminds me of my very, very early days as an IBM Operating Systems Programmer. Alas, that was when the S/360 Model 30 was powered by steam and our 36-hour work shifts were fueled by candy bars, coffee and cigarettes, followed, of course by long sessions at the tavern to tell each other how totallly cool we were!!!

Sounds like we're both throwbacks from the same technology era. I started my career in 1968 working for Interdata designing minicomputers based on IBM 360 architecture. What's interesting is that with all of the technology advances we've made the basic root-cause of problems never seems to change.

[BillAnderson1]

Indeed. I couldn't have said it better. I started in 1968 as a Systems Programmer for the National Bank of Washington. Good times!!!

[areohbee]

BillAnderson1 wrote:

Some issues are caused by the use of a rather weak and poor performing relational database system called SQLite. I don't know for sure...

This seems to be a popular theme, but I question it's validity.

SQLite seems very fast to me, and an appropriate choice for Lightroom (single-user edition...).

Disclaimer: I don't know enough to know for sure either, but I do know enough to question...

Certainly my benchmark tests indicate whole-catalog read/write operations are lightening fast, begging the question: if it does have an achilles heal, where, exactly, is that??? Transaction queueing?... (or is it really not a bottle-neck in Lightroom at all...??).

Rob

[Todd Shaner]

Some interesting reading on the subject:

http://feedback.photoshop.com/photoshop_family/topics/multi_user_multi_computer

For single-user applications database does not appear to be a bottle-neck. From my own experience slow LR performance was quickly remedied by replacing my 5+ year old system with a new mid-range system platform. The bigger issue is why do some people with very high-end systems still have LR performance issues?

Does SQLite run the same in a multi-threaded environment, on every new processor platform, with every compatible OS? Given that Intel processor and I/O system technology changes about every six-months it's not surprising to me that a new system platform may have issues running processor or I/O intensive applications.

On my first job one of the designers developed a new floating point processor architecture that provided faster computations and better rounding accuracy. NSA was one of first customer to beta test a system with the new floating point processor. All of their applications ran a background system health monitor that used results calculated on a known good system. The monitor immediately detected the "more accurate" floating point data as a "system error." NSA refused to purchase the new system until we changed the floating point processor to match all of our other system's calculations.

Many years later Intel ran into a similar problem, which they claimed was not serious. It cost Intel about $450,000,000:

http://en.wikipedia.org/wiki/Pentium_FDIV_bug

[areohbee]

Thanks Todd,

I am familiar with that thread, but it seems like most talk is from people who know enough to get into trouble, but not enough to get out of it... .

Good point about it may work better on some systems than others. Still, if I had to bet, I'd say the database engine is a smaller part of Lr's performance woes, but I'm mostly just guessing.

Cheers,

Rob

[Todd Shaner]

Rob, you have way more experience with LR's SQLite database than I do. I'm just guessing (as well) that on one or more of the newer Intel platforms perhaps it "can" become a bottle-neck due to changes in the processor architecture (i.e. thread handling?). Develop module image processing is the most processor intensive LR activity and I wouldn't expect the newer Intel six-core processor based systems to have severe performance issues, but some do. I'd be inclined to say it has nothing to do with the SQLite database, but I wouldn't rule it out either.

[areohbee]

trshaner wrote:

I'd be inclined to say it has nothing to do with the SQLite database, but I wouldn't rule it out either.

Yeah, until we know what is (are), really, the bottle-neck(s), I wouldn't rule it out either...

Riddle me this batman - if Lr is taking it's sweet time to do something, and not using 100% CPU, and not disk-bound, what is binding it?

Cheers,

Rob

[DdeGannes]

This thread header is refering to LR 4.2, is the discussion related to LR 4.4?????????????????.

[BillAnderson1]

It applies to LR 4.X. The change to LR 4 from prior versions raised many of these concerns.

[BillAnderson1]

an internal spin loop lock using only one core.

[bob frost]

From: "Rob Cole

Riddle me this batman - if Lr is taking it's sweet time to do something,

and not using 100% CPU, and not disk-bound, what is binding it?

IME quite often it is the preview system. My computer normally works well.

It is high-end with SSDs and fast internal HDD, Win8x64. But yesterday it

started misbehaving; slow and erratic scrolling in library, with the image

squares showing gaps between them that had glimpses of the underlying

desktop. In addition the badges were half-hidden at the bottom of the image

squares.

At first I blamed the graphics card (quadro 2000D) and uninstalled and

reinstalled the driver. No difference. So I uninstalled and reinstalled LR,

and then installed an earlier version of LR. No difference. After a lot of

head-scratching as to what else might be causing this, I resorted to my old

faithful cure for most LR ills: delete the preview folder and previews and

rootpixels dbs, and regenerate them all overnight. This morning it is back

to normal. QED.

Bob Frost

[BillAnderson1]

Thanks Bob, that is a very useful tip .

Again, just "speculating" here, I'm going with a corrupted and/or fragmented Previews DB as the culprit.

[BillAnderson1]

What we have here, ladies and gentlemen is the classic example of exponential growth or as we called it in the early days of online transaction processing: the turn-pike effect.

Here’s an example: you are a night shift worker in downtown Seattle and you commute in a van pool with 10 other people from Tacoma, 35 miles south. It takes you 40 minutes, day in and day out (or night in and night out). Then, you and all 10 of your car pool mates get promoted and you get moved to the first shift. On your first day all 11 of you need to drive separately and you're sharing I-5 with 10s of thousands of other commuters. You and your mates all leave at about the same time, but all  11 arrive at different times ranging from 60 to 75 minutes after leaving Tacoma. Wazzup with that? What happen was that the demand for the limited freeway space went up exponentially. Demand exceeded supply and your time on the freeway went up a ton and each of you experienced a different set of circumstances.

So how does that equate? Think of your PC as a whole set of freeways or more aptly put as a bundle of rare resources that must be shared. The resources are things like the core processors, control units, disks, memory, page files, monitors, files, databases etc. When a program (LR in our case) wants to gain access to a database it gets in the disk queue and waits its turn. Most times LR gets what it wants instantly, no waiting. Now suppose that particular DB sits on a disk that has other stuff on it, i.e. the page file. Now say that your email system receives a bunch of large messages and at the same time your browser wakes up to download the latest stock prices at exactly the same time that want LR to export 11 photos. The LR engineers, who designed LR Export, knew they could launch 11 independent threads to actually read the DNG file, apply all the editing, generate the JPG and write the new photo to its new location because all threads could work independently of each other. All 11 threads get in line but have to wait because that rare disk resource is busy doing lots of other stuff. So your export seems to drag on, but it didn’t when you exported a similar number of photos just before you went to bed last night.  Under the circumstance where many other tasks and threads are banging against rare resources, conflicts will come up, not so much during quiet times.

So there you have it, my “questionable speculation” on why Lightroom behaves so differently under a variety of scenarios. But, unlike others I do have solutions, but that’s for another time…

[areohbee]

BillAnderson1 wrote:

...unlike others I do have solutions, but that’s for another time…

Yes, now would be a particularly bad time for sharing solutions... .

Some aspects of computer lag are more readily explainable than others. I mean, Lr performance can be a bigger enigma than you may realize. People often assume that Lr is disk-bound, since their CPU is not maxing out very much, but then they get SSDs in there and it doesn't perform that much better (and yes - spreading disk files across multiple devices is far from a panacea when it comes to freeing up a copy of Lr that is not CPU bound). Likewise, people hear that Lr is CPU-bound and so buy a rip-snortin' CPU and again: improvement is disappointing. The days when the simplistic view of CPU vs. Disk being the main (or only) contenders, are over... Honestly, I used to understand this stuff pretty good, but my understanding is getting dated - there are factors in modern HW/OS/Apps that elude me now, especially where Lr is concerned...

So, I'd love to hear about your solutions now, but only if they are not just theoretical...

Summary:

------------

* Cuttin' through those big giant raw files in just a fraction of a second are you? - please, do tell.
* Theorizing about what it would take to do so? - not interested...

R

[BillAnderson1]

Rob, you give new meaning to the concept of doubting Thomas! Just kidding.

Believe me after many years (and many years ago) of assembler code writing for operating systems, telecommunications and yes, database management system, I am well aware that exponential growth applies to complexity. And Todd said it best “the root causes of system problems haven’t changed much”.

Ok, here’s an example of blazing speed: I just ran this test less than 5 minutes ago. I fired up LR in Library mode; I selected 14 photos that I took this morning, all DNG averaging 30MB; I exported them to a folder on a separate disk. I exported them as JPG with a 1620X960 aspect ratio with no sharpening. It took 1 minute 38 seconds. I shut down LR and deleted the JPGs from the folder. I than fired up Photoshop Elements 11, I had earlier imported the same 14 photos to PSE 11s catalogue, so I selected them for export. I set the export size to the same 1620X960 pixel ratios. The PSE 11 export took 9.4 seconds. The JPGs were pretty much the same in quality and size.

As to how to fix this: dump SQLite altogether. The database in a digital photography editing application is hierarchical in nature. That is you have a photo that is at the center (the parent) and then you have the dependants (children). The dependants are previews, catalogue entries, edit history, layers, etc. This is called a one-to-many relationship and it works very well. RDBs work very well for many-to-many relationships and that doesn’t apply here.

Bill

[areohbee]

BillAnderson1 wrote:

Ok, here’s an example of blazing speed: I just ran this test less than 5 minutes ago. I fired up LR in Library mode; I selected 14 photos that I took this morning, all DNG averaging 30MB; I exported them to a folder on a separate disk. I exported them as JPG with a 1620X960 aspect ratio with no sharpening. It took 1 minute 38 seconds. I shut down LR and deleted the JPGs from the folder. I than fired up Photoshop Elements 11, I had earlier imported the same 14 photos to PSE 11s catalogue, so I selected them for export. I set the export size to the same 1620X960 pixel ratios. The PSE 11 export took 9.4 seconds. The JPGs were pretty much the same in quality and size.

As to how to fix this: dump SQLite altogether...

That's an example of how PS11 is faster than Lightroom. I was kinda hoping for a (practical) solution to making Lightroom run faster .

Anyway, I don't think the Lr database has much to do with this performance issue. Why not? because if you consider what happens during export, it's:

1. read settings from catalog.

2. apply settings to image...

3. update catalog.

The first and 3rd take only a tiny fraction of a second - the vast majority of the time is spent in #2 and SQLite does not even enter into it during this phase.

PS - I have a computer engineering degree, but that, in and of itself, does not necessarily mean I know what I'm talking about, right? (in fact, I could be lieing, or just a really bad engineer, or never practiced after graduation...). - i.e. the proof is in the pudding...

Rob

[BillAnderson1]

And you know this how? How many SQLite DBs are in LR? One for the whole thing or... I'm pretty sure I found at least two: the cataloge and previews, plus I'm thinking something called pixels might be a DB and possibly a temp dataset with the extension .dat

SQLite is an embedded utility, meaning it doesn't stand alone and have it's resource usage monitored separately like MySQL. So you can't tell when CPU power is running through SQLite code or LR application code. That can account for at least some of the apparant high CPU usage.

[areohbee]

BillAnderson1 wrote:

And you know this how?

From experience.

There are 2 sqlite databases:

1. catalog.

2. previews.

PS - If you want to learn more about it, consider downloading the sqlite3 command line utility, and/or one of the sqlite3 gui apps, like SQLiteSpy - these can access preview database while Lr running, but require Lr shutdown in order to access catalog.

R

[bob frost]

From: "Rob Cole

There are 2 sqlite databases:

1. catalog.

2. previews.

What about the root-pixels.db?

Bob Frost

[BillAnderson1]

Thanks to Rob's excellent suggestion to download the SQLite utility, I can confirm that root-pixels.db is, indeed a SQLite DB. And there may be more because Adobe didn't use DB as the extension for the catalog so I had to look for *.* before I found it. They used .ircat as the extension.

[Jim Wilde]

It's actually Lrcat, as in LightRoom CATalog.

[BillAnderson1]

Thanks for catching that, typing is not my strong suit.

[bob frost]

From: "BillAnderson1

Thanks to Rob's excellent suggestion to download the SQLite utility, I can

confirm that root-pixels.db is, indeed a SQLite DB. And there may be more

because Adobe didn't use DB as the extension for the catalog so I had to

look for . before I found it. They used .ircat as the extension.

That is because you are using a Mac. Using Windows, they are named

previews.db and root-pixels.db. And I often use SQLite Browser to look

inside the main catalog and these others, to try and work out what is

happening. It can help when things go wrong, and in the past I've even used

it to edit entries in the catalog when LR was a bit more flaky than it is

now.

Bob Frost