Performance - From a Computer Science Point of View

Report · Sep 24, 2017

Like others, I am having a frustrating time with Lightroom editing photos from a performance standpoint.

My workflow typically has me creating a new catalog for a particular photo shoot, importing the (RAW) photos and applying a custom preset on import and then editing the photos one by one in the Develop Module.

After editing a number of photos (say, 10-25) the performance of Lightroom degrades significantly and I have to exit Lightroom and re-start it to continue editing (and continue to do so until I have completed my editing).

I have done significant research into Lightroom performance issues and have tried all of the suggestions and all possible permutations thereof to no avail. I have also profiled Lightroom with a number of Windows development tools to take a look at it's I/O, threads, memory, etc. and nothing specifically stands out.

I do understand that Lightroom stores it's editing changes (whether via a preset or manually) in a SQLite database and then applies these changes to a photo in 'real-time' to render what you see on the screen. That leads me to believe that the program is having a difficult time querying the database and applying the changes in 'real-time'... it is almost like there is a 'leak' as one moves from one photo to the next.

Edit: I took a quick look at the Lightroom .lcat file which is actually a SQLite database. In it, I found 102 tables. Still looking at the relationships between these tables.

All of the suggestions to add faster disks, larger caches, build previews, etc. are all masking the real culprit which I believe is an internal data structure, database, etc. issue. My 'evidence' for this is that exiting and re-invoking Lightroom will always provide me great performance until I reach 10-25 photos and then I have to 'rinse and repeat'.

What I would like to know from the Lightroom software development team is whether there is a way for uses to 'peek' inside the program to know what is actually going on from an operating system and computer science perspective (open files, memory allocation, threads, locks, etc.). I would assume that the Lightroom software development team knows exactly what is going on as the program is likely instrumented. Knowing what it is that is causing Lightroom to gradually degrade its performance will help users understand how our behavior might need to change when using Lightroom to improve our experience (like perhaps not/not applying presets to hundreds of photos at once rather, do so as you edit one photo at a time).

I do love Photoshop and Lightroom and recommend the products to everyone that ask me for advice but I also believe that Lightroom's performance issues are serious and need to be addressed.

Thank you for your consideration

Report · May 15, 2018

If Adobe would show you the internal processes they would have to change the name to Lightroom middle ages.
Thats why disabling the gpu support solves so much issues

Report · Nov 28, 2017

Guys, take a look at my last post. Maybe it could be relevant to some of you.
https://feedback.photoshop.com/photoshop_family/topics/lightroom-classic-cc-develop-module-still-slo...

Report · Nov 06, 2017

To those that have looked at the performance of LR with the Visual Studio product, were you able to successfully get the LR symbol files for their DLLs? I am seeing some interesting contention for resources and exceptions being thrown in the code but without the symbols, it is hard to determine exactly where it is occurring.

Report · Nov 06, 2017

Hi Guys, I have first alerted Adobe on this slowdown a year ago, on this thread which was largely ignored. it was related to export speed being double the time after working on images compared to export the same batch after a restart.

https://feedback.photoshop.com/photoshop_family/topics/lightroom-image-export-is-much-faster-after-r....

I have also linked the performance degradation in export results to the general slowdown in image editing. its all stem from the same issue to my analysis and by doing simple export benchmark which I will outline below we can give Adobe simple performance data. Adobe can try and replicate it very easily in their high-end CPU test machine.

Here are the steps.

1. Select 30 images for export as full size add them to a collection called '30 images export speed test'
2. Open the task manager and put it on the CPU graphs
3. Restart Lightroom
3. start the export and time it using your stopwatch on your phone.
4. you will notice the CPU shoots up to 100% on the first export
5. do a print screen of it if you want to post it here
6. once the 30 images have finished start the exact export again, without restarting Lightroom
7. Look the CPU graph and you will notice no more 100%, Lightroom is now slower.
8. Time it and take a print screen of the CPU graph if you wish to share it.
9. Do it again, and time it and see if it's getting even slower
10. Post your results here, with the OS, CPU/GPU type and No of Cores and type of Raw files

one big question is there anyone out there that the results are consistent with no performance degradation I would love to know their spec!

here are my results for

30 raw files of phase one IIQ 280 80MP
Windows 10, i7 5960X 8 Cores, Nvidia 980Ti, 64GB RAM

export times Lightroom Classic CC 7.0.1 in minutes per run in order
2:31, 3:17, 3:56, 4:35, 5:10

Here is the test again with the OpenGL instead of DirectX
Config.lua flags:
Develop.PreferOpenGL = true

2:24,3:12,3:55

Clear performance decrease in both scenarios, so the flag does not make any difference to performance degradation....

here are the CPU graphs
1. first run CPU maxed to 100% most of the time

2. second export CPU less busy export is slower

3. third export CPU even less busy and export is even slower

4. forth export even slower, I skipped the screenshot you got the idea ...

Here are the capture one 10.2 times for the same 30 images in minutes
1. 1:17
2. 1.16
3. 1.17
4. 1.16
5. 1.17

here is capture one CPU graph which you can see there is plenty of CPU to keep working on images while the export is running at consistent speed, no performance degradation after 5 runs.

Capture one puts Adobe to shame.

Its frustrating Adobe put their resources into lightroom cloud instead of fixing this year-old bug.

Anyone who can do this simple test and let us know in which specs there is or isn't performance degradation so we can narrow our understanding for the reasons behind this problem.

Report · Nov 05, 2017

For those of you on Windows (10) who are experiencing slowdown problems after editing some number of images in Lightroom - are you using DirectX or OpenGL with Lightroom?

Report · Nov 01, 2017

I ́ve tried your procedure with 30 Raw files from a 7D MK2. Export takes 12 sec. each and every time. No speed degradation here. No restart of LR after exports.
Win 10 (Anniversary Update), I7-5930K (6-core@4ghz); GTX 970 (Driver 376.33); 32GB RAM; LR Classic 7.01.
CPU always at 100%

Report · Nov 01, 2017

Hi Guys, I have first alerted Adobe on this slowdown a year ago, on this thread which was largely ignored. it was related to export speed being double the time after working on images compared to export the same batch after a restart.

https://feedback.photoshop.com/photoshop_family/topics/lightroom-image-export-is-much-faster-after-r...

I have also linked the performance degradation in export results to the general slowdown in image editing. its all stem from the same issue to my analysis and by doing simple export benchmark which I will outline below we can give Adobe simple performance data. Adobe can try and replicate it very easily in their high-end CPU test machine.

here are the steps.

1. select 30 images for export as full size add them to a collection called '30 images export speed test'
2. Open the task manager and put it on the CPU graphs
3. Restart Lightroom
3. start the export and time it using your stopwatch on your phone.
4. you will notice the CPU shoots up to 100% on the first export
5. do a print screen of it if you want to post it here
6. once the 30 images have finished start the exact export again, without restarting Lightroom
7. look the CPU graph and you will notice no more 100%, Lightroom is now slower.
8. time it and take a print screen of the CPU graph if you wish to share it.
9. do it again, and time it and see if it's getting even slower
10. post your results here, with the OS, CPU/GPU type and No of Cores and type of Raw files

one big question is there anyone out there that the results are consistent with no performance degradation I would love to know their spec!

here are my results for
30 raw files of phase one IIQ 280 80MP
Windows 10, i7 5960X 8 Cores,Nvidia 980Ti, 64GB RAM

export times Lightroom Classic CC 7.0.1 in minutes per run in order
1. 2:31
2. 3:17
3. 3:56
4. 4:35
5. 5.10

clear performance decrease...

here are the CPU graphs
1. first run CPU maxed to 100% most of the time

2. second export CPU less busy export is slower

3. third export CPU even less busy and export is even slower

4. forth export even slower, I the screenshot you got the idea ...

Here are the capture one 10.2 times for the same 30 images in minutes
1. 1:17
2. 1.16
3. 1.17
4. 1.16
5. 1.17

here is capture once CPU graph which you can see there is plenty of CPU to keep working on images while the export is running at consistent speed, no performance degradation.

Capture one puts Adobe to shame.

Report · Oct 31, 2017

I have just reinstalled a fresh version of Lightroom Classic on a fresh Windows 10 installation of the Fall Update. I have a high end system with NVMe SSDs and a GTX 1080Ti with a 20 core 7900X Processor. I have exactly the same problems as described here, a slowing down after a while. I am appalled at the reaction I get from Adobe, which usually is “disable the GPU” or “we can’t test every system”. To be clear this is a fresh installation with no other Software installed except BitDefender 2018 - is this the problem?

I’ve now let my subscription lapse and gone over to Capture One, which performs fine, but I will follow progress and perhaps come back to Adobe if the issues ever get sorted. Frankly though after years of the software going downhill performance wise, and the attitude of Adobe - I am not holding out much hope!

Report · Oct 31, 2017

Just read Joel's latest comments. I have exchanged emails with Joel and I can attest to the hardships he is having with Lightroom Classic. I too, have had issues with the product (although, I do see some performance enhancements in Classic but still observe a progressively sluggish system after editing 10-30 images).

I assume that what we are seeing is not the 'norm' but I do wonder what is going on because this type of sluggishness is not typical in other products that I also use.

As many of us are long term users of Lightroom (and Photoshop) and have updated many many times over the years (both the products themselves and I would suspect the underlying operating system) - is it at all possible that old files (.dlls, etc.) are contributing to the issues?

I have not completely wiped my system clean and installed a fresh copy of Lightroom - has anyone tried this? Even in a virtual machine?

I ask because I can not even begin to imagine that this is the 'norm' when the product team is testing the next version of the product... and I would assume that the product team likely starts with a clean system when testing?

Also, I would be interested in knowing if Camera RAW == Lightroom Develop Module... are individuals commenting on issues in Photoshop as well as it pertains to the Camera Raw module?

Report · Oct 20, 2017

I concur with the instrumentation observation but I do appreciate Adobe working diligently to find the error(s) and fix them. I have used sysinternals tools (on Windows 10) to try to track down the issue and nothing jumps out to me (yet). I am hoping that with Adobe Max over with, the team has a little more time to really dig into this issue. I would be happy to send the team the photos and preset I use on import to see if they can replicate the issue however, the set is many GB in size.

Simon - if you all do have a debug flag or compilation with symbols, etc., I would be happy to run it with my photos and send your team any logs that might be generated. I know chasing things like this down without being able to replicate {n} users's systems is frustrating.

Report · Oct 20, 2017

The team are working on and it just takes a little bit more design iterations...

Report · Sep 29, 2017

I worked 25 years designing systems and working with databases was the norm for me. I haven't done it again in four years.

I decided to look at the .lcat file to get an idea of the content. I found 102 tables but none related to any other!, this is contrary to what should be in a well designed relational database. Even I found tables without primary key!

I imagine that when Lightroom was born they handled all the information in 102 individual files scattered, years later they decided to manage the information in a database but sadly they only converted the files into tables and did not design a true relational database.

Relational database managers are very powerfull, I used to handle 90000 pdf files (embedded as BLOB data in a table) in a plain laptop with less resources than my actual 27” Imac with 24 GB RAM, crawling just to update the keywords of 4000 pictures and creating several Smart Collections.

If someone finds a table with a foreing key column, tell me which one it is, I can't find it. I never used SQLite before.

Report · Sep 25, 2017

Hi Amit,

I will post the system info dialog contents within a day's time once I get back to my studio. Thank you for being willing to take a look.

Fred

Report · Sep 25, 2017

Hi Fred,
Could you please reproduce the performance degradation issue and then post the contents of the System Info dialog?

- Launch Lightroom and reproduce the performance degradation issue.
- Invoke the System Info dialog by selecting 'Help > System Info...' menu command.
- Copy the contents of the System Info dialog and post it here.

Thanks,
Amit

Report · Sep 25, 2017

What I meant by 'real-time' rendering is that Lightroom is a parametric editor that applies adjustments to the source image to render what you see on the screen and export as a finished photo. I do not doubt that Lightroom creates a .jpg with all of your adjustments to display on the screen but that .jpg is created by taking the 'raw' image and then applying, one-by-one, all of the adjustments you have made to the image. And I believe each image's adjustments are stored in the .lrcat (SQLite database) file.

Speaking only about the database, if the data in the file becomes fragmented or the underlying B-Tree(s) become unbalanced, I can see how that might cause a performance issue.

Report · Sep 25, 2017

"I do understand that Lightroom stores it's editing changes (whether via a preset or manually) in a SQLite database and then applies these changes to a photo in 'real-time' to render what you see on the screen. "

No, it uses a slightly different method. Lightroom renders a jpeg preview with the settings. That preview is then presented on the screen. In the develop module this is effectively the same as rendering directly to the screen would be, but if you are in other modules there is no real time rendering. You just see the preview, which could actually be fairly old if the image has not been edited for a while. The previews for these modules are stored in '<catalog name> previews.lrdata' in your catalog folder. The previews used in the develop module are stored in the Camera Raw Cache elsewhere.

-- Johan W. Elzenga

Report · Sep 24, 2017

I downloaded the latest SQLite tools for Windows (my platform) and am using the sqlite3.exe program to take a look at the underlying Lightroom SQLite database (tables, indexes, etc.) found in the {filename}.lrcat file. While this will not pinpoint the performance issue, it is illuminating to see the 104 tables that Lightroom uses in its main database.

Report · Sep 24, 2017

This is a very interesting question ! I am a software developer too and I often wondered how I could try to analytically approach the LR performance problem. I never noticed this progressive degradation of performance that you mention but indeed, sometimes, LR is blazingly fast and I cannot really pinpoint why. I once read that applying some development presets in a different order led to a totally different experience (i.e. always disable expensive camera corrections before using the correction tool...). I really hope that Adobe engineers will accept to shed some light on this (an instrumented flag when LR starts would alllow us all to collect info and maybe try to figure it out).
(Pardon my strange english by I am french speaking)

Report · Oct 31, 2017

Open Begging to Simon Chen and Jeff Tranberry

Will Anyone at Adobe Ever break the Cone of Silence? Many of us have been begging for help and begging to help find and address the serious performance issues we are experiencing.

Simon, you keep telling us "the team are working on it" and I believe you. I am just not sure what "it" is that they are working on. Clearly it has not been the performance issue. I am waiting more than one year and perhaps everyone on the "team" were too busy inventing the new cloud based product to spend any effort on the existing performance issues. Seems to me that has been quite unfair considering I pay every month for software with a major issue you have essentially ignored for more than 12 months.

Jeff, you have remained totally silent on the issue. Please tell us to go away, tell us how to fix it, tell us when we can expect some help, but please tell us something!

Both of you saw the issue while connected to my PC over one year ago and still no help and no discussion. At the PC Expo in NYC last week, I spoke to someone from Adobe that seemed quite happy when I said that I was being forced to go to Capture One and he said good if that solves my issue. Well, besides that not being a great answer, I may need to do just that. Perhaps, if those of us with this major issue went away, Adobe would be much closer to cancelling the desktop version in lieu of the new cloud based scenario that none of us could use anyway due to the lack of features and sheer volume of photos we process daily or weekly.

Many of us with strong IT backgrounds have offered to help run test products to determine the source of the issue with no response from Adobe. We have posted information trying to show how LR runs so poorly on our systems while other products seem to race along without any slowdown at all. Some of us have been forced to go back to the prior 2015.12 version so we can get our work done albeit with a restart of LR after every 30 or so images.

I am quite sure none of us want to leave Adobe and none of us want to use other products but we need your help. So, here I am begging openly for some kind of help in solving this issue.

Most of us with the problem are using high-end PC's. I have Windows 10 on a PC with a 3.3Ghz 12-core processor, 64Mb 2400 speed RAM, NVidia GTX Titan Graphics with 12Gb Dedicated VRam, SSD drives and I can tell you that everything on my PC is lightning fast (Including Capture One and ON1 RAW 2018) except for Lightroom.

Some of us (including me) teach comprehensive classes in both Lightroom and Photoshop and get from 30-100 students per year to enroll in the Lightroom CC (now Classic) system. Please help me continue supporting Adobe and referring students to the cloud based systems.

I am aware that my CC monthly payment means nothing to Adobe and if I went away the company would never miss it. I am aware that I am a thorn in your side for constantly complaining about performance. I am aware that the company line is that Lightroom Classic is much faster than previous versions. Well, I have yet to talk to any of my colleagues, friends, students, or anyone for that matter that thinks the new program is faster. In fact, even several of my students using MAC OS feel that Classic is slower than the prior version. Okay, I am sure some see an improvement in performance but I am not sure who they are. No matter, every day I read forum comments from many of you longest loyal customers begging for help to solve the performance issue.

So, Adobe, Jeff, Simon: I and others stand before you begging for some relief. Our businesses are suffering as we cannot process our work. Our lives are being adversely affected as we have no time left after spending three or four times as long as before working on our daily shoot images.

Please do not force us to move away from Adobe. Many of us have been loyal supporters for dozens of years. Your company was built on our continued loyalty, software purchases, and upgrades. We really want to stay loyal. We really want to continue to recommend Adobe products to our colleagues, friends, and students. We want Adobe to continue to be the leading force in the imaging business. To do all of this, WE NEED YOUR HELP!!!

Please, please, please help us. Let us help you. We will do anything you ask of us (within reason and our busy schedules) to identify and correct this issue. I am sure you have read some of my comments on other forum threads and the times I report are real!

Many of us await your reply!!!! PLEASE HELP US! PLEASE TALK TO US

Report · Oct 23, 2017

So I had a small photo shoot this morning (200 Nikon D5 RAW images). I opened LR Classic CC and started importing from my 440Mb/Sec XQD memory Card at 10:57am. Lightroom has finished copying the photos to my hard drive (finished at 12:04 - one hour and 7 minutes) and is still building standard previews (it is now 12:20pm and it is almost done). As soon as LR finished, it ejected the XQD card. So, I removed it and reinserted it. While LR is still processing standard previews, I opened Capture One and started importing from the XQD card to a second folder on the same hard drive - essentially, the same procedure. Both are applying my basic start preset of adjustments and some metadata. Started the Capture One upload at 12:08 and it took 2 minutes and 20 seconds to copy the photos and another 1 minute and 7 seconds to finish building the previews. Capture One did this while LR was still creating standard previews.

WOW!

Lightroom finished at 12:22 so the entire process took 2 hours and 25 minutes to copy the images and create standard previews. Thank the lord I didn't create 1:1 previews. Capture One did the same job, while LR was running in 3 minutes 27 seconds.

In addition, during the import, Capture one is fully functional but a little slower. Lightroom Classic, one the other hand, is totally locked up during the import making it unusable until the import is completed.

Am I missing something here?????

Simon, what exactly is going on here???

Report · Oct 20, 2017

I'm new here and I just read the last several weeks of posts in this thread.

It appears to me that neither memory, I/O or processors (including a single core) are maxed out. We could get those symptoms with CPU cache thrashing or intra-core communication problems, but that would not explain performance degradation over time. It's not a memory leak problem, because the degradation continues with plenty of free memory. And there isn't an I/O bottleneck (SSDs don't make much of a difference).

I suspect the tactical problem is lock contention and the root cause is inadequate instrumentation/tools to detect which locks are not being released.

That said, I may be completely off-base. Do any of you have other ideas about the cause?

Report · Sep 25, 2017

First, I'll say I could not agree more.

Second, I'd like to mention whenever I enabled GPU, LR just crashed.

Report · Sep 25, 2017

I am a retired senior developer as well. I have two nearly identical monitors, both LG 27" and 1920x1080. When the GPU is enabled, my secondary monitor displays much quicker than the primary. When the GPU is disabled, both are fast at first and gradually slow to a crawl after 25-35 images being edited.

Since there is so much discussion regarding limiting the number of processor cores for LR use, I think the problem may be related to the age of the base coding for Lightroom which is not able to take advantage of this improved multi-code multi-threading.

All of us guessing seems wrong. As I said before, Adobe is not a small bunch of people in a garage somewhere writing code. This huge company needs to invest in the time to fix the product and tell the marketing guys that always seem to want more features to slow down and wait until what we have works for all the professionals that are paying every month!

Frankly, I think I am entitled to a huge refund as I have been suffering with this issue for more than a full year with no real hope of future improvement due to Adobe's total silence.

Report · Sep 25, 2017

I am a senior developer too and I have been thinking the possible problems with this performance issue since about a year ago when I started to suffer. I am sure everyone here had tried pretty much the same approaches and tests to root out configurations or settings, or hardware related problems. And from our end as an application user no matter what we tried we always and still end up with the same bad performance. So I am thinking the issue is with the application and somehow triggered by some system related reason. And here is what I observed and suspected.

1. it may be related to my 27inch 4K monitor as the performance is better if use my second monitor which is 1980X1080. The same with if I am using dual screen mode. Navigating the images on the second screen is always fast. Also I almost don't have performance issue with my 15inch Macbook Pro.

2 The performance is better if I convert to dng file instead of always using the Nikon raw file.

3. it seems to be much faster if I set the default viewing size to be a third or a quarter vs. 'fit' which is close to a half of the image size.

It seems all make sense as more calculations needed for displaying high resolution and large sized images plus the involvements of the video card. In the mean time, I am thinking there might be a design flaw somewhere as every move needs a lot of calculation and reloading of the image (I do see high CPU and memory load all the time), which the LR development team may have known already.