RAM & swap usage (100GB over 30 minutes of work)

Report · May 17, 2022

Hi all,

I am running Lightroom Classic 11.3.1 on my 2021 16" MacBook Pro (M1 Pro, 16GB RAM, 1TB SSD), on macOS Monterey 12.3.1.
I am noticing an absurd amount of RAM & swap usage. Over the course of 30 minutes of work, kernel_task and Lightroom Classic itself has written about 100GB to the SSD. This doesn't seem normal. Memory pressure is green with spikes to orange with light work, and over time it grows to consistent orange (60-75% memory pressure)

I'm working on 300 Sony a7S III files with Super Resolution applied to them beforehand (a new catalog with pre-enhanced photos already), so ~43MP DNG files around 200MB each.

Is there a solution to this high RAM & swap use?

Report · May 18, 2022

16GB of system RAM is pretty light to work on super res files which is why you are seeing a lot of swapping to the SSD.

To me, 32GB would be the absolute minimum, 64GB would be a lot better.

Report · May 18, 2022

Super Resolution is not a frequent part of my workflow. My work usually involves weddings with 3000-5000 images per project / catalog (after culling rejects from 7000-15000!). Still, Lightroom eats up a lot of RAM and swap in that scenario. In comparison, Capture One usually consumes 7-12 GB of RAM with very little swap use.

Report · May 18, 2022

You're correct, it's not normal, but is also proving difficult for LrC engineering to reproduce and thus fix. Unfortunatley, it gets worse in so far as there aren't any guaranteed pain free workarounds at present, although one that seems to help users reporting similar issues is to turn GPU acceleration Off in LrC Performance preferences. Doing so shouldn't hurt you too much since Adobe use the Apple Neural Engine (ANE) for Enhanced Raw and Super Resolution with the GPU primarily being used for rendering the display. The same applies to the Ai masks. Where you 'might' find a slow down are LrC features that make heavy use of the GPU for processing and working on images that have alrerady been processed in app such as DxO Pure Raw or Ai apps from On1, Topaz, etc. Even so, I encourage you try switching off the GPU and sharing your observations as same may help others in a similar situation.

Report · May 18, 2022

Turning off GPU acceleration doesn't prove much help. Lightroom eats less system RAM (say from 12 GB to around 5-7 GB) but swap use is still excessive.

Report · May 18, 2022

How are you measuring memory use? Activity Monitor, Top or third party app

Report · May 18, 2022

Activity Monitor and iStat Menus.

Report · May 18, 2022

I believe Ian’s answer is more correct. Although 32GB is better, 16GB Unified Memory on an M1 should be enough to get work done in Lightroom Classic with Memory Pressure not staying orange, and 64GB should be unnecessary. (The official Lightroom Classic system requirements are 8GB minimum and 16GB recommended.)

But there does appear to be a bug somewhere that causes macOS to think an app is consuming gargantuan amounts of RAM. I’ve seen it occasionally with different apps, including Photoshop and Bridge, even though the applications are not handling anything that would consume that much memory. It is not yet clear whether there is a bug on the Adobe side or the macOS side, but as Ian said, Adobe is getting a lot of reports about it and hopefully they will figure out what is going on.

Report · May 18, 2022

Yes, I've seen similar posts about the M1/macOS writing too much to the SSD even outside of LrC but I thought they'd fixed that with the latest update. Not positive as I'm a Windows user.

However, the OP mentions he's processing super res photos and that is a whole different story than Adobe's 'minimum requirements'. I know because I shoot medium format digital (102megapixel) with 16-bit raw files and I can see the amount of memory necessary. Even assuming Apple's claims about the M1 being more memory efficient, it's hard to get around the requirements of extremely large image processing. This is a vastly different scenario than "normal photo" processing.

Report · May 18, 2022

I agree about the different kind of demands that 16-bit medium format files impose. Although even with a more "normal" workflow, like 85% of the time I'm working on 3000-5000 image catalogs of wedding photos, from 24 to 46MP cameras. The same issues persist with massive swap usage.

Report · May 18, 2022

Then it's possible Apple hasn't fixed the memory architecture issue, yet, or it could be the way LrC uses memory is aggravating the issue.

Report · May 18, 2022

It appears to be a memory leak in that case. Especially because having the program open for a long time will see eventual slowdowns even if you're not working on anything demanding (most computers released even more than 5 years ago shouldn't have any issues with 20~ MP images and basic spot healing)

Report · May 18, 2022

> But there does appear to be a bug somewhere that causes macOS to think an app is consuming gargantuan amounts of RAM.

Yes, Conrad C has made a good point and one that should not be dismissed. By way of example, I've seen screenshot examples comparing iStats menus indicating many GB of free memory along with even more GB of compressed memory, which in theory is also free for use when memory is under pressure. Yet, at the same time both Activity Monitor and Terminal Top are indicating that memory pressure has far exceeded all that can be used and is therefore using large amounts of swap.

Above examples are on systems with 64GB of unified memory. So, any suggestion that the OP is asking too much of too little needs to be placed in context. Sure, 32GB of memory is preferable but not an absolute minimum. If it is a mimium, then Adobe are guilty of misrepresenting the requirements of their applications, and I very much doubt they'd be so stupid.

Report · May 18, 2022

@Ian Lyons Leaving aside whether or not there is a bug that is causing excessive swap, I want to address your comment about "minimum requirements". I see a lot of confusion over this point.

"Minimum requirements" means the program will operate and work. From the OP's comments, LrC is working and the OP is able to process images. However, there is excessive swap activity. That is a consequence of only having the minimum requirements. When physical memory isn't available, programs use "virtual memory" which is swapped in and out as necessary. When processing high res images, LrC needs more memory. If physical memory is not available, it uses virtual memory. So the program works but the penalty is lower performance and more swapping.

Now, as I said in one of my replies, I have heard of swap issues with the M1/macOS that affect all programs, not just LrC but I thought they had been addressed in the latest update. Not being an active mac user, I don't follow those issues as closely as I follow PC/Windows issues so I am not positive that it has been addressed.

Most tech companies try to market their products to as broad of a user base as possible so minimum requirements are often just that, minimum requirements for the program to operate. They are not necessarily the 'recommended requirements'. The recommeded configuration is highly dependent on the type of work being performed. Making small adjustments to JPG files is very different from detailed processing of high-res RAW files. I can easily max out cpu and memory working on 102megapixel raw files especially when stitching together panoramas or building an HDR. Would it work with the minimum requirements and swapping to disk? Probably... but I wouldn't be happy with the performance or the amout of swapping.

Report · May 18, 2022

32GB is preferable, but a workflow of slapping on simple edits / presets and one or two local masks to less than 10,000 24 to 46MP image catalogs should be doable with a 16GB machine with occasional spikes to orange memory pressure and not much swap. My office computer is a trashcan Mac Pro with a 6-core Xeon, 32GB RAM, and a 6GB AMD FirePro D700 graphics card. Lightroom eats up 14-20 GB of RAM with that workload, and around 2GB of VRAM. All things considered equal, a 16GB unified memory machine should be able to handle with no problem, maybe with 3-5 GB of swap without excessive disk writes which is fairly acceptable. I cannot test fairly because my work machine runs on LrC 10.3 while my personal machine is on LrC 11.3.1.

Report · May 18, 2022

@Charl Pantaleon I understand your frustration and sort of agree with you. I was a system programmer for many years and my colleagues and I often talk about how inefficient programs have become. I am not an Adobe employee nor do I have any special knowledge about their algorithms. However, I have noticed over the last few versions they keep claiming improved performance and I have to believe they are achieving a lot of that performance improvement by keeping more things 'cached' in memory which may partially explain the difference you're seeing between the two systems.

For the record, the number of images in the catalog is pretty irrelevant for most operations. The relevant factor would be how many images you scroll through causing LrC to load previews and, definitely, how many you take into the Develop module. Complexity of masks would be another factor that I can see consuming more memory.

The M1 processor has thrown a new wrinkle into development, as well. I think Apple developed the architecture with a specific usage model in mind and it may take several updates for software developers to figure out how best to adapt their software for the new architecture.

Here's a link to a thread from a year ago talking about the issue. https://forums.macrumors.com/threads/m1-mac-users-report-excessive-ssd-wear.2285892/

And here's another link claiming it was actually a reporting error, not a real issue and was fixed...

https://appleinsider.com/articles/21/06/04/apple-resolves-m1-mac-ssd-storage-longevity-issue-in-maco...

From the linked article:

"At the time, an AppleInsider source within Apple, not authorized to speak on behalf the company, told us that it was a data reporting error within the tools used to report SSD wear. According to that source, it was not believed to be an actual hardware issue with the SSD, nor were the SSDs aging notably faster than prior because of RAM swap or other reasons."

Report · May 19, 2022

Just a small comment about "unified memory", in response to no one in particular.

All "unified memory" means is that system and GPU share the same memory pool. So suddenly, 16 GB RAM is not nearly as much as you thought it was. The GPU may eat up a large chunk of it, but you may not know that, because it's, well, unified.

So you get swapping.

With a discrete GPU this isn't a problem, because it has its own onboard VRAM.

Other than that, I fully agree with Walt Thirion's observations re minimum requirements vs. large files.

Report · May 19, 2022

I spent a little time catching up on the unified memory architecture and, not only are you correct, but in my opinion, Apple targeted the 16GB memory config for users that only lightly use the GPU. For example, the recommended config for video processing seems to be the 64GB MacBook Pro.

Furthermore, from what I can discern, Apple uses a higher speed SSD specifically to speed up the increased swapping expected with lower memory configurations. The idea seems to be to use a smaller amount of higher speed ('more expensive') memory treated almost like a large cache and use swapping to the high-performance SSD like extended, lower speed memory.

Sharing fast main memory between the cpu and the gpu can have some efficiency improvements as data doesn't have to be moved as often but would require a larger main memory config for applications like LrC which can spawn multiple cpu threads and simultaneously load the gpu. The architecture also relies on the SSD being able to handle the larger number of writes from the increased swap usage. Apple does claim the increased SSD usage is NOT a problem...

Report · May 19, 2022

The idea seems to be to use a smaller amount of higher speed ('more expensive') memory treated almost like a large cache and use swapping to the high-performance SSD like extended, lower speed memory.

By @Walt Thirion Photography

That is a very precise description of how Photoshop already works. I often describe RAM as a cache to the scratch disk's main memory. With a fast NVMe as scratch disk, the old bottleneck is all but gone.

But Photoshop has its own memory management and doesn't have to go through the OS to write to scratch disk. It just reserves disk space and uses it as needed. That puts Lightroom at a disadvantage, since it has to queue up with the others for the OS page file.

Report · May 19, 2022

@D Fosse wrote:

All "unified memory" means is that system and GPU share the same memory pool. So suddenly, 16 GB RAM is not nearly as much as you thought it was. The GPU may eat up a large chunk of it, but you may not know that, because it's, well, unified.

It’s a fine point but it is important to note that isn’t really “all” it means, because that also describes traditional integrated graphics. What distinguishes Unified Memory from Intel integrated graphics is, as Walt Thirion Photography said, the more direct access that graphics hardware has to that shared pool, without the limitations of Intel integrated graphics, which allows graphics hardware to perform better under unified memory than with Intel integrated graphics.

But the concerns expressed by both are real: The more graphics memory an application needs, the more unified memory a Mac should have. If someone wants an application to have access to 16GB memory and 8GB VRAM, they should not spec a Mac with 16GB unified memory, they should spec the next level up which is 32GB.

Until recently Adobe photo and graphic design applications haven’t done much with the GPU, but that is changing now as Adobe extends GPU support to more areas of Photoshop and Lightroom. Chances are, in a few years we’ll have to specify unified memory amounts more in line with what video and 3D applications need.

Report · May 19, 2022

It’s a fine point but it is important to note that isn’t really “all” it means, because that also describes traditional integrated graphics.

By @Conrad_C

Yes, that was the implication. Lots of Windows laptops have worked this way for years. The big difference is that an Intel GPU isn't very powerful (Photoshop may or may not work with it), and it was always intended as a basic low end solution for basic needs.

I assume the M1 GPU is a lot more capable. But that also means it will use a lot more memory.

Report · May 19, 2022

@D Fosse wrote:
I assume the M1 GPU is a lot more capable. But that also means it will use a lot more memory.

This might have changed recently, but for many years Intel integrated graphics was limited to taking no more than 1.5GB from system RAM. That alone limited it compared to Apple unified memory, which as I understand it is free to give graphics any memory that isn’t already being used as application and system RAM. So if you have an M1 with 64GB unified memory but applications need only 34GB of it, the other 28GB is available to the GPU if it wants it. So it isn’t just that unified memory might use more graphics memory…but that it actually has that option to use a lot more, for the user’s benefit.

Where we used to choose from between integrated graphics and discrete graphics, now we have integrated (most limited), unified memory (more powerful but energy-efficient), and discrete graphics (still the most powerful but also the most energy-hungry).

Report · May 19, 2022

The Author stated in the initial post, "I'm working on 300 Sony a7S III files with Super Resolution applied to them beforehand (a new catalog with pre-enhanced photos already), so ~43MP DNG files around 200MB each."

Just thoughts out of the box, exactly what edits are being done, on individual images or batch of images?

Are you working with Auto-Sync activated?

Regards, Denis: iMac 27” mid-2015, macOS 11.7.10 Big Sur; 2TB SSD, 24 GB Ram, GPU 2 GB; LrC 12.5,; Lr 6.5, PS 24.7,; ACR 15.5,; (also Laptop Win 11, ver 23H2, LrC 14.0.1, ; ) Camera Oly OM-D E-M1.