Importing large amounts of photos to Lightroom (500k+) is failing

[Jorn-T]

Hi! 

It seems like Lightroom hangs when importing large amounts of photos. I've been having this problem several times over the past years, and right now I'm starting from scratch with a fresh install of Windows 11 and Lightroom Classic.

Trying to import a folder with 580k photos to a new, empty Lightroom Catalog.

Catalog files are sent to a 2TB, 7300 MB/sec m.2 SSD.

Photos usually resides on a 4x16 TB RAID-5 array, but right now I'm trying to import from a 3x14 TB RAID-0 array, read speeds about 600 MB/sec.

Computer specs: Intel i9 14900K CPU, ASUS ProArt Z790 motherboard, 64 GB 6000 MHz RAM.

It seems like the RAM is filling up after a while, and importing slows down.
Now, the import process has taken 48 hours and all photos seems to be finished imported to the catalog. BUT - "Fetching initial previews" still has a long way to go, probably a week or so, when looking at the progress bar. It should not be so time consuming! 

Lightroom now hangs when I try to press the "x" on the Fetching initial previews bar to stop preview generation. The whole system is slowed down and computer is practically unusable.

Seems like RAM usage has been up to 179 GB including page file? CPU/disk usage has went lower and lower, while memory usage has went upwards.

Is there a workaround for this? A setting I should change?

Kind regards,

- Jørn T

[Ian Lyons · Accepted solution]

The workaround is to break your import down into smaller batches. For example, 100,000.

[Jorn-T]

How do I break up the import without messing up the folder structure? By choosing subfolders and sometimes folders inside subfolders to import, the folder structure will be changed?

Is it possible to skip initial preview rendering?

[dj_paige]

I don't see a connection between the number of photos imported, and the folder structure.

As far as I know, you cannot turn off preview creation. You can select "Minimal Previews" which ought to be created quickly. https://helpx.adobe.com/lightroom-classic/kb/optimize-performance-lightroom.html However, this is a tradeoff, if you use minimal or even standard previews, Lightroom Classic spends less time at import, but other operations in the Library Module will slow down as Lightroom Classic has to then spend time to generate larger previews.

[Jorn-T]

I see..

I think I might run into another problem once the preview generation is completed, as the previews are quickly filling my system drive..

I now stopped the Lightroom process from Task Manager, and restarted Lightroom. Seems like preview generation is continuing in the background, as CPU/disk activity is high, but the progress bar is now hidden. 

Preview generation was at 31% when I restarted Lightroom, and it seems like the previews are filling 409 GB so far, probably will fill 1,3 TB when completed. I still got another folder with 400k photos to import, when the preview generation is completed, next week I guess 😕

Which means that my 2 TB system drive will be full before previews are completed.

What preview settings would you use for a catalog with 1 million photos? I'm using a 4k monitor.

[JohanElzenga]

How do I break up the import without messing up the folder structure? By choosing subfolders and sometimes folders inside subfolders to import, the folder structure will be changed?

By @Jorn-T

No, it won't if you just add the photos to the catalog without moving them. It will if you copy or move the photos to a new destination, but I assume that's not what you are doing.