Participating Frequently

Question

Lightroom Classic - import progresses till certain point and then ... no real progress

Forum|Forum|4 years ago
March 20, 2021
13 replies
1436 views

It is Lightroom Classic (not CC) version 10.2, Camera Raw 13.2

I'm rebuilding my catalog.

In a nutshell I am importing relatively big number of photos (150k+) into the catalog. It is a fresh import.

The behavior is the same every time I start the import:

Import starts, it progresses to some point and then no further visible progress.

Looking on the disk’s activity - nothing happens on the disc/network side.

Catalog seats on separated nmve SSD with sufficient disk space (400GB+ free, system SSD has 300GB+)

System: i9 CPU; 32GB RAM; Windows 10; Images are on the NAS with fast dedicated network.

Lightroom manages to import portion of images and then stops doing anything. The portion could be 1000, or it can be 15.000 images at once. Leaving the system for few hours, or 1,5 day – the result is the same: portion is imported and then it stops doing much.
When this stage is reached, Lightroom power and CPU consumption goes down to ~2-4%. Usually it happens when Lightroom occupies 15-18Gb of RAM.

The lightroom has not been able to import all images at the single run.

It is easier to interrupt (pressing the red square next to the progress bar on the top left side of the screen) and restart the process - this gives much higher "number of images per hour".

Any idea why this happens? What could be done to improve the process speed and stability?

This topic has been closed for replies.

DdeGannes

Community Expert

Ok I just have one additional thought. I still suspect that your system resourses is just insufficient to complete the 150,000 K image import in one go. Try the following first make sure the image files are located on disk in the desired folder structure, this may need to be done using your operating system.

When doing the import use the Add option at import, not the Copy or Move options as this will require additional system resourses.

Regards, Denis: iMac 27” mid-2015, macOS 11.7.10 Big Sur; 2TB SSD, 24 GB Ram, GPU 2 GB; LrC 12.5,; Lr 6.5, PS 24.7,; ACR 15.5,; (also Laptop Win 11, ver 24H2, LrC 15.0.1, PS 27.0; ) Camera Oly OM-D E-M1.

A

AndroGenamAuthor

Participating Frequently

"When doing the import use the Add option at import"

That's exactly the process I follow.

DdeGannes

Community Expert

Have you tried reducing the size of the previews at import and ensuring there is sufficient disk space to accomodate the Catalog file and the new previews files?

Regards, Denis: iMac 27” mid-2015, macOS 11.7.10 Big Sur; 2TB SSD, 24 GB Ram, GPU 2 GB; LrC 12.5,; Lr 6.5, PS 24.7,; ACR 15.5,; (also Laptop Win 11, ver 24H2, LrC 15.0.1, PS 27.0; ) Camera Oly OM-D E-M1.

DdeGannes

Community Expert

At Author, Quote “

In a nutshell, I am importing a relatively big number of photos (150k+) into the catalog. It is a fresh import.

Res. This is an unusually high number of images for one import. What file type?

The behavior is the same every time I start the import:

Import starts, it progresses to some point and then no further visible progress.

Looking on the disk’s activity - nothing happens on the disc/network side.

Catalog seats on separated nmve SSD with sufficient disk space (400GB+ free, system SSD has 300GB+)

Res. Lightroom stores the Catalog file in a specific folder with several ancillary file the major one being Preview files which are required for it to function. I have 41,600 files in my Catalog the majority being RAW camera images and the folder is now 109GB. I have built standard previews, no 1x1 previews or Smart-previews so if I use my experience as a guide the folder that houses the Catalog will grow to about 400GB.

You, state the drive is 400+GB what makes you believe the space on that drive is sufficient?

Regards, Denis: iMac 27” mid-2015, macOS 11.7.10 Big Sur; 2TB SSD, 24 GB Ram, GPU 2 GB; LrC 12.5,; Lr 6.5, PS 24.7,; ACR 15.5,; (also Laptop Win 11, ver 24H2, LrC 15.0.1, PS 27.0; ) Camera Oly OM-D E-M1.

A

AndroGenamAuthor

Participating Frequently

"This is an unusually high number of images for one import. What file type?"

Most of them RAW files from the camera (imported to the dedicated catalog some time back).

as I have stated - I had new PC, reinstalled Adobe products - and have decided instead of exporting / importing catalog - reimport all that (also because I also keep the settings / edits for each image file on the files system).

Unfortunately I experinece this "incomplete import" issue when I need to import high number of files - e.g. when I come from the sport event where I have shooted multiple games - this is the most obvious case, there are some others.

I expect system being able to process any number of files at once, at lease giving me the reasonable status of the situation.

"You, state the drive is 400+GB what makes you believe the space on that drive is sufficient?"

I simply check / monitor the system state, system logs and the Explorer showing me the free space - this basics have never failed on me.

At this moment Lightroom takes disk space:
- ~316 GB

- 167.669 Files

- 59.927 Folders
The system reports 614GB out of 931GB of free space

GoldingD

Legend

Was scratching my head as to why I was interacting with a several months old posting, relooked, and noticed because you had made a newer reply. And relooking at that, I see s good question has not been addressed.

And still - no reply from Adobe... and there is no any "formal" way to raise an issue ticket and get a formal reply.
or I am missing something?

That one response by what appears to be an actual Adobe employee is, or used to be (used to be defined as a week or two ago) a rare occasion. The Adobe Techs and/or employees rarely interact in these community pages. When they do (or did, things might be improving) they typically state, were sorry, and inquire as to Lr version, OS, and have you tried resetting the preference file, never to be here'd from again. Somewheres deeply hidden in the how to use these community pages is some sort of statement that the site supposedly for customers to interact, to communicate, (not for trouble tickets, etc) about clear as mud.

You might try the official feedback site where Adobe Techs respond.

https://feedback.photoshop.com/topics/lightroom-classic/5f5f2093785c1f1e6cc40872?cType=PROBLEM&page=1

GoldingD

Legend

One thing apparently lost in translation. I think the one time an Adobe employee interacted above, that employee asked about what if you imported from an actual hard drive instead of NAS. The inquiry appears to have been incomplete or misdirected. I do not think the Adobe employee made a good statement.

So, I will ask separately, with a distinct idea in mind. Communication capabilities.

Inquiry.

What if you copy all those thousands of photos to an actual hard drive, it can be an external drive, although, if internal, just a bit better as to eliminate another variable (first variable LAN speed ). On that hard drive, create a folder structure (folder name, subfolder names) exactly like that on NAS (could just copy the entire folder). In LrC, import those photos.

Later you can redefine the folder location in LrC, should you want to.

P.S. where do you keep backups of your photos? Not the catalog, but the photos. I assume you want to use the NAS for keeping originals on as to free up actual local hard drive space, but where are the backups? Personally, I would do it the other way, original photos on local hard drive, backup on NAS (and another ext hard drive that I lock away)

A

AndroGenamAuthor

Participating Frequently

Back-Up - on another system
Unfortunately, the disk space on the machine where the Lightroom is installed does not allow to have thousands of files.
It has more than enough space for the Lightroom catalog, but ... just small fraction of the images catalog.

GoldingD

Legend

Add another drive. If this is is a laptop/notebook, add an external drive. Placing your catalog on your fastest drive has advantages, also your Camera RAW CACHE, Photos do not need to be on a fast drive, a nice external drive will do.

GoldingD

Legend

(tested via 10Gb connection using Intel 10Gb NICs) more was not produced / tested due to the limitation on "another side", not the NAS.

Most likely component causing that is your switch. Very few of us have purchased 10Gbe switches.

Also, not all 10Gbe switches provide 10Gbe on all ports.

Second component would be the NAS, Only newer NAS for home use and NAS for big business have 10Gbe NIC's included.

I would not consider a 1Gb LAN as fast if communicating to a NAS is involved. Would expect long wait times, but would not expect complete failure.

Is that NAS timing out? Going to sleep?

A

AndroGenamAuthor

Participating Frequently

It is an enterprise grade NAS, the same is true for the Switch and other parts of the equipment - in this area I am completely confident as I have a computer and networks engineer degree and have technological background and profession.
The NAS and network parts have been fully tested and there are no issues in that specific area, as this equipment is used for some other purposes as well.

GoldingD

Legend

Disk activity - the NAS is attached as a local drive

So you have a NAS that you can treat as a hard drive and connect via a USB cable?

A NAS, not a external RAID Drive enclosure .

A

AndroGenamAuthor

Participating Frequently

No USB

NAS share can be mapped using standard Windows network capablities

NAS supports SMB protocol

GoldingD

Legend

this makes me curious, not nessacarily helpful, but...

Images are on the NAS with fast dedicated network

So, what is your LAN infrastructure?

Computer LAN connectivity. Through Ethernet? Using a NIC? As in not WiFi.
NIC in computer? What bandwidth/speed? 100 Mbps, 1,000 Mbps and 10 Gbps
Your NAS? What bandwidth/speed? 100 Mbps, 1,000 Mbps and 10 Gbps
Your NAS, if 10 Gbps, via one Ethernet cable? as in via a 10 Gbps port? Getting at, are you getting full capability.
Ethernet cables from computer to switch, from NAS to switch, Category? cat 5, cat 5e, 6, 7? Fiber?
Switch bandwidth/speed
Computer and NAS same switch? If not, capabilities of switches.

A

AndroGenamAuthor

Participating Frequently

1Gb LAN, connection via the same / single switch, Cat 6 or higher cables are used.
NAS and the computer have NIC on the MB - not extra components, almost a direct connection.
No Fiberoptic.

tested: NAS can coop up to 2,5 Gb LAN with no issues (tested via 10Gb connection using Intel 10Gb NICs) more was not produced / tested due to the limitation on "another side", not the NAS.
Switch capacity:

Fabric: 56 Gbps line-rate

Packet Forwarding Rate (64 byte): 41.664Mfps

Jumbo Frames (bytes): up to 10240

GoldingD

Legend

The portion could be 1000, or it can be 15.000 images at once

During import, what settings for preview creation? With that many, just to speed import up, would think that not creating any previews would help. The can be created later.

During import, any Develop presets being allowed? Would slow things down.

A

AndroGenamAuthor

Participating Frequently

Preview can be on or off - no impact on the issue.
No development presets
add the tagging during the import - on or off - also shows no impact on the situation

GoldingD

Legend

A ohh by the way

Face detection and Address lookup paused, correct?

A

AndroGenamAuthor

Participating Frequently

@GoldingD ,It looks like I would need to add further details – happy to do so.
Face detection - off

Disk activity - the NAS is attached as a local drive - hence I can see the activities using the resource monitor (or even a process monitor - it just generates huge log, hence used in "emergency" cases).
I always use Add when import from the local drive and copy when from the card.
If copy from the SD Card - the destination is always in the "import" folder. Usually files are moved to the "final" decapitation after being processed.

.XMP files are generated when the file is touched in the Lightroom. Hence should not be an issue.

When I refer to the disk activities I mean: Light room has e.g. 80% of import completion on the top progress bar, and system stops showing the progress. If lightroom is not touched – the progress bar frizzes on one position (it might be different one depending on the process) and that’s basically it. Lightroom neither finishes the import nor shows any progress, nor does any real work, however it consumes some CPU “power”. In some situations, Lightroom might keep memory consumption increasing up to the limit then system starts swapping.
The only possibility in such cases is to stop import by pressing the red cross next to the progress bar.

and no DNG

GoldingD

Legend

And the above leads to

Looking on the disk’s activity - nothing happens on the disc/network side.

Are you referring to a folder on your NAS that contains the photos?

or

Whatever hard drive, whatever folder, your catalog is in?

Whatever drive, whatever folder you keep your photos on, as long as it is not the exact same folder as the catalog is in. If you do not create those xmp files. Nothing will be written.

But, where the catalog is, the catalog gets larger, previews are created, so, yes, folder space used gors up.

During import that is, Export a different can of worms.

Show more replies

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded