P: Allow Catalog to be stored on a networked drive.

Report · May 01, 2011

I'd love to make LR more multi-computer friendly. I have no doubt that there's probably database architecture issues and a host of other barriers... But I have to believe that the need for either multi-user or at at lease multi-computer use is widely desired. And yes, I know you can do the catalog import export thing but I find this less than ideal.

Report · Dec 05, 2022

More accurately, SQLite relies on an industry standard feature that wasn't correctly implemented by many network-drive vendors 15 years ago when Adobe tested it:

https://community.adobe.com/t5/lightroom-classic-discussions/operating-lightroom-cc-classic-via-netw...

SQLite works well with correctly implemented servers.

Report · May 11, 2023

Hey all.
I am a photographer of 15 years and I've been using LR for most of that time. My workflow to date has been to work off of a Drobo Direct Attached Storage RAID system. but now that Drobo is bankrupt and no longer supports firmware updates I have been forced to buy a new system to keep my iOS and all other software up to date.
After much research, I went with a Synology DS1823xs+ NAS. I just set up the system, got all of my data copied over into the new drives, double-clicked one of my LR catalogs so I can get back to work, and BOOM: "Lightroom Catalogs can not be opened on network volumes, removable storage, or read only volumes."
This is news to me! How could I have known that LR Cats cannot open on a NAS?
So I am looking for some advice on how to find a solution. Either some tips on a new workflow or work arounds to make this work.
And if anyone at Adobe could tell us why Catalogs cannot open off of a NAS, that would be wonderful. Thanks for the help, all!

Report · May 11, 2023

This lack of support for NAS (or even the cloud) is just one of the many reasons Lightroom becomes more and more irrelevant as time goes on.

Report · May 11, 2023

Synology DS1823xs+ NAS…How could I have known that LR Cats cannot open on a NAS?…tips on a new workflow or work arounds to make this work.
And if anyone at Adobe could tell us why Catalogs cannot open off of a NAS, that would be wonderful.
By

The specs for that NAS say it has three USB 3 5Gbps ports. Is it possible then to simply use it the same way as the Drobo, as a DAS connected by USB? That would get around the network limitation.

The explanation about why catalogs are not supported on network drives is earlier in this thread. Although there are many saying it ought to work, in fact many other DAMs use the same SQLite and have the same limitation. I think Apple Aperture was this way too. It takes a real enterprise level DAM designed for networks to get around this, and although I am no export on them, those seem to be in a whole different price bracket than these consumer DAMs.

Report · May 11, 2023

The only practical way to deal with this - is to split the catalog from the data files. Put the catalog on the local drive and the data on the NAS. Thats it. Don't waste ONE moment of time trying to get the catalog to work over NAS - it will not. You can spend three or four hours reading this entire chain of emails - to find out: Adobe does not support Catalogs on NAS.

Throughout this chain the basis for the reasoning the capability to use network attached storage is that SQLite is not capable of supporting transactions is supported with a 15 year old internal study. While it might still be true for Data Center spans (even less likely), Cloud or Public networks (likely) - it is simply not true for lans, particularly with the two devices connected to the same switch. The idea that SQLLite does not support database transaction semantics today - is also now very false.

For consumers - there is no practical difference between direct attached storage (examples are: Thunderbolt Drobos and Pegusus Arrays) and the network attached storage of a Synology. Adobe is being silly, and we should all consider this to be a marketing restriction. My own implementation of NAS is better than any retail DAS... but it simply doesn't matter. Said more practically - just get over it - Adobe has demonstrated for more than a decade that it is unwilling to lift this restriction. Since Adobe will not even offer a reasonable explaination - we must assume Adobe is not going to change this restriction. That is Adobe's decision to make.

My 5 TB of photos are now on two geographically distributed synology systems, with a local physical backup on both (sync is not the same as a backup), cross linked with file versioning, switch connected optically with multi-link 10G. I built this for completely unrelated reasons, but it is more reliable than my previous DAS gear - Drobos, and then two Pegasus R6 arrays. FWIW - the R6 arrays are fantastic and I remain a huge fan (don't use AFS on them), but Apple's declining support for any kind of server/professional/enterprise anything made the use of those systems obviously impractical, ergo the switch to Synology.

10 years ago, Apple started to divest itself of any semblance of support for direct attached storage in the form of disk arrays - see if you can find any system admin tools any longer... They completed that work several years back when the system was forced onto its own partition in the implementation of AFS. Great reasons for it - but the penalty was that Apple Software RAID really just doesn't work - which I learned the hard way when drive reconstruction - didn't reconstruct. Twice. Because of the direction that kernel drivers are going to support security - don't be surpriseed when SCSI and other drivers needed to run a DAS - do not work.

Hence - the only pratical way to deal with this problem - is to split the catalog from the data files / working directories. Put the Catalog on the local "Data" partition - and take pains not to purchase the smallest Apple internal drives. The conversion to the split catalog / data takes a little doing - but it works, and it is supported by Adobe. Many of us have a workflow which is to create a catalog for every working set, that works too - its just a bit more setup. If you keep separate catalogs like that - don't try to copy them to/from Local Storage <--> NAS. You _can_ - but you are much more likely to make a mistake and lose something.

There are other applications which do 50 to 80% of what LightRoom does. I've tried many. None came close to matching the efficiency of the LR workflow capabilities. So - maddening as it is - I stopped fretting over it, and made it work for me.

Report · May 12, 2023

I find I have to summarize the issues every couple of years to avoid mistaken information from propagating:

LR uses the widely used open-source database SQLite to implement its catalog.

To avoid corruption when a database is accessed concurrently, SQLite relies on the file system to correctly implement POSIX or Windows file locking.

In years past, circa 2007, many NAS implementations, including the then-popular enterprise NFS, didn't implement file locking correctly. The SQLite authors reported having been told that some Windows SMB implementations didn't either, but they didn't provide any authoritative references.

Around 2007, Adobe employee Dan Tull tested LR with a catalog stored on a Mac SMB share with a Windows XP client, and the catalog got corrupted when he unplugged the network cable while LR was running.

Early versions of LR had a config-file option to allow catalogs on NAS, but that was removed many years ago. Regardless of whatever warnings are provided to users, Adobe was afraid that there would be too many users who would ignore the warnings and then come to them to repair their corrupted catalogs.

These days, I think nearly all NAS owned by LR users is based on SMB, Microsoft's protocol. Microsoft has their own implementation, Apple has theirs, and I believe most everyone else (including NAS vendors) uses the open-source Samba. Everytime this issue rears up, I search for current information about the reliability of file locking on these SMB implementations, and I never find anything new.

Since Dan Tull's post years ago, I've seen no other information from Adobe suggesting they've retested LR Classic on commonly used NAS. At this point, I doubt that Adobe would ever put the effort in, given their focus on LR Cloud and their publicly stated reduced effort on those parts of LR Classic not related to Develop.

Report · May 15, 2023

Several years ago, LrC switched to accessing SQLite in WAL (Write-Ahead Log) mode. According to the documentation, WAL mode limits database access to the (LrC, in this case) processes on a single PC, and these processes coordinate through shared memory. I used several tools (watch, inotifywait, wireshark) to observe catalog operations while having LrC sync develop settings across >3K photos and found that other than the initial oplocks, no other lock requests are sent to the server. SMB/Samba implementations with file locking defects, should there still be any after all these years, shouldn't be an issue preventing a catalog from being stored on a network drive, as I've been doing since 2011 using the method described here.

However, I recently changed my workflow when I began using LrC v12.3 to process timelapse photos into videos, and have found that LrC becomes unresponsive while importing 400+ photos when both the photos and catalog reside on the NAS. As a result, I've had to kill LrC quite a few times, leaving behind a .lrcat-wal file with the changes that haven't been applied to the catalog. Restarting LrC automatically recovers these, and the catalog passes an integrity check.

I've run some tests, with the following results:

Photos: NAS, Catalog: NAS, Previews: NAS: Fails
Photos: NAS, Catalog: NAS, Previews: SSD: Fails
Photos: SSD, Catalog: NAS, Previews: NAS: OK

Thus, it's not clear that the problem is directly related to storing the catalog on a network drive.

Anyway, processing timelapse photos residing on the NAS with LrTimelapse isn't practical, since it's far too slow.

Report · May 15, 2023

However, I recently changed my workflow when I began using LrC v12.3 to process timelapse photos into videos, and have found that LrC becomes unresponsive while importing 400+ photos when both the photos and catalog reside on the NAS. As a result, I've had to kill LrC quite a few times, leaving behind a .lrcat-wal file with the changes that haven't been applied to the catalog. Restarting LrC automatically recovers these, and the catalog passes an integrity check.

I've run some tests, with the following results:
Photos: NAS, Catalog: NAS, Previews: NAS: Fails
Photos: NAS, Catalog: NAS, Previews: SSD: Fails
Photos: SSD, Catalog: NAS, Previews: NAS: OK

Thus, it's not clear that the problem is directly related to storing the catalog on a network drive.

I just tested:

Photos: NAS, Catalog: SSD, Previews: SSD: Fails

So this appears to be an LrC bug, and not related to storing the catalog on a network drive.

Report · May 15, 2023

"Photos: NAS, Catalog: SSD, Previews: SSD: Fails So this appears to be an LrC bug"

Very interesting. It conceivably could be a problem with the NAS being overwhelmed by too many concurrent requests from LR. You could test that with these steps:

1. Restrict LR to use just one processor:

start "Lightroom" /affinity 1 "c:\Program Files\Adobe\Adobe Lightroom Classic\Lightroom.exe"

2. Disable the option Preferences > General > Replace Embedded Previews With Standard Previews During Idle Time.

3. Disable the option Preferences > Performance > Generate Previews In Parallel.

4. Import with the option Build Previews: Standard.

If the the import succeeded, that would point the finger more towards the NAS. It's possible there's a concurrency bug in LR that is triggered by your configuration that hasn't been encountered and reported by others, but that seems less likely, since many people store their photos on NAS.

Report · May 15, 2023

I tested with the suggested settings ("Replace Embedded Previews With Standard Previews During Idle Time" and "Generate Previews In Parallel" were already disabled) and found that Import/Add worked, but Synchronize Folder failed (twice). I then retested Import/Add with normal start options, Build Previews:1:1 and everything on the NAS, and that also worked, though I'm fairly sure Import/Add with this configuration has also failed at least once. I don't recall Synchronize Folder ever succeeding with a large number of photos. I'm surprised there's much difference between Import/Add and Synchronize Folder.

Report · May 15, 2023

So ambiguous results above where the problem might be...

Report · May 16, 2023

Yes, but at least it's clear that storing the catalog on a network drive isn't causing this problem.

Report · Feb 03, 2024

Feature Request for a possibility to use Adobe Lightroom Classic on different computers across a local network, with implementing a similar solution as DaVinci Resolve Project Server.

With DaVinci Resolve you can choose if you have projects saved locally on a computer, or you connect to the Project Server running on a main computer to open a project from there. The Project Server take care of making sure that two computers are not working on the same project at the same time.

Can this be possible?

Report · Feb 03, 2024

Two issues. One is that the constant writing that's happening with the catalog gets corrupted on a network. The second is database is SQLite which is single user. You'd need a complete rewrite of the database which would probably require a licensed multiuser database increasing the cost of Lightroom.

Sean McCormack. Author of 'Essential Development 3'. Magazine Writer. Former Official Fuji X-Photographer.

Report · Feb 03, 2024

I have been using this method and it works well

Report · Sep 09, 2024

Allow a catalog to be stored on a network drive. This is a much-requested feature that is simple to implement -- the check to prevent it just needs to be removed. I've stored catalogs on a NAS for over a decade without experiencing corruption. I've researched this thorougly, please see the attachement for details and references.

A related feature, for which I'm submitting a separate request, is to allow the location of the previews cache to be changed so that it can be stored separately from the catalog. This would allow the previews to be stored on an SSD to improve performance when the catalog is on a network drive.

Report · Sep 09, 2024

@keg415, thanks for the clear write-up -- that could be useful to many others.

So making a Windows junction to a symbolic link to the remote directory containing the catalog fools LR into thinking that directory is not on a network volume.

A question: LR uses WAL (Write Ahead Logging) journaling mode for the SQLite catalog database. But the SQLite documentation says "WAL does not work over a network filesystem". That page says that when an app requests WAL via:

PRAGMA journal_mode=WAL;

the journaling mode will remain the default mode (rollback) if the filesystem containing the database doesn't support WAL.

So it seems your LR is running in the older and somewhat slower rollback journaling mode. I wouldn't hazard any guesses whether that makes a differnce to LR performance in practice.

Report · Sep 09, 2024

@jrellis: A question: LR uses WAL (Write Ahead Logging) journaling mode for the SQLite catalog database. But the SQLite documentation says "WAL does not work over a network filesystem". That page says that when an app requests WAL via:

PRAGMA journal_mode=WAL;

the journaling mode will remain the default mode (rollback) if the filesystem containing the database doesn't support WAL.

So it seems your LR is running in the older and somewhat slower rollback journaling mode. I wouldn't hazard any guesses whether that makes a differnce to LR performance in practice.

The full quote from the SQLite documentation is:

1. All processes using a database must be on the same host computer; WAL does not work over a network filesystem. This is because WAL requires all processes to share a small amount of memory and processes on separate host machines obviously cannot share memory with each other.

This restriction doesn't apply when all processes using a database do reside on the same host computer, which is the situation with LrC. LrC locks the catalog with both the SQLite database lock and by creating a file in the catalog directory, preventing another instance of LrC from accessing the catalog from another PC.

I am running the latest LrC (13.5.1) and it is using WAL.

Report · Sep 18, 2024

It's a little late but the only way to "properly" host your catalog and images and everything on a NAS is to use LUNs and attach them via iSCSI. I've been doing this for years with zero issues and i have had network fails and power outages and all kinds of madness over the last 9 years and never had a problem with corruption and even if i did i ahve proper backups becuase it's on my NAS.

One thing i will say is my NAS is attached via a fiber 10gig conenction (or it was now i use 25gig) and so i never have speed issues and 100% of my LR data is over the network. Keep in mind iSCSI is basically a local drive that is connected over a network so thats why LR works off of it.