Participating Frequently

Question

Multiple catalogs and "portability"

14 replies
49 views
Translate

I’m using Lightroom classic since I want all of my files locally.

I am scanning using a Nikon Coolscan 9000 every negative and slide I ever took from the time I started photographing in the early 70s until I finally went digital in mid 2000. I estimate there are 50,000 frames all scanned at 4000dpi. The Silverfast work flow works great for color, especially the infrared based scratch and dust removal.

That all changes for B&W. The Silverfast software based scratch and dust removaal really doesn’t work.

So I’m back to Adobe for pretty much the rest of the project. I’d been using the standalone version of Photoshop Elements up until V10, but I’m scanning even the B&W into 16 bit format. Elements down samples to 8 bit so I lose headroom. I briefly used the full version of Photoshop (which included lightroom) in 2014 but tabled the project. Now I want to explore lightroom.

My usage is “project” based. Essentially each roll of film scanned is a unique “project” and the files are organized this way.

That’s the backstory. Now to the Lightroom questions. Photoshop is good for single frame edits but lightroom seems to be more “batch” oriented so I’d like to give it a try.

Sometimes I process the files directly on the server, especially 35mm scans but 6x6 scans are over 500MB each so I will copy the working directory to the laptop, do the editing, and then copy it back to the server. As an example each directory tree will start off as the folllowing:
/roll/catalog - target for the Lightroom catalog files

/roll/raw - Raw negative scans from Silverfast 4000dpi 16bits per channel

/roll/tiff - full sized 16 bit tiff images from silverfast HDR after conversion from raw negatives to positives. Minimal post processing. These are the inputs to lightroom.

/roll/Lightroom_tiff - output from Lightroom to full size tiff images. Saved for subsequent “tweeking” if making enlargements, etc.

/roll/jpeg full - full sized jpegs for sharing
/roll/jpeg 1024 - 1024 pixel downsized for web posting, email sharing etc

I see that I can have multiple “catalogs”. I’d like to have a catalog for each roll of film that is scanned. I think I’ve figured out how to do that. But my quesiton about multiple catalogs is more about “portability”.

So what happens with Lightroom when I do the following operations.

The questions revolve around moving the entire tree. I copy the entire /roll/… tree from the server to the local hard drive. What happens if I create the catalog on the server and start doing some edits, etc. The hard drive copy includes the entire path, so the catalog contents and all the images and edits to that point. If I open the hard drive catalog, will it track the image files through the relative directory paths?
The catalog can’t be shared between two systems at the same time. I understand the database issues. But Is it safe to use two separate systems, do work on System 1 then close lightroom on system 1, then open the catalog on system 2. Does system 1 cache any data that would make System 2’s access see stale data?

tpriborsAuthor

Participating Frequently

So I’m pretty much settled on just using Photoshop and saving the intermediate work at PSD with the adjustments in layers. That way the original is preserved but the saved PSD is portable across the editiing complex. So I thought about going back to Adobe and turning in the Photographer’s bundle for just Photoshop. Turns out the bundle costs less than PS alone. I recall I went through that thought process about 15 years ago too. Get Photoshop for $22/mo, get PS+LR+LRC for 19. Go figure.

Translate

tpriborsAuthor

Participating Frequently

Yes my references to LR are to classic so LRC. I dislike the cloud for editing 700mb files over a 100mbit link.

It's part of why I copy from server to local hdd. Sometimes I'm at the station and on gbit ethernet. Other times I'm in my recliner watching a movie or TV and wifi for a 700mb file sucks

Translate

tpriborsAuthor

Participating Frequently

Another question. I edit in LR and save. If i open with PS are the edits there or donu have to tell LR to edit with PS and how does PS pass changes back to LR?

Translate

gary_sc

Community Expert

Good question. But first, I have to ask, earlier you were referring to LRC, now you’re saying LR. Which are you planning to use? I’ll focus my answer to LRC.

If you have an image you wish to do some further adjustments in PS, you need to first go into LRC’s Preferences/Settings, the third tab is called External Editing. Select that and then you can place PS as one of your editing software. I also have Topaz in the other one. Then, if you right-click on the image, you can select Edit in PS, the image will be sent to PS and when done, returned back to LRC. (If you are starting from a raw file, the image will go stright to PS and will be converted into a TIF or PSD. If you’re sending over a TIF or PSD file, you will be asked if you want to send the original image, or a copy.)

Now, if you right-click on the file’s thumbnail, and select “Show in Finder/Explorer” and you’ve not selected “Automatically send changes to the side-car file” if you open THAT image into PS, it will not show all of your changes. If you have selected that option. It will show all of your changes. If you now save that file (and you have not gone through LRC to send the file to PS, that file will not automatically go back to LRC. To get it into the catalog, place that file in the same folder where you want it from within the Finder/Explorer, then go back to LRC, and in the folder in question, in the Library mode, right-click on that folder and select Synchronize folder…” Any file in that folder that’s not in the catelog will now be added to the catelog and will now show up in the desired folder.

[Note: this happened to me today. I had a pair of photos that would no combine into a Panorama in LRC. So I opened them in PS, did the Panorama in PS and when I saved them, I had to make sure I saved them in the desired folder and to the Synchronize thing, and everything came out just fine.]

Anyhow, getting files from LRC to PS is an easy thing to do, I do it all the time.

Translate

tpriborsAuthor

Participating Frequently

Yes, you can’t scan B&W with IR. That’s why the dust&scratch removal will be done in LR or PS. Silverfast’s IR is so clean, but their software based doesn’t recognize the basic dust unless it’s really aggressive and at that point it attacks “boundaries”. Even their video goes to drawing circles and excluding such areas. I’d started using the clone tool, but it seems to lose the edits so you can’t go back to them. I’ve filed a bug report with Silverfast but they’re in Germany and the time lag is too big. So the amount of work to manually find the spots in Silverfast is the same as PS/LR so I’ll switch to those tools in the work flow. What I’m finding about LR is that it doesn’t deal with even directory name changes. I have them frequently and sort directories by date first. Sometimes a roll of film has only a month/day on it, and after finding other rolls later is the stack it might help zero in on a better date. So the whole environment is always changing, and after processing a directory called “197x-xx Wedding” and finding it’s “1978-11-10 Bill’s Wedding” the link to the database will be long forgotten. It’s too bad the data isn’t kept closer to the files but oh well. The attraction to Lightroom was it’s more batch like handling, but Photoshop files are self-contained.

Think about it. If you put your files on a hard drive, and that hard drive gets a different letter assigned during a USB rescramble, the link to every file is changed. That’s not very user friendly. The whole system is designed around an assumption that it’s on a single system and that the underlying file system never changes… In the end it looks like more work than just keeping PSD files…

Yet another dangling thing is let’s say I’m working on a local copy to start. The starting tiffs, catalog and output directories are on my desktop. I move the entire tree to the server. Point lightroom to the new location catalog. It’s going to decide the files are still on the desktop directory unless I lickety split change it. Copy it to another hard drive and the file location is still on the server. heaven forbid shoudl the server path change…

It actually seems to me right now that the care and feeding of a central monolithic database is prone to lots of silent disconnects too. This was implemented by a bunch of programmers, not a team of software engineers. There is a difference.

Translate

gary_sc

Community Expert

A couple of things: What Silverfast does very well are image adjustments. As I’m sure you’ve already observed, scanning is like taking photographs: the more work you put in your adjustments at the scanning stage will make any adjustments in PS easier and better. (The better the photograph, the less you have to do in PS. And, because scanning is closer to making the original digital image, the less damage it does to the image.)

What Silverfast does poorly, is thinking it’s an alternative to PS. Using the Rubber Stamp tool is an excellent example. It is not efficient in Silverfast and it’s very efficient in PS. Also, Silverfast prides itself on giving you tools to change a color, but PS is much easier and gives you much better results with greater control. As long as you know what you can and should do in Silverfast and what you do later in PS or LRC, the better and quicker you’ll be getting your job done.

I’ve never had an issue of moving fies to different folders, moving folders from one location to another, or any loss of contents causing the dredded “!” to show up on a file. I’ve always done everything within the confines of LRC. However, I’ve never worked with a server or even a NAS, so I cannot provide any support or comment about them.

What LRC is great in is keeping your organization in your catalog up to date. I’m not sure I understand what issues your describing about a USB scramble. Maybe it’s because I have all of my images on a 4 TB drive that I back up to a 2nd 4 TB drive using Chronosync. In one folder, I have all my images, the catelogs, everything in that one folder. When I backup, I’m comparing the contents of that one folder on drive A to that one folder in drive B. So, with all of my images, catalogs, etc., it can take a while to do that initial backup; the syncing is relatively fast and takes place while I’m having lunch or dinner or whatever.

But, what you cannot do, as I’m sure you are aware, is anything in your system in the Finder or Explorer. I had been using Bridge for many years before I started using LRC and what you can do in Bridge is NOT what you should be doing in LRC. It took me an embarrasingly long time to train my muscle memory what I can and cannot do.

I do not know if you read that piece I wrote, but if you have any questions on it, please feel free.

Translate

tpriborsAuthor

Participating Frequently

The workflow is to capture raw images on a mac with the scanner. It captures the raw image but remembers i metadata what negative profile you select and passes that through. It can be overridden. In color I selected from among the negative profiles that gave me the best color and exposure and sometimes it wasn’t even the original film type but most often was the Portra160 NC 6x6 even for 35mm. I found that for the “gold” family and its successors that Gold with an adjusted profile ended up nearly at the Portra 6x6 so I just started using it. For Tri-X (most of my B&W) at least so far on the 6x6 the standard 6x6 profile is kind of flat while what they call TX 400TC brings up highlights and better shadows. In any case, my starting point is the global adjustment that HDR does (sometimes minimal). B&W is much harder because the dynamic range is bigger than color by at least 2 stops so finding the right raw to TIFF step is a search. Like I said before, that’s just to get the basic 4x6 drugstore print with minimal time investment. I have had several negatives that I looked for for ages that I had spent hours in the darkroom - both color and B&W - to get a camera club print. When I found them in the mass of envelopes I pulled some of them aside and spent as much time with Photoshop tweeking the scan to get what I got for the print. In most cases, the scanned image with the processing was better but the prints have faded over the years so a then and now comparison is based on really old memories :)

Now I’m at the end of the color negs. One of the last batch is 6x6 “portraits” I took of my wife in 1975 in her wedding dress for the newspaper wedding annnouncement. The newspaper prints had to be B&W and I shot a roll of CPS. They’re in the queue but my self-developed C22 and C41 negatives have survived 50+ years exceedingly well. PS. We’ve been married 51 years.

I decided almost 18 years ago to scan the film and devoid myself of prints. I scan the negatives and throw the drug store prints (and even the Luster Color prints) away. I shot alot of weddings when I was in college and for the few people that I did as friends I’ve tried to track down and get them the scan and original negatives. They should still ahve prints in their original album. But I’ve got two stacks of 4x6 prints that are going to the trash. My wife pulls a few for her wall frame, but the rest go away. If I put all the prints in albums I’d have a stack to the ceiling. Then when I’m done I’ll copy all the scans to HDDs and give them to my daughters.

If you’ve got a lot of work and it’s over a long time the NAS box with RAID5 is a requirement. I started this project in Colorado and after we moved to Vegas I did a little bit but the house didn’t make it easy. I used the NAS box for other stuff and literally had a single drive failure that was recovered using the RAID5 rebuild. Backups are time consuming and at the time there wasn’t a single HDD that was big enough to copy the entire RAID set. That’s changed recently though.

Translate

gary_sc

Community Expert

Hi, @tpribors, First off, I completely agree with both @dj_paige and @Conrad_C about no reason for multiple catalogs. The ONLY reason I’ve ever had to use more than one is when I was photographing events and people for an organization I belonged to. There was no reason for those photographs to be included with my personal ones.

Next, you beat me! A number of years ago I wanted to digitize about 10k slides I had taken over the years. I have an Epson V800 and I have Silverfast, so yes. A great setup you have as well. What I chose to do was to photograph every slide as opposed to scan them becuase I had a hunch that not every photo I had taken was special, but I also did not wish to toss them out of hand. So, what I did was to photograph them all and as I went thourgh them, if I found a single image or a group of images, I could go back and do a proper scan of them. I am including a link to the process I used below.

As far as scanning silver film with the IR, it can’t be done because the silver-oxide crystals, screw up the light passing through the film. What you can’t get is a good scan, but what you can get are amazing crystialized patterns. I was going to post some of these, but currently these forums are broken and images cannot be attached.

I tethered my camera to my computer, and if you’re not doing that Conrad’s approach of using an Auto Import folder is fantastic.

As far as naming them go, I don’t know if this would be helpful to you or not, but most of my images are from travels I’ve done as well as events. Because of this, I have folder trees of big, small, and smaller. So, for example, I have one folder called UK, within that I’ve got United Kingdom. Inside that I’ve a folder called a folder called “2024-Scotland.” And, inside that I’ve got “9-11-24a_River Forth,” “9-11-2024b_Blackness Castle,” etc. This give me the date and the place we were. In addition to this, since the film-photographed trips were before GPS could be linked to the photo, I also found the location in the Maps section in LRC and dragged the appropriate photos to that location on the LRC (Google) map. This way, I can “see where I’ve been,” which I’ve found to be very helpful on many ways. Lastly, if there is anything special about any of the images, I keyword as needed.

As I said, I’m fairly certain that this is not necessarily relevant to your needs, but perhaps it might give you some thoughts for your needs.

Translate

gary_sc

Community Expert

@tpribors Oh, I forgot to mention, as far as linking your enhancements to your images from within the catalog, one extra thing I do is in the Catalog preferences, there is an option in the Metadata tab for “Automaticly write changes into images’ sidecar files” (or directly into DNG as well). While complete catelog failure is not common, so are house fires so I have that data set into each image so if anything were to happen, the last adjustments would be there (and I also save all of my images and catelogs in the Clouds as well.

Translate

tpriborsAuthor

Participating Frequently

In reading the document referenced it seems that even renaming a directory in the basic configuration will cause work.

This would be really easy if the implementation of multiple catalogs where the catalog is in the same directory tree as the files were implemented as relative file locations. Static file locations that depend on Windows drive letters is very restrictive. Back when I was taking lots of my pictures I also worked for Digital Equipment Corp implementing storage systems (cluster storage) for VAX Clusters. VMS hid physical drive locations by using “logical names”. A volume’s logical name was the same whatever unit number plug was on the drive… I thought Cutler would have taught Windows about this some 30 or 40 years ago.

Translate

tpriborsAuthor

Participating Frequently

OK, so I think I need to add a little more information.

I understand about the restrictions on having multiple catalogs open. That’s a restriction I can live with. But something I didn’t mention is that leaving the files on the server doesn’t work. Some of these images are approaching a gigabyte each. Direct server access is tolerable for smaller 35mm scans, but for the 6x6 scans image load time is an issue. If editing a single file I might leave it on the server but if I’m working on a group, local drive access times are 100x faster. So my plan was to copy the directory tree, make the bulk edits, and then copy it back to the server.

The other thing is that once a “roll” is processed, I rarely come back to it. I might have to do so with individual files, but once the final raw->tiff->(lightroom or photoshop)->final_tiff->jpeg is complete, I only go back for those images that are production worthy. In other words, the first work product is as good as the original prints I got from wherever they came from (most from Luster Color back in the day). When I go back today for enlargment quality I go back to the tiff or final_tiff file individually. So essentially what I’m getting in the original Silverfast post processing (color) or the lightroom step is scans of the same quality of the drugstore prints.

But the key to all of this, is that once done, they’re pretty static and the overall catalog wouldn’t even have had my work flow included it from the beginning.

Adidtionally, no matter what, folder names include date and location and are changed frequently as I find better information about the details. I changed 3 folder names in the last week. I imagine the basic configuration will struggle with this. In the end, neither the catalog OR files will be static. I have several machines and portability between and among them is important. What IS important is that the changes remain with the image when moving them between these systems. It is my understanding that the catalog directory is the location of these changes and if the catalog has to be on one system then lightroom isn’t the place I want to be. I’m happy to take additional work to make it so, but if it depends on such a static arrangement in an ever-changing environent of file names and locations I’d probably be better off using Photoshop and retaining the PSD files. At least they are self-contained.

Translate

Conrad_C

Community Expert

Given the requirements of your workflow, maybe Lightroom Classic isn’t the answer. If your goal is to use the editing tools in Lightroom Classic, but the catalog doesn’t fit how you work (you don’t really need to have constant access to past projects) and the catalog actually gets in your way, maybe this would be better:

Stick with a portable folder hierarchy, but use Adobe Bridge to browse them wherever they are at the moment. And when you want to edit one or more scans using the Adobe raw toolset, select them in Bridge, then choose File > Open in Adobe Camera Raw to see the selected files in a filmstrip in Camera Raw. Then you can use all the same processing and masking tools that you would find in the Develop module in Lightroom Classic, and you would never have to deal with the overhead of a catalog.

Instead, you would want to be mindful of how Camera Raw edits are stored as embedded metadata (for TIFF/DNG film scans) or sidecar files (for camera raw files) instead of a central catalog.

When you need a copy with the edits baked in, click the Save icon in Camera Raw to get a set of export options that matches most of the same options you would have used in the Export dialog box of Lightroom Classic. Or, in Camera Raw click Open to send Photoshop a copy with the Camera Raw edits baked in.

Translate

Conrad_C

Community Expert

I have also been scanning my film archive with an old CoolScan for many years.

Multiple catalogs: Don’t do this unless you have a very good reason to have one catalog per roll. Otherwise multiple catalogs can be very limiting, largely because Lightroom Classic can’t have more than one catalog open at a time combined with not being able to do things like search across catalogs. I have my archive in a single catalog, so at any time I can organize and curate a project by easily gathering images taken in any camera, film or digital, from any folder. That “playlist” of photos can be stored as a Collection. That would be challenging and slow to do if I ever needed pictures that were in separate catalogs that could only be opened one at a time, and it is not possible to make a Collection across catalogs.

The other major penalty for multiple catalogs is when they upgrade the catalog format. If you have one catalog, the next time they update the Lightroom Classic catalog format (usually once per major version), you have to update the catalog. With one catalog, you are one and done. If you scanned 500 rolls of film organized into 500 catalogs, then for example three years from now if you are using a later version of Lightroom Classic and need to create a presentation of photos from 23 different rolls, and you haven’t looked at those catalogs in more than a year or two, every time you open a catalog Lightroom Classic will prompt a format upgrade for that catalog. So instead of opening old catalogs just to quickly look for a photo, you would have to sit through 23 catalog upgrades. It will help to not be in a hurry…

Is there any reason seeing the folder tree in a single catalog wouldn’t be good enough? The folder for each roll would automatically be listed separately in the Folders panel because that represents the folder organization on the desktop. And it would make multi-roll searches and organization fast and easy.

Regarding the paths, Lightroom Classic works a lot like traditional video editing and page layout applications in that you can re-establish broken path links to source files. If you move the folder tree between volumes, and in the catalog that makes a question mark icon (missing folder) appear in the Folders panel, just right-click the folder and choose Find Missing Folder. If you do this to the missing folder that is highest up in the hierarchy, Lightroom Classic will relink not only that folder, but all folders within its hierarchy, so in most cases this should take only one click.

One tip, if you haven’t already thought of this: Set up an Auto-Import watch folder, and point your scanning software to that. For example, I feed a 6-frame filmstrip into my CoolScan, have my scanning software (VueScan) scan each frame in the strip, and it saves each frame to my Auto-Import watch folder. So all I have to do is sit back and as VueScan scans the filmstrip, I watch each frame automatically import and appear in the Lightroom Classic catalog – so I get to skip the manual import step. I also made an import preset customized for my preferred default Develop settings for film scans, so when it’s done the images are automatically in the catalog at a good starting point for editing.

Translate

dj_paige

Legend

There is no need to copy photos from one location to another in order to edit them.

Catalogs cannot be on a server, but photos can be. Moving the entire tree of folders containing your photos can be done by following the instructions in “Part 2 — Updating Folder Location” in this document.

Your question 2 works if the catalog file is on an external HD and the external HD is moved from system 1 to system 2. Otherwise I don’t think it works as all.

Translate

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.