Best way to archive RAW files with Lightroom edits -- outside of Lightroom?

[runswithsizzers]

Currently my Lightroom Classic catalog includes two folders for every roll of film which I have digitized.

1. One folder has the RAW versions of camera-scans of my negatives. Those RAWs have been edited including NLP conversions, titles, captions, keywords, etc.

2. The second folder consists of positive JPEGs created from the first folder.

I would like to remove the first set of folders from Lightroom to conserve space on my computer’s hard drive, but first I want to archive the RAW files to external storage in case I decide to re-edit them later. If I save the RAW files, only, then I will loose the considerable amount of labor I spent on the edits and metadata.

What is the best method to save my RAW files along with my edits/metadata so they can be reimported back into Lightroom later, if needed?

[dj_paige]

I would like to remove the first set of folders from Lightroom to conserve space on my computer’s hard drive, but first I want to archive the RAW files to external storage in case I decide to re-edit them later. If I save the RAW files, only, then I will loose the considerable amount of labor I spent on the edits and metadata.

By @runswithsizzers

This is a misunderstanding. If you save the RAW files and they are still cataloged in the Lightroom Classic catalog (in other words you haven't removed these photos from the catalog), then simply moving the RAW photos to some other disk and re-linking the LrC catalog maintains the edits. You lose nothing by saving the RAW files only. Thus you save space on your internal disk and don't have to perform the task of removing the photos from the catalog.

The idea of removing photos from the LrC catalog and then re-importing them back into the LrC catalog is flawed, and unnecessary extra work. Leave the photos in the catalog, its less work and guaranteed to work, how simple is that?

If you have already removed the photos from the catalog, then the question becomes where are the edits stored? Did they get written to XMP files? Then simply importing will allow the edits in the XMP files to be used with the associated edits.

[runswithsizzers]

Excellent! Thanks for your reply. 

Additional information: I plan to archive the RAW files on optical media which is read only. So, if I link Lightroom to the moved folders for re-editing would that even work?

And if I wanted to work on those RAW photos using a different computer / different version of Lightroom, would that still work?

Presently, I do not have Lightroom set to write to XMP files for all photos. But I did just learn how to make Lightroom create XMP sidecar files for the specific files I want to archive.

Would this work?

Create XMP sidecar files in the same folder with the RAW files

Quit Lightroom

Copy the the RAW files + XMP sidecar files to optical media

Delete folders with RAW+XMP files from my hard drive -- or should I delete the folders from within Lightroom?

Then, if I want to edit an archived file (unlikely, but possible), will I be able to import the RAW+XMP files back into Lightroom and see my previous edits, just as if the RAW file had always been in my Lightroom catalog?

[dj_paige]

I will stick with what I described as the "best" method, in my opinion. Move the photos from the internal drive to an external HD and re-link in LrC. There is no re-importing of archived files. I don't know if this process will work with optical read-only disks, as I have never tried nor have I ever heard this discussed. The other drawback on optical read-only media is that you cannot save XMP files for re-edits. However, since the edits are in the catalog, and you are not going to remove the photos from the catalog, it may not matter if the xmp files are not updated, LrC will use the edits in the catalog.

[runswithsizzers]

Let me ask the question another way:

If a friend uses Lightroom Classic to edit a RAW file on their their computer, and they create an XMP sidecar file for that RAW, and they provide me with copies of those files, can I import them into my Lightroom and see the edits made by my friend, as well as any metadata like captions, titles, etc?

[GoldingD]

What is the best method to save my RAW files along with my edits/metadata so they can be reimported back into Lightroom later, if needed?

1. Do not under any circumstance remove the original photos from the catalog.
2. Understand that the original photos that you imported into the catalog do not actually reside as part of the catalog, in the catalog they only exist as a record in that database. Some members do not understand this.
3. Understand that the exported photos also do not actually exist in the catalog other than a record in the database
4. Remember that the original photos are kept wherever you decided upon import
5. Remember the export are kept wherever you choose at export
6. Do not ever ever completely delete your original photos, they are your masters
7. As for exports, they can be easily re-exported, not happy with them, sure delete.
8. If you want to relocate your original photos, then accomplish that within LrC in the Library module within the Folders panel

It sound like you want to cut down on the size of the catalog, some things to remember

1. The catalog is a database, it contains records of the actions taken. Records include photo location, library mods, develop mods, etc. The increase in space used by that catalog increases a bit with each mid, but not by much, and the number of photos is not a big concern
2. the Library previews folder does get directly effected by the number of photos. Some delete the previews to make space, but much of that will be recreated/used upon LrC restart.

Ref:

• https://mastering-lightroom.com/lightroom-space-part-1/
• https://mastering-lightroom.com/lightroom-space-part-2/

You mention that you want to archive the original photos. In one location. This brings up points about backups that I suspect you are not doing.

1. Backups should never ever be on the same hard drive as the original work. That hard drive dies, both original and backups die.
2. More than one backup should exist, and different media should be used for them.Typically a 3-2-1 data protection strategy that recommends having three copies of your data, stored on two different types of media, with one copy kept off-site. For example an external hard drive, a different external hard drive you remove and hide away, and perhaps a NAS or the cloud.

Ref:

• https://www.veeam.com/blog/321-backup-rule.html#:~:text=strategy%20with%20it.-,What%20is%20the%203%2....
• https://www.seagate.com/blog/what-is-a-3-2-1-backup-strategy/
•  

[runswithsizzers]

Yes, thank you, I do understand how Lightroom's catalog works. And yes, all of my files are regularly backed up to two external drives via Apple's Time machine (one HDD, one SSD) and a third copy gets backed up to a HDD via SuperDuper! There are additional copies of my photos, as well.

Right now I am trying to figure out how to archive a very limited sub-set of my RAW files to optical media. These particular photos are digital copies of film negatives which I don't expect to need anytime soon. And I still have the negatives, so I could re-copy them if needed. The problem is how do I archive these RAW files without loosing my LIghtroom edits, captions, keywords, etc?

Will this work:

From within Lightroom, create a folder called "RAW Archive" and move the RAW photos into that folder. 

Burn the RAW files in the "RAW Archive" folder to an optical disc (or two). 

Log out of Lightroom.

From the Finder, trash the contents of the "RAW Archive" folder (but leave the files in Lightroom's catalog)

Now if I try to edit one of the (missing) RAW files I would expect Lightroom to ask me to find it -- at which point I could copy the file from the optical disc back into the RAW Archive folder, and point Lightroom to that location?

Will Lightroom freak out as soon as the files go missing, and start nagging me about it? Or will I only be notified about the missing files if I try to select / edit one?

[runswithsizzers]

To answer my own question:

First, in Lightroom Classic, I selected the folder of RAW files which I wanted to remove from my working hard drive and used: File>Export as Catalog. I then burned that exported LR Catalog to a Verbatim BluRay M-Disc for archival storage.

As a test, in Lightroom Classic, I selected all the RAW files in the folder and deleted them (Delete from Disc), and then I Removed that folder from Lightroom. All gone from my iMac.

When I tried to re-import the archived LR Catalog directly from the M-Disc, that did not work. But after copying the archived LR catalog from the M-Disc to my Desktop, then I was able to re-import the folder back into Lightroom (File>Import from another catalog). It appears that all my edits, Captions, Keywords, etc. (including Negative Lab Pro custom metadata) survived the export/import process -- so for me, this is a workable solution.

[runswithsizzers]

I forgot to mention, for this to work, the Exported LR Catalog must include the original RAW files, which will happen only if the box in the Export dialog -- "Export Negative Files" -- is checked.

[dj_paige]

I still feel this is more complicated than necessary. A simpler method is to make copies of the photos onto whatever disc you want, optical or hard disk. Then you delete the photos from the disk where they are stored (and where you have space issues) but you don't remove the photos from the catalog. You don't have to do anything to the catalog, no exporting a catalog, no importing the catalog later. When you want to use these photos in Lightroom Classic at some point in the future, just use an operating system move to copy the photos back where they were. Photos take up far more space than the catalog itself.

A drawback to this method is that once you create the catalog(s) on the optical disc, and let's say you do this every month or two, then how will you identify which of the many archived catalogs is the one you need? A few years from now you won't remember. On the other hand if you don't remove the photos from the entire catalog, that catalog will help you find the photos and know what dates/months the photos were taken and give you some idea which catalog and photos on the optical disk is the one you want.

For anyone reading along, this process does NOT eliminate the need to have backups of these photos, or backups of the catalog(s). Writing photos (and catalogs) to an optical disc does not mean you don't need backups.

Even easier is to buy an external HD and move all your photos to the external disk, while leaving them in the catalog. See www.computer-darkroom.com/lr2_find_folder/find-folder.htm or https://www.lightroomqueen.com/move-photos-another-hard-drive-leaving-catalog/ Fragmenting your catalog as suggested by using the archive to optical disk method seems to me to be a major disadvantage.

[runswithsizzers]

@dj_paige Thank you for your reply - which I am sure is very good advice for someone wanting to use an external hard drive to store the archived files. However for my purposes, the one point that makes me reluctant to use your method is your comment, "I don't know if this process will work with optical read-only disks, as I have never tried nor have I ever heard this discussed."

My test has proven that Lightroom does not want to import a Lr catalog directly from an optical disc, so I suspect (but do not know) Lr will not be able to link to RAW files stored on optical media, either.

---

To further clarify what I am trying to do, RE your comment:

"...and let's say you do this every month or two, then how will you identify which of the many archived catalogs is the one you need? A few years from now you won't remember."

Because my archived catalogs for each roll of film have descriptive names, and because I use software called NeoFinder to make an index of each optical disc which resides on my iMac, I can easily search NeoFinder to tell me which optical disc contains the catalog I am looking for.

So far, my collection of archived catalogs of all the rolls of film I have shot in the past 6 or 7 years fit on two 25GB M-Discs, So at my age, and the rate I shoot film, I doubt I will ever need more than five or six M-Discs, which shouldn't be too hard to keep track of.

---

"On the other hand if you don't remove the photos from the entire catalog, that catalog will help you find the photos and know what dates/months the photos were taken and give you some idea which catalog and photos on the optical disk is the one you want."

For every folder of RAW files to be removed from Lightroom, there is an identical folder of JPEG copies of the photos which will remain in Lightroom. I give each of my photos a descriptive Title in Lightroom, and the JPEG copies have the same titles as the original RAW files. So if I decide I want to re-edit a particular JPEG, but starting over with the original RAW file, then all I have to do is search NeoFinder to see which optical disc has the RAW file I want. This not something I expect happen very often - if ever.

---

"For anyone reading along, this process does NOT eliminate the need to have backups of these photos, or backups of the catalog(s). Writing photos (and catalogs) to an optical disc does not mean you don't need backups"

So true. In my case, optical storage of these RAW scans is a special situation, and not a general backup strategy. I prefer to use my collection of external hard drives for regular backups of the files I really care about, and not have them tied up archiving these large, but mostly redundant RAW scans.

[Bill Sprague]

"I would like to remove the first set of folders from Lightroom to conserve space on my computer’s hard drive...."

There is a lot of writing here and lots of good suggestions.   

I conserve space on my primary drive routinely in a simple procedure.  I have a single externl USB drive.  I use the drag and drop functions inside Lightroom Classic's Library module to select folders or sub folders and drag them (in large batches) to the external with a similarly named folder structure.   Every couple of years I have to buy a bigger external.  It is now a 5TB model.   Moving files inside of LrC is a little slower than using Explorer or Finder, but everything remains organized and cataloged.   For current work on the main drive, the external is in a drawer.  It does not need to be plugged in.  

[Jao vdL]

You already got a lot of great answers on strategys on how to integrate offline storage. One thing that wasn't mentioned is the direct answer to your title question. There are two best ways to archive raw files. One is to export as "originals" to the location where you want them. You will get the original raw files with xmp sidecars for all your files. When you reimport these, all the edits, keywords, etc. will be intact. Second is to export as dng. This will give you a neat small package  that includes the raw data and all edits and metadata in a single file. I prefer exporting as original as you will have a untouched copy of the actual raw file. How you now deal with the files is up to your file management strategy but the fundamental action is the export.

[runswithsizzers]

"There are two best ways to archive raw files. One is to export as "originals" to the location where you want them. You will get the original raw files with xmp sidecars for all your files. When you reimport these, all the edits, keywords, etc. will be intact."

Thanks for your reply. It was my hope that the "Export" method you describe would work for me -- but apparently not as well as I hoped. My editing of these RAW files includes using a Lightroom plug-in called Negative Lab Pro (NLP). The NLP plug-in includes the option to record a lot of unique metatdata which are very useful to film photographers who are digitizing their negatives (things like: the film camera aperture and shutterspeed, date of film capture, processing time and temperature, etc.)

I have not tested this, but I have read on the NLP forums that NLP's special metadata does not get written to the XMP sidecar files. One user of that forum posts:

"A quick test confirms that the metadata fields that can be edited under the “Negative Lab Pro” filter are neither written to XMP sidecars nor to files exported by LrC."

To which Nate (who wrote the NLP software) replied, "Correct. The NLP metadata for the original negatives is only stored with the LR catalogue. It’s important to keep a backup of the catalogue."

Apparently, the same limitation exists for DNG files -- that is, the custom NLP metadata is not stored in the DNG file.

So while the exported file could likely be re-imported for further editing, it looks like I would probably loose any metadata which was laboriously entered into the special NLP matadata fields.