Storage workflow - with removable storage

Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

How to manage storage of photos?

My ideal general approach might be: (but I welcome discussing other methods for interest)

  1. SDcards for the prime originals - we only add to these and buy cards as needed. Never delete. No, not even total dross. They live (usually) offsite (in my work drawer). I already do this.
  2. Treat some sort of removable storage (USB-C flash sticks or fast SDcards) as the "working originals" that I keep nearby at home. We freely delete complete dross via "Flag as Reject" from these.
  3. Adobe Cloud as "backup+sharing2family". Ideally I'll shove up only "Flagged as Picked" and maybe even "Rated 3+ stars" or something.
  4. Adobe Cloud is also useful for "I have some phone pics that I'd like to share back to my Mac and ultimately put on other storage".

I have just bought a Macbook Pro with 512GB of local SSD. I don't want this to be clogged up with my entire collection and I cannot let it sync down everything that might end up on the cloud, as that's going to grow. Use of built in SSD for caching and catalogue and previews would be preferred.

  1. Am I going to need to work off a big external disk/NAS that can always hold my photo collection and expand this as my collection expands?
  2. Or can I work off multiple flash devices, plugging them in as required?
  3. Something else?

2 is preferable as I can work on the train, taking a small subset of work with me.

2 is also more organic - run out of space, buy another device.

Small USB-C flash devices will also be faster than my NAS because I'll buy the good ones.

  • But I'm not sure if it fits with LR's handling of storage?
  • Is LR happy with subsets of photos being available intermittently under one catalogue?
  • Will it prompt me to insert the correct storage flash (stick/drive) if I try to work with a particular photo?

Sorry if the question seems dumb and confused - I'm starting with LR(web/CC+Classic) from cold and I'd like to get organised with the right storage workflow straightaway (processing workflows are easier to evolve).

Thanks folks

Views

1.1K

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
LEGEND ,
Apr 28, 2018

Copy link to clipboard

Copied

Generally, Lightroom does not allow photos to be on removable flash drives (although bugs have been reported where users can do this). I don't know why a "taking a small subset of work with me" requires a flash drive, if it truly is a small subset it should be able to fit on your computer's internal drive.

My only other comment is that using SD cards as your storage for originals is probably not the best use of these cards, as I'm pretty sure that true hard drives provide longer term life than SD cards. I never understood the need to keep the photos on the original cards anyway, computers make EXACT duplicates of the files on these cards (and if the process fails, which happens raretly it is pretty obvious that the process has failed when you try to open the file in LR, it will either not be openable or it will be corrupted).

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

Hi,

Thanks for the reply. To answer your queries:

I thought I saw LR (classic) a few years ago as being sold on the concept it could work with multiple sources of originals (but one catalogue).

1) How do you add the subset to the computer on demand? I presume you mean copy them off the flash stick and onto the internal storage. Is this any different to LR to just plugging in external storage (which appears in a fairly integrated way with MacOSX)? It would still be an intermittent presentation of the originals to LR - which is the bit I'm not sure about...

And it is very convenient as a workflow to think "I'll work on the 2017 collection today, that's this stick here" - No messing around copying back and forth.

2) There's a very good reason to back up originals:

a) To remove the originals to another location as a physical backup;

b) To mitigate "user error" in case accidental mass deletion. That's why I run with a policy of "add, never delete and buy new, not very expensive slow but quality, cards".

c) I'll trust an SDcard over a spinning disk for archival - yes, one needs to periodically check and copy when the card gets older - same with a spinning disk.

d) SDCards are fantastically compact - I can keep lots in a small hard waterproof impact resistant case.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
LEGEND ,
Apr 28, 2018

Copy link to clipboard

Copied

I thought I saw LR (classic) a few years ago as being sold on the concept it could work with multiple sources of originals (but one catalogue).

Sure it does, but what does this have to do with your proposed workflow?

1) How do you add the subset to the computer on demand? I presume you mean copy them off the flash stick and onto the internal storage. Is this any different to LR to just plugging in external storage (which appears in a fairly integrated way with MacOSX)? It would still be an intermittent presentation of the originals to LR - which is the bit I'm not sure about...

Are you referring here to photos that have previously been imported into LR, or photos that have not been previously imported into LR?

It's really not clear to me what you're asking about. Any photos can be imported into LR at any time (assuming they have not been previously imported into LR), and then you can perform whatever work on them you want. If they have already been imported, again you can work on them at any time.

And it is very convenient as a workflow to think "I'll work on the 2017 collection today, that's this stick here"  - No messing around copying back and forth.

I'm afraid this probably won't work, as I said, flash drives cannot be the place for photo storage (except that some people have reported bugs where flash drives can be the place for photo storage).

2) There's a very good reason to back up originals:

I have never said you should NOT make backups. I said that I would not use the original SD cards as a form of backup.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

dj_paige  wrote

I thought I saw LR (classic) a few years ago as being sold on the concept it could work with multiple sources of originals (but one catalogue).

Sure it does, but what does this have to do with your proposed workflow?

I was thinking to copy the originals in batches to USB sticks or SDcards labelled "2017" or "2015-2017" depending how many fit.

Then import those into LR and work with them. It's this part I'm wondering about.

1) How do you add the subset to the computer on demand? I presume you mean copy them off the flash stick and onto the internal storage. Is this any different to LR to just plugging in external storage (which appears in a fairly integrated way with MacOSX)? It would still be an intermittent presentation of the originals to LR - which is the bit I'm not sure about...

Are you referring here to photos that have previously been imported into LR, or photos that have not been previously imported into LR?

Previously imported

It's really not clear to me what you're asking about. Any photos can be imported into LR at any time (assuming they have not been previously imported into LR), and then you can perform whatever work on them you want. If they have already been imported, again you can work on them at any time.

OK - So if I try to edit a photo that's been imported and LR can't immediately locate the original, it will tell me (to insert the device which has the original)? I'm assuming here LR keeps the metadata in the Catalogue that's always available, and a preview on the Mac - but not the whole original.

And it is very convenient as a workflow to think "I'll work on the 2017 collection today, that's this stick here"  - No messing around copying back and forth.

I'm afraid this probably won't work, as I said, flash drives cannot be the place for photo storage (except that some people have reported bugs where flash drives can be the place for photo storage).

?

2) There's a very good reason to back up originals:

I have never said you should NOT make backups. I said that I would not use the original SD cards as a form of backup.

Then we have a misunderstanding - these are NOT the camera's SDCards - these are a backup set I copy everything to on batches, then put away. These are purely for backup/offsite archiving.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

It might be better if I restate the question. How about:

My Mac does not have enough internal storage to hold all my photos[1]. It does have enough space to cache previews and hold the catalogue.

How do I work with LR + a random number of external storage devices?

[1] Jumping to 1TB or even 2TB internal SSD is expensive on a MacBook and, well, one day I may exceed that too, so the problem is fairly generic

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
LEGEND ,
Apr 28, 2018

Copy link to clipboard

Copied

I really feel lost here. On the one hand you say you are talking about previously imported photos, while in the next part of your text you want to import these into LR. Re-importing previously exported photos is never (NEVER) a good idea.

You seem to be treating Lightroom as a file explorer, where you can just attach the flash disk with the photos and work on them. Lightroom is not a file explorer, it is a database, and you must work within the database. You not only have to import the photos (once), but after that if the file is not in the expected location, you have to tell LR where the file is. As I said, Lightroom does not generally allow photos to be stored on a removable flash disk ... so ... you would have to copy the photos from the flash disk to some actual hard disk (internal, external or network) and then tell Lightroom where the photo is located on the hard disk in order for any of this to work.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

dj_paige  wrote

I really feel lost here. On the one hand you say you are talking about previously imported photos, while in the next part of your text you want to import these into LR. Re-importing previously exported photos is never (NEVER) a good idea.

You seem to be treating Lightroom as a file explorer, where you can just attach the flash disk with the photos and work on them. Lightroom is not a file explorer, it is a database, and you must work within the database. You not only have to import the photos (once), but after that if the file is not in the expected location, you have to tell LR where the file is. As I said, Lightroom does not generally allow photos to be stored on a removable flash disk ... so ... you would have to copy the photos from the flash disk to some actual hard disk (internal, external or network) and then tell Lightroom where the photo is located on the hard disk in order for any of this to work.

Let's step back a bit...

If I have photos on external storage devices A,B and C:

I would import the photos on A,B and C once.

At some point, I would want to work on some photos from B and do some editing.

There is insufficient space to have the originals from A,B and C all at once in the Mac's internal storage.

I understand how the catalogue works and the fact it contains a preview.

There is no importing twice.

So when LR needs access to the full sized file so I can see it and work on it, is it capable of dealing with those full sized(original) files being on a number of external devices?

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

Sorry - I thought this would be a standard problem with a stock set of solutions. Even if a professional photographer has a 2TB MacBook, they will exceed that storage soon enough. How do they manage their enormous sets of full sized original files?

On an aside, how does an external disk differ from a flash stick? They usually present in the same way to the OS...

Anyway - my USB-C to USB-A adaptors have just turned up, so perhaps it is time to just try some things with a couple of USB sticks

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Adobe Community Professional ,
Apr 28, 2018

Copy link to clipboard

Copied

I'm a Windows user, so I'm really speaking out of my element here. But I think a better choice would be for you to have your original images on an external hard drive and consider working that way rather than trying to keep your images on SD cards. I don't know if the Mac operating system and Lightroom will allow you to work with images on the SD card, But it really isn't a very secure way to work in my opinion. Depending on how much room you have on your internal hard drive, you could consider using smart previews In conjunction with images stored on an external hard drive. The Smart previews enable editing of your images with the master images disconnected, but the adjustments will apply to the master images when they become connected again. The only restriction is that if you use Photoshop in conjunction with Lightroom, that feature is not available with Smart previews. This is just a suggestion, don't know if it would work in your situation.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

Thanks Jim. I noticed the concept of smart previews and am aware that LR lets you do some stuff with those.

So, if it *can* manage an external disk, can it manage several external disks, or do I need to buy a single disk that's big enough for everything I've got plus the next few years?

Cheers,

Tim

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Adobe Community Professional ,
Apr 28, 2018

Copy link to clipboard

Copied

It would be the same as trying to manage multiple SD cards. You have the freedom to store your images on ANY external hard drive that you can connect to your computer. The more external hard drives you use, the more complex the tracking system will need to be that you use to keep track of where images are located. But you can have any number of external hard drives that you want. My only concern is that your internal hard drive is only 512 GB. That's where the smart previews would have to reside and they do take up space. I think you ought to consider doubling the size of that drive and then work with smart previews and having your Masters on external hard drives. Then, in my opinion, you should be in good shape for quite a while. How much free space do you have on your main drive now?

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

Thanks Jim.

I'm not phased by tracking - I'll have relatively few external disks in very simple sets by year

Yes - the 512GB internal SSD was all we could afford (Mac SSD's upgrades at purchase time are fairly expensive, another £360 to jump to 1TB). I knew 256GB (starting point) was too small, but couldn't stretch to 1TB.

Technically the internal SSD is NOT up-gradable post-purchase (though some companies can manage it).

We'll have to see how we get on...

On the plus side, I'm the one who works with photos - everyone else does music and animation and their stuff stores just fine on iCloud so they won't be using much space

As I have a method for making backup copies onto archival SDcards before I do much, I can at least freely delete total dross from the catalogue and thus the smart previews, so it should hopefully be maintainable.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Adobe Community Professional ,
Apr 28, 2018

Copy link to clipboard

Copied

TimJWatts  wrote

...So, if it *can* manage an external disk, can it manage several external disks, or do I need to buy a single disk that's big enough for everything I've got plus the next few years?

Simple answer, Yes, Lightroom can handle several EHDs, you do not need a single large EHD to handle all your images.

Lightroom can also handle editing these images without the need to load them into your Internal Drive; it will work nicely directly off the EHD.  A two TB portable EHD is not that cumbersome and can be moved with you wherever you need to work.

I find that when I am working, all of the photos I am editing are more than likely on the same EHD so there is no need to keep them all connected.  If you are working out of your house, like on a train, you probably need only the one EHD.  Working directly off the EHD will not slow the editing process.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

Thanks Joe. That's clarified things nicely (with previous caveats from Jim noted re catalogue/smart-preview space).

On an aside, I'm not sure I'm getting why some folks are against external flash (sticks or cards in USB adaptors) - at least in Linux, there's no presentational difference between a USB stick, an SDcard in an adaptor or an external "disk".

Not sure if MacOSX differs in this respect, but a Sandisk Extreme USB3 stick I just plugged in via a USB-C-A adaptor just presents as external storage and I just formatted it ExFAT (most portable format with large capacity support). The only practical difference I can see is "disks" are generally available with larger capacities than cards or sticks - but regarding speed and reliability, Sandisk Ultra USB-C sticks are faster than any spinning disk.

Am I missing something here? Or do folks just equate sticks and cards as "cheap and unreliable" (which they can be if you don't buy decent brands).

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Adobe Community Professional ,
Apr 28, 2018

Copy link to clipboard

Copied

TimJWatts  wrote

On an aside, I'm not sure I'm getting why some folks are against external flash (sticks or cards in USB adaptors) - at least in Linux, there's no presentational difference between a USB stick, an SDcard in an adaptor or an external "disk".

The internal code path used by Lr for devices such as SD, CF cards USB sticks, etc is different from the code path for conventional disk drives & SSDs. The purpose of these devices is for temporary storage (usually between camera and fixed drives/SSDs). Ideally, they should not be written to by Lr, but this seems to be an inherent feature of your workflow. Others have gently tried to dissuade you from adopting this storage regime on the basis that a more conventional regime tend to be more reliable.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Adobe Community Professional ,
Apr 28, 2018

Copy link to clipboard

Copied

I think you identified the main reasons for not using a stick.  I generally see flash sticks as and easy way to transfer files from one computer to another and hence see it as a temporary storage device that easily can be cleaned and used again.  I have a pile of sticks in my desk drawer just for that purpose.  Whereas, the 2 TB EHD is bought and used for a specific purpose.  I have one EHD for photos and another identical EHD for backups using SuperDuper from Shirt Pocket.  Same for all my music.  I buy Avery labels to easily identify these EHDs.

My working catalog is on my main Internal Drive but the one thing I always do is have Lightroom backup that Catalog to the latest working EHD.  Dealing with one catalog and a few EHDs keeps everything simple.  It would be too easy for me to get flash sticks mixed up and I would probably lose images through my own carelessness.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

Hi Ian -

I see - that's interesting (code paths) - I assumed LR would just deal with the standard filesystem layer in the OS without much caring about what the actual endpoint device is. However, it is also clear it must be aware it's dealing with removable media (below). I'm used to the generally POSIX filesystem handing of Linux, rather than whatever Darwin+MacOSX does.

I'm happy to be gently dissuaded, by I like to understand why (is there a reason other than "everyone does it that way" 

I've just done some experimentation with LR Classic connected to my Adobe Cloud account and a few images on both a  Sandisk stick and a Lexar SDCard

Screen Shot 2018-04-28 at 19.13.31.png

I plugged them in, imported the 2-3 test pictures and then ejected the devices. The smart preview remains available, as does a shadow of the 2 devices (above, SDCard-2 and SANDISK-1). Any attempt to edit in PS fails until the relevant device is remounted.

Interestingly - as soon as I connected to my Adobe Cloud account, it started syncing down full copies of 5000 photos I have previously uploaded via LR Web. Obviously, I don't want that, so I was pleased to find that the local sync location can be moved.

In fact, you can move it to any external device, including the SDCard-2 as above ("Lighroom" folder with a typo)

Thank you - I'm starting to understand the mental model a bit more.

As I've not done anything with the images on Adobe Cloud, I might trash the catalogue and reimport the originals.

It would seem reasonable, as you suggest to get a decent SSD external drive and shove everything on there, then reimport. It's clear that if that drive fills up, then it's a simple matter to add a second drive.

The only thing I'll have to think about is the default sync location for stuff that originates from Adobe Cloud - eg stuff I uploaded from my phone direct. If the volumes remain small, that could be the internal SSD.

Cheers and thank you,

Tim

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

Hi Joe - yes, I see where you are coming from. Whilst sticks/cards are becoming very much faster (but not as fast as SSD disks), I have realised I have overlooked the flash endurance as a factor. I haven't got the data to hand, but bashing the wotsits out of a stick might lead to early failure compared to an SSD EHD which has a massive endurance (block erase cycles).

Perhaps that matters less for occasionally pulling a photo into PS, but you probably would want to put a catalogue or the Cloud sync target on it.

But overall, given the screenshot in my last post, I can see how 1 or 2 big EHDs would get less messy on the panel than a dozen cards or sticks as every volume that touches it leaves an entry.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

Thanks everyone - some very useful information there

Now I need to go buy a decent SSD external disk...

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Adobe Community Professional ,
Apr 28, 2018

Copy link to clipboard

Copied

Not really. If you're expecting a big increase in speed with an SSD, you won't get it. The catalog is where all the work is done.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

I try very hard not to buy spinning rust if I can help it SSDs are so much more robust and reliable...

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
LEGEND ,
Apr 28, 2018

Copy link to clipboard

Copied

I have my catalog and photos on an external Lacie thunderbolt SSD (one of the orange ones) on my iMac and it works great.

  As to your original question, about a year ago I formatted an SD card in my MAC and exported about 30 raws and a catalog to it to transport back and forth to a friends house to use on his Lightroom and it worked well.  I was surprised how fast it worked.  I would only do this for special occasions but it did work.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Adobe Community Professional ,
Apr 28, 2018

Copy link to clipboard

Copied

I guess I'm out of my league here since I'm using Windows 10, on an old desktop, and have virtually no experience using a Mac. I will bow out of this discussion now.

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Explorer ,
Apr 28, 2018

Copy link to clipboard

Copied

Thanks mate   I was trialling LR on Windows on VMWare on an old Linux laptop. That my friend is the definition of pain Enough pain to say "hell, I'm gonna get a Mac..."

Likes

Translate

Translate

Report

Report
Reply
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more