Exporting pictures (RAW) not working

[RobertSmits]

Hi All,

I am a professional photographer, and know Lightroom pretty much inside and out.  However, since a few months now, I am having difficulties exporting photos, which are located on my NAS, but I even copied the raw files over to my external SSD (Sandisk extreme Pro V2 4Tb) and still, it is exporting images upto a certain point and then stops the export.  Some collections export as expected, while some just "hang". 

I am just exporting edited raw files to jpeg (high res) with jpeg mini plugin installed to limit file size, but I have also turned off jpeg mini pro and tried the export that way, but it doesn't help either.  I can't see weird things in my RAW images in LR and the pictures/RAWs are all linked correctly.  Some had a metadata issue, but I have overwritten the metadata of the files with issues in LR, which also doesn't do anything...

I have even tried to export the collection in smaller parts (24 images, but even that 'hangs'). Seems it is often hanging on the same image, but as stated, when I look that up, I can't find anything wrong with the RAWfile.

Any other ideas I can try?

Thank you in advance, I really need to get some of these images to a few clients.

Best regards,

DrBobke

[GoldingD]

Where is the catalog kept?

[RobertSmits]

Hi GoldingD, since Adobe doesn't want you to keep LR Cats on a network drive, it is on my internal SSD (and then synced through Synology Drive to the Synology and then to my Desktop PC).  But, I have been using only my MBP for months now and the sync with Synology Drive has been 'suspended' for the last 40 days, as I would sometimes get corrupted catalogs, which I think is the reason (synology drive sync, or possibly a network glitch which causes an issue - I haven't had an issue when it was 'suspended').

I have however, always been able to successfully rebuild the corrupted catalog(s), the only thing was, that I always had to rebuild the smart previews and update the AI settings (local adjustments using subject selection). I have also noticed that virtual copies seems to have lost some of the metadata (which lens, focal lenght and camera settings, I think date and time are still there, yet it always puts the virtual copies (b&w) in my library first, grouped together, then followed by the original raws, after all the B&Ws. whereas it should be original (colour), followed by a B&W image...

The catalogs work fine for the rest of the time, just exporting is an issue (even when it's on the external SSD, but I have also tried moving the raws to my internal SSD in the past, to no avail).

[JohanElzenga]

There used to be (and I think still is, but it is long ago that I last needed to do this) an issue with repairing a corrupted catalog. During that process Lightroom renames the previews and smart previews (and apparently also the masks). That should be a temporary rename action in my opinion. When the catalog has been successfully repaired, the previews, smart previews and the .lrcat-data should be renamed again so they match the catalog name again, but that did 't (and still doesn't?) happen. You can do it manually, however. That will bring these things back.

[RobertSmits]

Interesting, I didn't have so many corrupt catalogs in years, it was only after looking for a more centralized option of storing photos on a NAS, that I am sometimes running into these corrupt catalogs, but as said, when I turned off Synology Drive backups (40 days), I didn't have any issues with corrupted LR catalogs.  And to be honest, I don't always do a backup of my catalogs when shutting down, as I often switch between 2 master catalogs (work and private) and sometimes another 'travel/on the road' catalog.  My master catalogs house around 70k RAWS and the other around 160k (reason for getting the NAS is that those images eat a lot of HDD space 🙂 ).

In my case, apparently, it does not rename the catalog after the repair.  I am intrigued by the manual operation though - Do you mean that after repair, I have to check the name of the catalog (it often renames itself to "Lightroom Catalog private-v13_MacBook-Pro-van-Robert.local_Dec-25-185107-2023_Conflict-2" the bold text is what is added by LR.  I have sometimes deleted the "new" part in the past and checked the other "sidecar" files for naming and if applicable, changed those too, but that still didn't "attach" the smart previews, local adjustments etc to it again. 

Is it something like that what you mean? All the help is greatly appreciated!

[JohanElzenga]

 Do you mean that after repair, I have to check the name of the catalog...

By @RobertSmits

No, that is not what I mean. I did not say that the catalog is renamed. What I said is that during the repair process, the previews and other help files are renamed for some reason. After the catalog has been repaired successfully, those renamed helper files still are renamed (while they should not be). That means they don't match the catalog name and so Lightroom Classic does not see them anymore as helper files of this catalog. That is what you observed: you said you lose previews, smart previews and masks after the catalog is repaired.

 it often renames itself to "Lightroom Catalog private-v13_MacBook-Pro-van-Robert.local_Dec-25-185107-2023_Conflict-2" the bold text is what is added by LR. 

By @RobertSmits

I'm pretty sure that Lightroom Classic does not do that, but the synchronization with your NAS causes it. The word 'conflict' suggests that the catalog file is seen as changed in both locations, and that triggers the creation of a 'conflict' copy. And that is very likely also the cause of the corruption.

[RobertSmits]

Hi Johan,

Thanks a lot for clarifying.  However, I am interested in where the helper files/... are so I can rename them, or what would be the suggested course of action I can undertake to *hopefully* solve the issues I am having?

I also think that it's likely Synology that changes the name of the lr catalog file due to sync issues (although that is very weird, as I am usually only working from my MBP, thus files in the Synology Drive (or my desktop), should not have been changed at all...

[JohanElzenga]

These files are all in your catalog folder. To be clear: I am talking about 'catalogname previews.lrdata', 'catalogname smart previews.lrdata' and 'catalogname.lrcat-data'. Because Lightroom will start to rebuild the first and the third file as soon as you launch it, you will probably see two of them. The old renamed copy, and the new copy that hardly contains anything yet. Trash the new copies and rename the old ones, that's the idea.

[GoldingD]

Hi GoldingD, since Adobe doesn't want you to keep LR Cats on a network drive, it is on my internal SSD (and then synced through Synology Drive to the Synology and then to my Desktop PC). 

By @RobertSmits

Hmm, additional reply's below makes me rethink this. 

You state SYNC to the NAS, why?

And it sounds like a simple copy to NAS has been an issue, is that correct?

Then you SYNC to your desktop? From where?

Please expand upon the syncing, step by step, with explanations of how and why. List all hardware involved..

note: I keep my catalog(s) and photos on internal SSD hard drives, I keep copy's of my catalogs and photos on other hard drives, and I keep copy's of same on my NAS. I have plenty of space, so I simply use MS File Explorer to copy those files (as opposed to backup via LrC) and I have no issues. I do not sync from my working catalog or photos to my copy's. Yes that goes against what most do, but... Point is I have no issues with copying to my NAS., and my NAS is just for copys/backups.

[RobertSmits]

Hi GoldingD,

So, my working projects are on my internal SSD of my MBP, but in my master catalog (whether it be "Private" or "Work" pictures), all the "other" edited RAW files, are on my Synology NAS.  Once my files on my MBP internal SSD are processed, I export the jpegs and move the raws from my internal SSD to my NAS.

Since Adobe doesn't want anyone to store CATALOGS on a NAS/networked drive, I have them on my internal SSD drive (catalog and helper/... files).  Those catalog files etc are sync'ed by Synology Drive onto my Synology DS1821+ with 4 HDD's (14 Tb each), in SHR (Synology Hybrid Raid) using an M.2 NVMe drive as cache. This was done on a 1 GBe connection from my MBP to my NAS (which has 3 UTP cables attached with a load balance bond) until today, as I finally received my 10 Gbe PCIe card for the Synology today (and I already had a 10G OWC Thunderbolt 4 converter with a Cat7 cable attached).

So this is all sync'ed onto my Synology and the Synology then syncs that data (again, only catalog files and helper,.. files for LRc) to my WIN11 Desktop PC and vice versa.  So if I would open LR cat on my desktop pc, close it down, Synology drive would backup the cat file and all the changed helper files to the synology and then push that to my MBP (2 way sync).  This goes pretty fast and I know for a fact, that I always wait until this sync is done (verifying that "saved at" timestamp matches on the machine I last worked on the files to the one I want to open/edit on).  But as said, I am using my MBP 99% of the time, as it is faster than my PC.  Also fact : I didn't open the corrupted LR cat on my PC before it got corrupted.  So I was editing on my MBP, closed LRc correctly (for example to open another cat), then went back to the one I was working on, using the same machine and then I would sometimes get a notification that the LR cat was corrupt and needed to rebuild..

Hope that makes sense and provides enough information?

[GoldingD]

Ok.

Hopefully a more capable member, one with network database experience/knowledge will way in. Some of the following may be utterly incorrect.

I suspect that syncing from one catalog, properly on a computers hard drive, to a catalog on a server share (In this case, a NAS) involves database communications/actions to/from both the working and the master catalogs. You are opening and editing both databases, both catalogs. As such, the master catalog on the NAS is being opened or acted upon as a database and at risk of corruption.

Now, typically, NAS do not come with the capability to safely handle a running database, including the very simple but not industry standard LrC catalog.

As such the catalog on the NAS is at risk.

Also, that bit about syncing to the Windows PC. That is from the NAS master catalog? Also at risk.

My risk/benefit approach would be to find room, perhaps on an external hard drive for that master catalog, and copy it to the NAS post edits, just as a backup/copy, never ever opening or syncing the NAS copy.  As for the Windows PC, similar involving hard drives, be they internal or external. Some might partition the external hard drive used on the MAC as to support both MACOS and WINOS (both ways). (be aware that may involve a poor partition standard), Or perhaps copy the master from NAS to external connected to Windows PC (by the way, the copy(s) to an external HD or two can be additional backups as such.)

[RobertSmits]

Thank you, GoldingD,

One of the reasons Synology has started Synology Drive, is because loads of (pro) photographers need more storage than most HDD's can accomodate (in my desktop pc, I have 6 spinning drives (two of which are 14TB, the others 12, 2*6 and a 4TB (I think)), two NVMe's and 1 SSD.  But that set up was never going to last as my storage needs are growing, thus the reason for looking at a NAS.  

With SHR, I have created """one disk""", which is 38,2 TB, which makes everything a lot simpler.  Also having an easy connection for my MBP, in stead of sharing disks from my PC (which obviously is just one disk and therefore, read/write speeds will be lower than in a NAS, as the disks are 'stacked'). That also means I can connect via a VPN to my master RAW files while I am 'on the road' and a client absolutely need a picture "yesterday" because he/she deleted the image accidentally on their side and can't recover it (yes, happened a bunch of times already).

But then you get into the issues that Adobe put in our way - no support on NAS-systems, that is where the Synology team shines in my opinion, as they created Synology Drive.  You "should" always have an up to date LRcat and helper files on both (or more) machines that you are using to edit from. So the master catalog is changed on my MBP upon shutdown, Synology Drive is then sending over that data to the NAS "Synology Drive", which is then sent over to my Desktop PC. I can open the same catalog, including all the changes I have made on my MBP, or on my PC and I can continue to edit.  This process runs smooth and I know many other photographers using the same approach (for example @henryhudson photography), that also had/has a Mac and a WIN pc. All is working (except drive letter assignment for the NAS in WIN11, which is not carried over to Mac, but he found a workaround, which works for me too).

Maybe he can weigh in here too, see if he has similar issues and maybe found a workaround for that part too?

When I create a backup of my LR Catalogs, I do that to a separate "backup" folder on the NAS, but as said, I am working big catalogs and the backup usually runs for an hour or even more, which I obviously can't do while I am working, but I try to do it once every couple of weeks before going to bed...

[dj_paige]

I suppose that the usual things to try apply here

• Turn off the GPU for exporting (uncheck the box shown below)
  
  
• Reset preferences https://www.lightroomqueen.com/how-do-i-reset-lightrooms-preferences/

[RobertSmits]

Hi dj_paige,

Thanks for chiming in. I have also tried exporting the edited raws on my WIN11 desktop PC and in one folder, I noticed one issue on 2 PSD files.  Both showed a Black exclamation mark on the photo (library view - "Lightroom has encountered problems reading this photo), which I don't think I have seen when I was on my MBP (but it could have been there, as I didn't check in detail).  When I went through explorer and viewed the files, tried opening them in PS, I had another error (photoshop could not open because an unexpected end-of-file was encountered).  I have then looked up the two PSD files from an old backup, copied them over to the ext SSD (where I put the rest of the files) and all exported without issues...

However, I still have two other old folders that I need to export, which don't have the same error with the black exclamation mark.  I did however, copy over all the RAW files, incl the xmp files to the ext SSD location, but the export still encounters issues and I think it's always the same virtual copies that have the issue.  

What is weird, is that in grid view, all images appear, but when I go through them in the library or develop mode, some virtual copies, don't appear, the screen stays grey (background colour).

Those virtual copies, along with the "working" virtual copies, do not appear to have any metadata (lens/camera info, date/time,..) Also, the metadata options (rightclick) on the virtual copies (working and non-working), stay greyed out, the originals are fine to change (save metadata to file/ read metadata from file).

There are no errors to be seen on the files or virtual copies in the grid view, but I do think that images where I don't see the image of the virtual copy in library/develop mode, is then ones the export is hanging on..  This is partially confirmed by the fact that when I select "Attribute" and then "kind - originals" and export those, I have NO issues with the export.

I have disabled my hardware GPU acceleration, but that didn't help either (Nvidia GeForce RTX 2070 Trio Super 8Gb). But that seems to have been weird, as it also happened on my MBP, which obviously uses different hardware.

I did not reset my preferences yet, as I have a lot of custom stuff setup, which I don't want to loose or having to setup again on a "maybe" (add-ons, publish services, self-made presets, bought presets, etc).  Especcially because I feel it's having to do with the virtual copies, not the originals themselves...

[dj_paige]

I have disabled my hardware GPU acceleration

Using the LrC Preferences dialog as I showed?

If you have customized preferences, make a copy before you try deleting the preference file. Then give it a try; if it doesn't work then restore the customized preferences. I keep a "Known Good" copy of a good preferences in a place where LrC won't change it at all times, so I can always get back to them if needed.

[RobertSmits]

Hi and thanks for your quick reply!

I turned the GPU accelaration to OFF first, but have just tried by using the suggested approach in just disabling the GPU export option, but it doesn't change anything for the export of the virtual copies..

Just to 100% sure : (on WIN11), the preferences file that I want to make a copy of, is located :\Users\username\AppData\Roaming\Adobe, correct? I have an old copy somewhere I'm sure, but just copying over all the Adobe files under roaming (20Gb).

[dj_paige]

Preference file location given here: https://www.lightroomqueen.com/how-do-i-reset-lightrooms-preferences/

An old copy of preferences is nice to have, but you want to make a copy of your preferences as they exist today before deleting the preference file.

I don't understand what you are saying in this sentence. I think you are saying something important, but I simply don't understand.

I have an old copy somewhere I'm sure, but just copying over all the Adobe files under roaming (20Gb).

[RobertSmits]

Here goes nothing... 😮  Hope it'll work.

So when I did the "reset all preferences and relaunch", I was greeted with "Lightroom catalog was not found." 

Which is very strange.. Anyway, I have selected the correct one by searching for the LRcat and opened it.

I did lose some of the plugins I installed, which is pretty crappy, so hope that if I can undo the reset, it all magically comes back again.
Anyway, I still can't export the virtual copies with the missing data... 😞 

Any other ideas that I can try?

[RobertSmits]

Preference file location given here: https://www.lightroomqueen.com/how-do-i-reset-lightrooms-preferences/

An old copy of preferences is nice to have, but you want to make a copy of your preferences as they exist today before deleting the preference file.

I don't understand what you are saying in this sentence. I think you are saying something important, but I simply don't understand.

I have an old copy somewhere I'm sure, but just copying over all the Adobe files under roaming (20Gb).

By @dj_paige

Sorry, forgot to answer this part - on my machine, I have AppData\Roaming\Adobe, with a lot of different files in, which I copied all (also Camera Raw, Illustrator, Photoshop, etc), which altogheter amounts to a whopping 20Gb...