P: Does not export all selected images

[GardenersAssistant]

With Lightroom 6 on Windows 7 I have exactly the same problem as described at http://feedback.photoshop.com/photosh...
with Lightroom 5.4.

I select a number of images and use the Export dialog, exporting to JPEG. After the export is finished I get a dialog box saying "Some export operations were not performed. The file could not be opened ()", and then lists the file names of the images that were not exported.

If I delete the exported JPEGs and try again a different subset of the selected images gets exported. On average around 20 out of 70 selected images don't get exported.

Watching the directory where the images are being exported with a file viewer while the export is in progress I can see that the images that are later shown in the "not exported" dialog box list are in fact being created in the target directory but are quickly deleted.

I have had one example of exporting of a single image failing with exactly the same error message.

I have never had this problem with version 5. I just ran an export of 126 images using Lightroom 5.6 which completed successfully, as has always been the case with version 5 and before that with version 4.

[GardenersAssistant]

What an excellent piece of research John. Thanks for taking the time to do it, and to explain it so clearly.

(Now I understand why I sometimes saw an icon for an exported file appear in the target directory then almost instantly disappear.)

[jwnielsen]

Thank you John. You wrote "But a file viewer like Bridge may have just opened the file right after step 3 to construct a thumbnail, and if it still has the file open at step 4, LR's reopen for read/write access will fail". I use AcDsee to view the exported files, and it builds thumbnails on the run as the files are created by Lightroom. AcDsee also sees the swp files (which is shortly after deleted Again by Lightroom). So this effect could be similar to that caused by Bridge.

[Tbabij]

Thank for this great detective work John. Truly shows Adobe broke Lightroom 6 or CC. Kind of like when 5 came out and don't remember how long it took to export with all the actual developed images. Can only hope next update. Are you listening Adobe?

I had to start using WD backup running a continuous backup of all image drives because Acronis was error prone. This explains why my exports are occasionally error filled. For now I track all files that have an issue and Export them again appears to fix just more unpaid work.

[ronald_w]

Having the same problem since day one with exporting and I usually have no other programs open.
It is a shame we had to pay for the program just to be beta testers and to get a fix will take some time I'm sure.

[ssprengel]

John, could you clarify if a watermark is being added to the JPG in either LR 5 or LR 6?

[johnrellis]

No watermarks were added in the exports that I traced. (I'm tracing exports with watermarks right now. I can reproduce the white-image problem, but the trace seems to indicate it's unrelated to having Bridge or another program open.)

[ssprengel]

I would guess everyone has the white-image problem with watermarks or at least I haven't seen anyone say they haven't been able to reproduce it.

Where is the delete for file.jpg before the rename of file.jpg.swp to file.jpg? I don't see that step in your list.

[johnrellis]

I believe that the rename call is deleting the destination file (file.jpg) -- I think that's an option on Windows (but not on Unixes).

[ssprengel]

Another question, do you have PS-CC / ACR 9? If so does the same thing happen Save Image from within ACR?

[johnrellis]

I looked at the trace again and here's how the rename really happens:

10. Rename file.jpg to file.tmp
11. Rename file.jpg.swp to file.jpg
12. Delete file.jpg.tmp

[johnrellis]

I'll look at PS CC / ACR 9 tomorrow or the day after.

[ssprengel]

Do you have Auto Write XMP turned on for either LR 5 or LR 6?

[FeedbackCommunityMember]

I have this problem as well. The files that LR CC cannot open are random as well. I went through and exported the same 108 files and looked for the ones that were missed and the ones that completed and after a few runs there was no rhyme or reason. In fact I even tried exporting single file that was missing and got the error. Then I tried again on the same file and it went through.

I don't have bridge open. The number of files affected is random. In a batch of 108 files anywhere between 2 and 20 files will fail with the files being random and all over the batch.

I have never had this issue in previous versions of Lightroom so I not sure what is going on though this is not cool at all. Export is a vital function of Lightroom. Without it, what is the point of lightroom?

[johnrellis]

No, it was off.

[Rikk Flohr: Photography]

Thanks for the detailed information, John. I have forwarded this to the team.

[johnrellis]

Great, thanks.

[Julie Kmoch]

Great find, John. Can you tell us what export options you're using? I haven't been able to reproduce this so far.

[Julie Kmoch]

After a quick check with some of the engineers, the behavior John Ellis reports is actually expected and longstanding. I'm not sure why it appears different in 5, other than possibly having different export options enabled?

The temp file dance that John describes is intended to preserve the original in case something goes wrong with the file update. Which we find often does on Windows.

We can't write to a Windows file that another process has opened for reading. Export is done in two passes: first we write the pixels, then update the metadata. If another process, like Bridge, Windows Explorer's thumbnailing, or search indexing, grabs hold of the file before we finish the metadata write phase, we'll try our safe-save 10 times before giving up. Our best theory is that the gap between these two stages may have increased slightly, giving other processes a foothold to steal the file from us.

Closing Bridge or other apps that will be watching these fresh exports should indeed help. We'll look at other ways to stretch out the retries to work around it.

[johnrellis]

Julie, I redid the traces of exports in LR 4.4.1, LR 5.7.1, and LR CC 2015.0.1, and Save Image in PS CC 2014 / ACR 9.0 (all on Windows 8.1). I observed the same things as in my previous report and confirmed some of my guesses about the metadata.

I've uploaded a folder of all my traces, detailed notes on how they were made, and Exiftool dumps:

https://dl.dropboxusercontent.com/u/2...

I encourage your engineers (and anyone else) to examine these in detail, and I'd be glad to confer with any interested party on- or off-line.

Observations

- LR 4, LR 5, and ACR 9 use a single pass to export a .jpg.

- LR 6 uses three passes: 1) write the .jpg, 2) read the .jpg back into memory, 3) write .jpg.swp and rename to .jpg.

- The first pass writes all of the metadata, using a typical layout for the EXIF IFDs (as used by LR 4 and 5).

- The second pass opens the file for read/write access even though it is just going to read the file into memory.

- The third pass rewrites all of the metadata, using the problematic atypical layout described in this bug report:

http://feedback.photoshop.com/photosh...

This pass changes the Region Area fields, the XMP History fields, and Instance ID field but no other metadata field.

The LR exports used the exact same export preset, with these options: Export To: Specific folder, Image Format: JPEG, Quality: 60, Color Space: sRGB, Metadata: Include All Metadata (no other options checked). The preset is included in the folder above.

Fixes

As I described above, there are three possible fixes that would guarantee no conflict with Bridge, other file viewers, backup programs, and utilities like Dropbox:

- Use a single pass to write the file, as LR 4 and 5 do, ACR 9 does, and nearly every other program does. Why were the additional passes added to LR 6? Architecturally, there is no need for the inefficiency of additional passes. (Perhaps the additional passes were added by the engineers implementing face recognition, since the only non-history fields changed on the third pass are the Region Area fields.)

- Keep the file open throughout all three passes. (Don't close the file at the end of the first and second passes, but instead seek back to the beginning.)

- In the second pass, open the file for read access rather than read/write access. Then there would be no sharing violation as the other programs and LR both read the file concurrently. This may be the easiest patch but is least desirable, since it still allows other programs to read stale metadata from the file (as they're doing now).

The latter two fixes are just very simple patches to the current inefficient multi-pass design, but at least they ensure a lack of conflict with other programs.

[johnrellis]

To further increase confidence in my observations, I traced LR 5.7.1 and LR 6.0.1 on OS X 10.10.3 using the "dtruss" tool. I added these traces and details to the Dropbox folder linked above.

As on Windows, Mac LR 5 writes exported .jpgs with a single pass, while Mac LR 6 uses three passes.

[Julie Kmoch]

John, I'll file a bug to have someone take a closer look. Thank you for the depth of your research on this.

[FeedbackCommunityMember]

Julie, I need to do more tests as I think I might be able to reproduce the problem, but definitely not on a per picture basis. My scenario is:
- Open Windows Photo Gallery and view already existing pictures in a folder
- Export pictures to the current folder opened by Windows Photo Gallery or any sub folder of the one Photo Gallery has open.
- Wait for Errors.

I believe you should be able to reproduce it as well, though you will need more than 1 or 2 pictures. I need at least 10 pictures to export for this issue to crop up.

As far as I know, export options don't matter. I am only exporting to Jpg at 90%. Everything else is default settings.

I will do more testing tonight, but after reading this post and running a few informal tests, this issue only seems to happen when I have Photo Gallery open. I think even an explorer window open could also cause this issue, but I am not near as detailed as Mr. Ellis is.

[Fondaphotos]

Impressive!

[Sharon VL]

I am having the same problem. Windows 7 pro 64 bit. I don't use bridge or an image viewer.

Sharon

[Julie Kmoch]

To update, we are able to reproduce this and are working on a fix. In the meantime, if you are exporting JPEGs, one workaround you can try is to check the "Limit File Size To:" option (and set the limit high assuming size is not actually a concern). This should avoid the path that's causing the second update.