P: Invalid XMP metadata written to exported JPEGs, tripping up reading software

[JohnRolfeEllis]

Lightroom 6 often writes invalid XMP metadata in exported JPEGs, including for photos containing lots of brush strokes made with the adjustment brush. This trips up Google Photos, preventing it from showing any of the EXIF metadata. It may well trip up other software.

To reproduce:

1. Start with any image.

2. Use a small adjustment brush with Exposure = 100.

3. Make lots and lots of brush strokes (see the example pic below).

4. Export the image as a JPEG, including all metadata.

5. Load the image to Google Photos and observe that it doesn't show any EXIF metadata.

6. Delete all of the XMP develop settings with:

exiftool -xmp-crs:all= file.jpg

7. Upload that modified file to Google Photos and observe that it now shows the EXIF metadata.

Here's an example pic exported from step 4:

https://dl.dropboxusercontent.com/u/2...

If you extract the XMP metadata with:

exiftool -a -m -b -xmp file.jpg

you'll see that LR has recorded all of the develop settings twice, including all the brush strokes.

Worse, if you examine the file layout with:

exiftool -m -htmldump file.jpg > file.htm

you'll see that the APP1 Extended XMP segments recording the second copy of the develop settings have incorrect segment offsets, with the first segment of the duplicate settings having offset 0, and the following segments with offsets based on that. That's not allowed by the XMP standard, which requires all the segments to have linearly increasing offsets with no gaps. It's easy to imagine how this might trip up a program trying to read the metadata.

[JohnRolfeEllis]

See this thread for users encountering this problem: https://forums.adobe.com/message/7502...

[JohnRolfeEllis]

A workaround is to use the Metadata Wrangler plugin on your exports: http://regex.info/blog/lightroom-good...

Choose the option to remove the XMP "crs" block (which contains the develop settings).

[Mollysnoot]

I'm one if the users affected by this issue in the thread referenced by John.

This is causing me a bug problem and I hope it gets fixed quickly; a DAM application generating invalid XMP metadata is unacceptable.

[SentimentalPhotos]

I'm having the same problem with my Phorganizer Android app. It says it can't find any EXIF data with JPGs exported with LR6. But it can find the EXIF data with JPGs exported with LR5.7 Also with JPGs exported with LR6, the Gallary Android app slideshow function doesn't display the photos in shot date order anymore. It displays the photos in file "date modified" order.

Hope Adobe fixes this soon!

[Drugstore]

Same problem here (among many others)! Culling my photos with Photo Mechanic and having set most metadata there when I import them to LR6 metadata are invalid for jpgs.

[Dawid_Ciecierski]

Came across this issue when Picasa would not see some of the EXIF metadata stored in files (photo location and exposure details among others).

Crazy thing to see such an important aspect of the software goofed up. Hoping for a point release sometime soon!

[boardhead_]

Actually, this is allowed by the XMP specification, which states:

"The GUID is also stored in the StandardXMP as the value of the xmpNote:HasExtendedXMP property. This allows detection of mismatched or modified ExtendedXMP. A reader must incorporate only ExtendedXMP blocks whose GUID matches the value of xmpNote:HasExtendedXMP"

The GUID of the second extended XMP doesn't match, so technically it should be ignored. ExifTool doesn't ignore it though.

- Phil

[JohnRolfeEllis]

Good catch, I didn't notice the different GUIDs, thanks.

So this issue now falls into the same category as the other JPEG format issue: http://feedback.photoshop.com/photosh.... While technically conforming to the industry standard, the atypical layout of the XMP information, surely not intended by the developers, will cause LR's users significant practical problems:

1. Software like Google Photos/Picasa chokes on the XMP metadata and refuses to show any metadata at all. If Exiftool, the widely acknowledged authority on metadata, didn't properly handle LR's layout of XMP metadata, what's the likelihood of the typical mediocre software app handling it properly?

2. Duplicating the XMP-crs information twice can add a lot of bloat to files containing lots of adjustment-brush strokes. For example, it could take 1.5 MB or more to represent the strokes in a photo that's had a lot of brush work. This bug causes that to be duplicated, adding another 1.5 MB to the file size needlessly. This could be significant when publishing Web-quality photos. E.g. a high-quality 2400-wide JPEG might take 1 MB without metadata; with the XMP metadata stored properly, that would expand to 2.5 MB; with the XMP metadata stored by LR 6, it would expand a further 60% to 4 MB. Telling a user not to export any metadata isn't always an option, because she may well want EXIF camera info, keywords, captions etc. exported, and LR natively doesn't provide any mechanism for easily suppressing the develop settings only.

[SentimentalPhotos]

Thanks John! As you pointed out, this was "surely not intended by the developers".

[dfranzen_camera_raw]

John,

Just to be clear, if you omit step 6 in the steps of your original post, do you still see a problem in Picassa?

Thank you,
David

[JohnRolfeEllis]

Yes, if you don't do step 6 (remove all the XMP-crs metadata), then you can't see any of the EXIF metadata in Google Photos.

[dfranzen_camera_raw]

Thanks -- I sort of misread your original post.

[SentimentalPhotos]

Thanks David for looking into this. And thanks again John for your work pin-pointing this situation. Hope this gets corrected soon.

[7obi07801]

Same problem! Thought I was going mad! I've got a website which displays the EXIF data of uploaded images and it displays garbled up and useless information since the update to Lighroom 6! This needs to be fixed ASAP as it's a KEY feature for most Photographers to have accurate EXIF data in their images!

[7obi07801]

Camera Make (Manufacturer) : Empty (used to display NIKON CORPORATION)
Camera Model: CORPORATION (used to display NIKON D750)
Photographer: xx xx:xx:xx (used to display Photographer's name)
Copyright: shows Photographer's last name (used to display camera copyright text)

[7obi07801]

Same problem! Thought I was going mad! My website relies on the EXIF data of uploaded images and it displays garbled up and useless information since the update to Lighroom 6! This needs to be fixed ASAP as it's a KEY feature for most Photographers to have accurate EXIF data in their images!

Some examples:

Camera Make (Manufacturer) : Empty (used to display NIKON CORPORATION)
Camera Model: CORPORATION (used to display NIKON D750)
Photographer: xx xx:xx:xx (used to display Photographer's name)
Copyright: shows Photographer's last name from field above (used to display camera copyright text)

[johnrellis]

I suspect you're encountering a different problem. Does this problem occur for photos that don't have a lot of brush strokes from the adjustment brush? If so, then it's the other problem: http://feedback.photoshop.com/photosh....

[7obi07801]

Thanks, didn't realise there were two bugs! That other link looks more likely as there are no brush strokes used here, just standard adjustments.

[johnrellis]

A workaround is to set the export option Limit File Size To to be a very high number, e.g. 30000 K (30 MB). This was suggested by an Adobe employee in another bug report that might have the same cause as this issue:

http://feedback.photoshop.com/photosh...

I've tried a couple of exports and it seems to work.

[SentimentalPhotos]

First off, thank you so much John for all your hard work. I know it takes a lot of time & effort to figure out this stuff. Adobe should put you on their payroll.

So I tried some cursory tests with this Limit File Size workaround. Now the Android Gallery Slideshow function seems to work better. But still there are some photos that are out of sequence when they were taken within the same second (used to sequence down to Sec./100 I think, no time to research this in detail right now). Also, now one export process runs at 100% CPU. I used to run three parallel exports to use up CPU and still had a little CPU left. Now it's too frustrating getting LR6 to respond quickly enough so I can get the other exports started.

Unfortunately, I don't have the time to do more in-depth research and testing on this. I need to get my work out the door and don't want to redesign/change my work flow processes because of this LR 6 issue. So I back to using LR 5.7.1 even though it doesn't support my GPU & 4K monitor well. Nevertheless, it would be nice to have LR 6 working like LR 5.7.1 EXIF data wise.

[johnrellis]

"Also, now one export process runs at 100% CPU. I used to run three parallel exports to use up CPU and still had a little CPU left. Now it's too frustrating getting LR6 to respond quickly enough so I can get the other exports started."

LR 6 uses all your processors to do an export, making each export go faster, so you don't need to split a large export into smaller pieces and fire them off manually, which is what people used to do in LR 5. But I've seen a lot of reports that LR is unusable during an export now, which suggests it's being too aggressive at using all available CPU for the export.

[SentimentalPhotos]

Thanks John for your reply. I was unaware that LR 6 uses all my processors at 100% for exports compared to LR 5. I've only run one or two large export batches using LR 6 and didn't see the 100% CPU until my "Limit File Size" test. And I guess it would be much easier to start one large batch export anyway.

However, since realizing the EXIF data problem a month ago (which messed up my work flow processes with my customers), I haven't been using LR 6 at all. There is much less risk running LR 5.7.1 for my production work since I know it works within my work flow.

Moreover, using the "Limit File Size" workaround, any ideas about photos taken within the same second? I think the IPTC "Date Created" goes out to 1/100 sec. and maybe LR 6 is truncating the 1/100 sec. part (haven't had time to looking to this). Any help would be appreciated.

All in all, I would love to use LR 6 for its use of my GPU and increasing its response when using my 4K monitor. But right now, I can't trust it for my production work.

Anyway, Adobe should hire you. You are great. Thanks again!
Frank

[johnrellis]

"any ideas about photos taken within the same second? I think the IPTC "Date Created" goes out to 1/100 sec. and maybe LR 6 is truncating the 1/100 sec. part (haven't had time to looking to this)."

I just did a quick test, and LR 6's export does preserve fractional seconds in capture time. Note that the EXIF and XMP capture-time fields can store fractional seconds, but IPTC cannot. So when you export a photo whose capture time has fractional seconds, LR will properly set EXIF:DateTimeOriginal, EXIF:SubSecOriginal, and XMP:DateCreated to have the fractional seconds. But it will set IPTC:TimeCreated to omit the fractional seconds.

[SentimentalPhotos]

Thanks John for verifying this for me. However, given my cursory test, the sequencing in Android's Gallery Slideshow is off when displaying photos taken within the same second.

Never-the-less, once I get a chance, I think I have a test in mind to verify the sequencing in Android's Gallery Slideshow. I'll upgrade one of my smaller LR 5.7.1 catalogs to LR 6. Run an LR 6 export with the Limit File Size workaround. Then see if Android's Gallery Slideshow sequences the same sub-second photos the same way as photos exported with LR 5.7.1.. I hope to do this test next week, maybe Thursday or Friday. I'll get back to you with my result.

[SentimentalPhotos]

I just notice that the LR6.1 June update is ready for download, but I haven't tried it yet. Has anyone tried it and if so, does it fix this issue? The "Bug Fixed" list doesn't seem to reference this issue ( https://blogs.adobe.com/lightroomjour... ). However, I hopeful that it does and they just left it off the list.