Export Keywords doesn't include a byte order mark and Import Keywords doesn't accept one

[johnrellis]

The command Export Keywords > Include Keyword Tag Options (.csv)  should include a byte order mark (BOM) and the Import Keywords command should skip a BOM if present.  Since the use of BOMs is a de facto standard for UTF-8 CSVs, LR's lack of proper BOM handling is a design defect.

Currently, if a user with Excel (the most common tool for manipulating CSVs) opens a LR-exported .csv, Windows Excel will assume the file is encoded in the current Windows code page, garbling any non-ASCII Unicode characters; Mac Excel will similarly garble Unicode characters.  The user must use an obscure seven-step process to import the .csv into Excel properly.  In the first 24 hours of LR 12.2, a user and Rikk Flohr both stumbled over this, thinking the Export Keywords command was broken.

And if a user tries to import a CSV modified by Excel into LR, she'll get an error message, "The file cannot be imported, because it contains corrupted data", because Excel has written a BOM that LR doesn't recognize. The user will have to use Windows Notepad or Mac Text Edit to save the modified file without the BOM. 

Only Unicode nerds whose brains are filled with such useless details will be able to figure this out.

Widely used apps that recognize the BOM: Windows Excel, Mac Excel, Google Sheets, Windows Notepad, Mac Text Edit, Sublime, Mac Numbers. Mac Pages, Windows Word, Mac Word.  I'm sure there are many others.

[Lumigraphics]

Checking files in Notepad++ on Windows and BBEdit on Mac, both Lightroom Classic and Excel CSV exports are shown as UTF-8 without BOM. Both text editors have an indicator and can convert between different code sets. This is on a US English system, CSV exports done on Windows. BBEdit display is similar.

I was able to open the LR export CSV just fine in Excel 2016, right-click Open With (not using the import wizard) and it opened correctly without errors. Have not tested in Mac Excel or Lightroom.

[johnrellis]

[This post contains formatting and embedded images that don't appear in email. View the post in your Web browser.]

The issue with Excel arises with non-ASCII Unicode characters encoded as multiple bytes in UTF-8.  For example, create a keyword "Activité" and do Export Keywords.  Double-clicking or right-clicking Open With on the .csv opens in Windows Excel 2016 and Mac Excel with the keyword garbled:

Since the .csv doesn't include a BOM, Excel interprets the file as encoded in another 8-bit character set (the current code page on Windows; I'm not sure which set on Mac), and the "é" character gets garbled.

Importing the .csv into Excel using the seven-step recipe correctly opens the file as UTF-8:

If you modify the file by replacing "Activité" with "XActivité" and then do File > Save As, with type CSV UTF-8, then Excel correctly includes a BOM in the saved file. But when you import that saved .csv with LR's Import Keywords, you get this error message:

[stoflom]

If you export a keyword file with non-ASCII Unicode characters (e.g. "ë"), then create a new catalog and try to import the exported file you get the error message.

[johnrellis]

[This post contains formatting and embedded images that don't appear in email. View the post in your Web browser.]

"If you export a keyword file with non-ASCII Unicode characters (e.g. "ë"), then create a new catalog and try to import the exported file you get the error message."

Hmm, I don't observe that importing either the .csv or the .txt file into a new catalog, with these exported keywords:

I wonder what accounts for the different behavior?

[stoflom]

I suppose it must be something else in my keywords? I assumed it was the ë when I came across this thread. I tried both txt and csv without success although it worked without any problem on previous releases. I can drop my export file here if you would like to try.

[johnrellis]

Sure, attach the file here. If it's a .csv file, you might have to add .txt to the end.

[stoflom]

I've attached the txt file, the csv does exactly the same. I'm adding my System Info also. This file is a fresh export from my catalog, I checked again it gives an error: on import in a new blank catalog

[johnrellis]

@stoflom, the problem is with the keyword "Moon, Other, Planets", which contains two commas in its name. If you delete the commas, the files will import correctly.

LR doesn't allow you to create keywords containing commas. Most likely you imported a photo that had been assigned that keyword by an external program, or you used a plugin (like my Any Tag plugin) that lets you create LR keywords containing commas.  It's a separate bug that LR will import photo metadata with keywords containing commas and allow plugins to create keywords containing keywords.

It's unfortunate that the error message wasn't more specific.

[stoflom]

THANK YOU I really appreciate your trouble. I have not used a keyword manager nor did I import a photo so I do not know how this happened. Nevertheless, it does not make sense that LR can export a file and then not import it!

You have been a big help. I will definitely take a look at your plugin.

[johnrellis]

Moderators, @Rikk Flohr: Photography, since the developers don't consider this a bug, please move this to Ideas.

[Rikk Flohr: Photography]

Engineering has not made a final determination on this thread. 

[PhilBurton]

Rikk,

How can Engineering make any determination other than "Fix this issue, soon?" 

First, I would observe that Adobe is benefitting from the "free consulting" provided by John Ellis (and others).  Second this issue affects most, if not all customers outside the USA, which is a significant share of the customer base.

Sometimes it's better to take a fast decision than get paralyzed by over-analysis.