Data merge losing special characters with some sort of encoding conflict

[MTPmedia]

I have a fairly common problem - losing special characters on a data merge.

I thought the cause of the problem was with the data file exported from Excel, but using a csv or even a plain text file correctly encoded to UTF-8 and testing all possible combinations of import options results in losing extended characters on data merge. So a name like Świeżewski is losing those non-standard characters and is coming through as ≈Öwie≈ªewski.

It is not the font (Minion Pro - and I've tested every font I have), because I can copy the word out of Excel and paste it into InDesign without any problems. I've also tried various different encodings and regional character settings.

I do this sort of work all the time and various workarounds have always fixed it in the past, but on this occasion I can't make it work. I would appreciate any suggestions. Using a Mac, which may be part of the problem, I don't know.

Slight update: it appears that InDesign is ignoring the Unicode option if it is selected in the data merge input options. It works if you import using ASCII and appears to also work if you select Unicode, but in reality it is doing nothing in that case.

[Mike Witherell]

Hi MTP,

On macOS, when you export/save as the .csv file, did you try choosing for advanced options changed to western-macos-roman?

[MTPmedia]

Hi,

It doesn't appear to be possible to do that, at least on the Macs we're using. The only saving option for a csv is an option to add a password, nothing else.

I have tried saving as a comma delimited csv and then using Notepad++ to ensure the encoding and languages is correct (and have tested combinations to be sure), but even doing that and converting a a comma separated csv results in InDesign not reading the file. Similarly, converting this file to a plaintext file also results in InDesign ignoring the file when trying to import, regardless of which import options are selected.

[MTPmedia · Accepted solution]

I seem to have finally resolved this after about 6 endless hours of tearing my hair out.

The problem was an encoding one, but also an InDesign bug. The only way I've been able to get this to work is to create a plain text unicode file BUT InDesign was keeping older incorrect versions of the file in its cache, so it would never work. Unlinking the linked data, saving the file, then fully quitting and reopening InDesign, then relinking the exact same file using Tab and Unicode import options has suddenly fixed it. I had been doing that this entire time, but I suddenly noticed that the file name in the data merge panel was slightly different to the file name in the links panel, meaning it was not trying to read the file properly.