Highlighted

Indesign Data Merge (Datenzusammenführung) breaks German Umlaut (ä, ö, ü) despite UTF-8

New Here ,
Apr 15, 2020

Copy link to clipboard

Copied

Hi everybody,

 

The Problem:

Although data merge correctly reads a tab-separated text file, it breaks German Umlaute (ä, ö, ü) that were read in. This seems to be a common problem without a solution. In the original text the Umlaute are correct, after merging into Indesign they are broken. This seems to be a real bug.

 

Solutions that I have tried:

  • Encode the file as UTF-8 - not working after data merge
  • Encode the file as UTF-16 (breaks the new line characters, not working)
  • All possible data import options, following the suggestion here  (Unicode, ASCII) - not working
  • Use csv (not working)

 

This seems like an impossible problem. How can I get this done?

 

Specs

Indesign 15.0.2 x64

Windows 10

 

 

Cheers

Jana

 

TOPICS
Bug, How to, Import and export

Views

175

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

Indesign Data Merge (Datenzusammenführung) breaks German Umlaut (ä, ö, ü) despite UTF-8

New Here ,
Apr 15, 2020

Copy link to clipboard

Copied

Hi everybody,

 

The Problem:

Although data merge correctly reads a tab-separated text file, it breaks German Umlaute (ä, ö, ü) that were read in. This seems to be a common problem without a solution. In the original text the Umlaute are correct, after merging into Indesign they are broken. This seems to be a real bug.

 

Solutions that I have tried:

  • Encode the file as UTF-8 - not working after data merge
  • Encode the file as UTF-16 (breaks the new line characters, not working)
  • All possible data import options, following the suggestion here  (Unicode, ASCII) - not working
  • Use csv (not working)

 

This seems like an impossible problem. How can I get this done?

 

Specs

Indesign 15.0.2 x64

Windows 10

 

 

Cheers

Jana

 

TOPICS
Bug, How to, Import and export

Views

176

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Apr 15, 2020 0
Most Valuable Participant ,
Apr 15, 2020

Copy link to clipboard

Copied

So you get the wrong characters? (This is called mojibake). UTF-8 should be a good choice. What encoding do you choose in InDesign, and what characters does it set for a particular accented character?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 15, 2020 1
New Here ,
Apr 15, 2020

Copy link to clipboard

Copied

Hi there, thanks for the quick reply.

I have attached a screenshot to my original post.

 

I have tried all possible encodings in Indesign without success, the encodings include ASCII, Unicode, and Shift-JIS.

 

The characters are transformed as follows: ü becomes ü, ö becomes Ã¶, ß becomes ÃŸ.

 

 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 15, 2020 0
New Here ,
Apr 15, 2020

Copy link to clipboard

Copied

Let me add:

UTF-16 LE works with respect to the Umlaute without mojibake (fantastic term), but Indesign then fails to recognize the line breaks, i.e. the import itself fails

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 15, 2020 0
Most Valuable Participant ,
Apr 15, 2020

Copy link to clipboard

Copied

Thanks I missed the screen shot. The character sequence Ã¼ is exactly what happens (mojibake) when you choose Windows or ISO-1 encoding for information that is actually UTF-8.  What app are you saving the file from? What do you get when you choose Unicode for this file (because ASCII must give exactly the results you posted)?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 15, 2020 0
New Here ,
Apr 15, 2020

Copy link to clipboard

Copied

Yeah, I know, that's why I am at a loss why changing the encoding does not solve the problem.

What app are you saving the file from?

Sublime text and save it manually as UTF-8. Should be solid. Sidenote: A weird feature is that although the file is UTF-8 and Sublime Text shows the file as UTF-8, if I open the file in Excel the Umlaute are wrong as well. (WTF!?)

What do you get when you choose Unicode for this file (because ASCII must give exactly the results you posted)?

I can't in Sublime (sorry).

 

 

 

 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 15, 2020 0
Most Valuable Participant ,
Apr 15, 2020

Copy link to clipboard

Copied

I mean, choosing Unicode in InDesign. Anyway, please try saving from Excel or similar. It will add something called a "BOM" which often sorts this sort of thing out.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 15, 2020 1
New Here ,
Apr 15, 2020

Copy link to clipboard

Copied

Sight ... been there 😞 Sublime allows me to save as UTF-8 bom or UTF-16 bom, I tried all of them, nothing works. I have the suspicion that the original file is very corrupt, because that's all that remains. I'll try to re-format it by hand. Phew.


Thank you so much for the quick attempt and responses!

 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 15, 2020 0
Adobe Community Professional ,
Apr 15, 2020

Copy link to clipboard

Copied

Hi Jana,

the issue lies in the source text file. Did you recieve it from a client?

Can you make a sample available? A download link from Dropbox or a similar service?

I'd like to have a look with Notepad++ . Think, we can correct this and convert it to a working UTF-8 file.

 

Regards,
Uwe Laubender

( ACP )

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 15, 2020 0
Participant ,
Apr 19, 2020

Copy link to clipboard

Copied

Hi Jana,

this is a common issue with client side delivered files.

Instead of trying to reformat, resave or handing guidelines to our clients we switched to MyDataMerge, an App for macOS (only) with a drag and drop interface (Indesign layout + Excel database or CSV) If you're on that plattform i'd definetely give it a try.

Developed by a german team (as far as i know - it has a german translation), imports excels, exports linebreaks, handles almost every encoding. It basically does all the (annoying) things for you in the background and you can focus on work.

 

edit: here's the link https://mydatamerge.com/

 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 19, 2020 0