How Comprehensive is InDesign's Support for the Unicode Standard?

[Roger Breton]

I watched an InDesignSecrets video on YouTube called "Zero-width space" :

InDesign How-To: Use a Zero Width Space (Video Tutorial)

I admit I was impressed by what the use of this character can do in terms of typography.

I looked around for more information. 
I quickly landed on Wifipedia which was helpful.

But then I landed on the "Unicode Standard"? Which I didn't know existed.

I confess my ignorance in typgraphy.

I have yet to experiment with a Zero-width space in an InDesign document -- I will.

Since the use of this "character" is suported (see above video) makes me wonder :

A) Will it have some kind of representation with Show Invisible Characcters?
B) And it makes me wonder what are the other kinds of space characters that are out there?
C) Whether they're all supported in InDesign?
Hence the subject of my broader inquiry, how "complete" is InDesign support for Unicode?

Super interesting question 🙂
Thank you all for helping in advance.

<Title renamed by MOD>

[Roger Breton]

Made some progress with regards to Zero-width space support. 

The following is a screen capture of two text frames containing the same A to Z 26 letters alphabet.

This is with Text > Show Hidden Characters ON.

Do you see the large blue diagonal lines between each characters?

I wonder if this "blue character" is specific for this Unicode character?
Or is it "generic"?

[Robert at ID-Tasker]

@Roger Breton

InDesign is an Unicode app.

Here you can find a list of all "hidden" symbols:

https://helpx.adobe.com/uk/indesign/using/editing-text.html

And here you can find some PDFs with more special characters:

https://community.adobe.com/t5/indesign-discussions/is-there-an-up-to-date-guide-to-hidden-character...

[Brad @ Roaring Mouse]

The zero width Character (i.e. Unicode 200B) is merely a Discretionary Line Break. This IS in the Insert Break Character menu despite what he says in his video, like so:

It's also represented in a Text search as "^k" or in GREP search as "~k", so your search query only needs to look like this: e.g. GREP search:

 

The blue line is InDesign's way to show you that particular hidden character where they are (same as a paragraph return or tab mark, etc).

Unicode has been around for many many years now. Most fonts now are Unicode fonts, allowing thousands of characters (e.g. a TrueType font could have 65,535), but you won't find U200B as a character you can "see" in a font file; it's more of an instruction.

[Roger Breton]

I did a search on "Discretionary Line Break" and found this in InDesign Help:

Indicates where a line of text should break if the line needs to break. 
I fail to grasp what that does to the composition, Brad? Here is my example:

There are Conditional Line breaks between each characters. I just can't put down in words how the presence of this character in the text is being processed by the composition engine : is it "keeping the lines from breaking"? Is that what it does?

Is it keeping the lines from hyphenatig?

[leo.r]

I just can't put down in words how the presence of this character in the text is being processed by the composition engine : is it "keeping the lines from breaking"?

By @Roger Breton

it allows your line to break when needed. try to remove them - how will your line look without discretionary line breaks?

[Brad @ Roaring Mouse]

It's because, in a way, each letter is now considered a word. There is no hyphenation involved. The same way as if you put a REAL space between each letter – only the space you are using has no width.

By the way, in that guy's video, he wonders why the two methods he used didn't break the same way. Although he didn't show his Character panel, I believe it was because he used Optical letter spacing. Optical spacing uses an algorithm to adjust space between each character which is much different than the built in font metrics of the font. I think that algorithm probably has a routine to deal with regular spaces but not the Zero Width Spaces, hence each will be spaced differently, and hence wrap differently.

In the end, either method works, so pick your poison!

[Joel Cherney]

I think that algorithm probably has a routine to deal with regular spaces but not the Zero Width Spaces, hence each will be spaced differently, and hence wrap differently.

In the end, either method works, so pick your poison!

That's an excellent explanation, Brad. I haver seen, over the years, that Optical Kerning doesn't always play nice with zero-width spaces. This may be because Optical Kerning is clearly optimized for Latin script, and when applied to languages like Arabic or Chinese, it causes errors and undesirable spacing issues. Languages that require the zero-width space also seem to not work well with Optical Kerning. 

Here's how I learned this: many years ago, I had a client who had a major dispute with a minor spacing issue in Lao. Lao is, like Thai and Burmese, one of those languages where usually there aren't spaces between words. In fact, it's typical for translators working in these languages to insert zero-width spaces in the text where they think it ought to break. 

Long story short,  my client didn't like that their designers didn't understand how these matters worked - so their designers '"fixed" my "errors" in the Lao, and went to print without a review, which cost them a fair bit in a wasted print run that was close to illegible. So in the name of client education, I made a file that tested the rendering of a wide variety of types of spaces in a variety of conditions, in order to demonstrate how to use these tools correctly. The test file was full of edge cases - e.g. where a given line of text was right on the verge of wrapping. 

In those experiments, I did findthat the Adobe Paragraph Composer handles ZWSP slightly differently than normal U+0020 spaces that have been reduced to 0% width. In some conditions, they render identically, but if you (for example) reduce the tracking, the 0% space happily "tracks in" while the ZWSP does not. There are a lot of moving parts, here; the World-Ready Composer behaves differntly than the Adobe Paragraph composer, when you cram a line full of text separated by ZWSP. Glyph Scaling and kerning values can also affect exactly where two almost-identical lines will wrap. 

[Roger Breton]

Thank you, Brad, I was "slowly" getting it but your reply is very helpful :

> in a way, each letter is now considered a word. There is no hyphenation involved. The same way as if you put a > REAL space between each letter – only the space you are using has no width.

In other words, when inserting Discretionary line breaks or Zero-Width spaces, the Composition engine no longer sees one 26 letter word but 26 individual words. In the extreme, if the width of the text frame we very narrow and only accommodate one character, the Composition engine would be "glad" to have the 26 letters sitting on separate lines. Thank you so much for taking the time to explain.

I'm just curious? I tried to research the "regular" use of the Zero-With space in "regular life". I had difficulty finding words that would no break "properly", words that would warrant the insertion of ZWS character between some sylables? To observe how they break when I reduce the width of the text frame. Seems all I could find was the case of URLs? In order to avoid the insertion of a hyphen after each "\"?

[Brad @ Roaring Mouse]

@Joel Cherney . I did a test where I reproduced the same sample the youtube person created using the same font (Balboa Extra Condensed). If I used tracking of Metrics, there was an ever so slight difference, but using Optical netted wildly different results in the wrapping. However, using "0" tracking netted identical results.

But it depends on the font, so it might depend on what kerning pairs are built into it and t what characters they apply. I did a second test with a similar font (Helvetica Ultra Compressed), and there were no differences at all between Metric and "0", but Optical again netted wildly different results.

So I guess it all depends!!

[Roger Breton]

There is no end to InDesign's capabilities.
Thank you for taking the time to conduct this experiment to document it here 🙂