Pasted Unicode character shows up as ?

[whlotus]

Using FrameMaker 2019. When I try to paste a Unicode symbol, such as ∧ (U+2227) using Character Map, it shows up in FM as a ? character. I've tried a number of different fonts in FM with no luck. The same font in Word or a web browser looks correct. I don't think the font selection is the problem. Is this a bug? I tried Paste Special and each option available (Rich Text, Text, and Unicode Text).

How can I insert such a character and have it display correctly? Is there a shortcut to type this in directly?

I tried using the built-in Insert > Character > Character Palette but there must be a bug: it just disappears after a second or two.

One thing that seems to work, but is not practical, is to paste into MS Word, copy it in Word, switch to FM and use Paste Special > Rich Text Format. 

[Bob_Niland · Accepted solution]

If U+2227 [∧] renders as [?], it's because your font simply doesn't populate that codepoint, and FM doesn't yet do fallback. The font would have been invoked by Para tag, Char tag, variable or override; and the override could come in with the paste. U+FFFF and below are Unicode BMP, and FM supports the BMP range.

If U+10000 {5 digit hex} or higher is used, it's going to render as [?] every time (in FM8-FM2019, and FM2020 to date), even if the font does populate the glyph, because FM doesn't yet support Unicode SMP. Curiously, the actual Unicode character is preserved in the FM binary source, so might work someday.

I've had to deal with this quite a bit on a current project, and use Variables to handle it. For BMP codepoints not in my Body font, the var def invokes a character format calling for a font that does. For SMP codepoints, the var has to invoke a legacy/overlay font that populates the desired glyph at some low order (usually ISO 8859-1) hex value.

For example:

For [∧]
Name: U+2227 LOGICAL AND
Definition: <SegoeUISymbol>\u2227

For [🌢]
Name: U+1F322 BLACK DROPLET
Definition: <Wingdings>\x53

(\x53 is "S")
(<Strings> are Character Formats, declaring only the needed Font Family)

I use the U+ notation to float the Names down to a self-sorting block at the bottom of the var catalog, and use the exact Unicode formal codepoint name for consistency, de-duplication, and ease of later revision as fonts & FM itself evolve.

After creating such a variable, FM collapses the \u and \x notation to the actual characters (∧ and S). This can be annoying where the character is whitespace or otherwise visually ambiguous.

[whlotus]

Bob, thanks so much for your elegant solution. Yes, indeed the font I was using does not have that code point. I didn't think the font mattered because it seemed to work in other programs. Also the Windows text rendering APIs automatically use a fall back font to render code points correctly even if a font does not contain it. I guess it makes some sense that FM would strictly adhere to the font specification and not allow arbitrary system-dependent fall backs that authors would not expect. 

It's ironic that the ∧ character displays correctly in the Variables pod GUI, but doesn't when you insert it into the document (unless you add a font format to the definition). 

[Bob_Niland]

whlotus: I didn't think the font mattered because it seemed to work in other programs.

Many apps nowadays do fallback (recent FM chatter here). Whether or not to do it may be both a theological and an API matter. FM doesn't, yet, but I won't be surprised if it happens as part of a text rendering engine update (needed for SMP, OTF variable fonts, Unicode combining and likely other things). Other Adobe apps do fallback, so in addition to the Windows APIs, there's probably some library of corporate code that can be leveraged.

re: I guess it makes some sense that FM would strictly adhere to the font specification and not allow arbitrary system-dependent fall backs that authors would not expect.

Switching FB on and off would seem to be a requirement, as would being able to control the cascade.

re: It's ironic that the ∧ character displays correctly in the Variables pod GUI, but doesn't when you insert it into the document (unless you add a font format to the definition). 

FM seems, internally, to know about all of Unicode. The rendering engine doesn't (nor does the Char Palette). We await developments.

[Bob_Niland]

re: I've had to deal with this quite a bit on a current project, and use Variables to handle it. For BMP codepoints not in my Body font, the var def invokes a character format calling for a font that does. For SMP codepoints, the var has to invoke a legacy/overlay font that populates the desired glyph at some low order (usually ISO 8859-1) hex value.

This SMP problem is still with us (and I have no information on whether or when it might get fixed)…

The variable hack works to PDF and print, but has the drawback that the actual codepoint for the mimicked SMP character is not going to be U+1F322, but \x53 (in the example earlier); something in the \x00…\xFF (U+0000…U+00FF) range. This will then be an entirely different glyph in the ASCII…Latin1 range if copied out of your PDF and pasted into Notepad, “S” for the example.

Where you want to include a Unicode SMP character in an FM document (FM8…FM17.0.8) and have it be preserved as a selectable Unicode character in the PDF or XML workflow, there appear to be two methods, and you may need to implement them both (via Conditional Text) if both output workflows are needed:

1.  To PDF
   Bring the SMP text in as an EPS text object with embedded {subset} font. I've been doing this, and it works. You will not be able to Index, Xref that as text within FM, but it's otherwise treated as any other EPS object. A later edition of the prior project switched to doing it this way. Being EPS, of course, it won't flow to HTML or XML with any elegancy at all.
2.  To HTML/XML
   Bring the SMP text in as an SVG text object. And here, the challenge is font declaration or embedding, and is not a workflow I have explored. There are apparently some 4 ways to do it. But because the target output is HTML or XML, font control is presumably under control for the project.

[Winfried Reng]

Hi Whlotus,

Which font do you use? I checked this, and I found this character only in Cambria Math. However, only when I use BabelMap. In the Windows Character Map I could not find this character. And even in BabelMap, which has the reputation to be much more reliable than the Windows Character Map, I could not select this character.

When you have a font which supports this character, you can use this also in FrameMaker. E.g. create a variable with this unicode character and inser this variable in your text.

https://community.adobe.com/t5/framemaker/enter-unicode-u-21cc-code-point-in-dialogs/m-p/4372050?pag...

Best regards

Winfried

[whlotus]

I was using the font Source Sans Pro. The character shows up in Windows Character Map, but I don't think the font contains the code point glyph. If I load it into fontdrop.info, the glyph doesn't seem to exist in that font. So Character Map must be applying a font fall back for some fonts, but I'm not sure. However, the code point does exist in SegoeUISymbol.

Regards

[Bob_Niland]

I've switched to using Bablestone's BabelMap app. Both FM CP and MS CM have issues.

[LinSims]

You're not the only one for whom the Character Palette/Hex Input won't stay open. It's definitely a bug, and I reported it as such for FM 2019, athough it was also an issue in FM 2017. Vote for the bug; Adobe prioritizes fixes, and the more people who vote for a bug, the more likely they are to take a serious look at it.