Known Participant

Answered

Pasted Unicode character shows up as ?

Forum|Forum|5 years ago
November 16, 2020
3 replies
1026 views

Using FrameMaker 2019. When I try to paste a Unicode symbol, such as ∧ (U+2227) using Character Map, it shows up in FM as a ? character. I've tried a number of different fonts in FM with no luck. The same font in Word or a web browser looks correct. I don't think the font selection is the problem. Is this a bug? I tried Paste Special and each option available (Rich Text, Text, and Unicode Text).

How can I insert such a character and have it display correctly? Is there a shortcut to type this in directly?

I tried using the built-in Insert > Character > Character Palette but there must be a bug: it just disappears after a second or two.

One thing that seems to work, but is not practical, is to paste into MS Word, copy it in Word, switch to FM and use Paste Special > Rich Text Format.

This topic has been closed for replies.

Correct answer Bob_Niland

If U+2227 [∧] renders as [?], it's because your font simply doesn't populate that codepoint, and FM doesn't yet do fallback. The font would have been invoked by Para tag, Char tag, variable or override; and the override could come in with the paste. U+FFFF and below are Unicode BMP, and FM supports the BMP range.

If U+10000 {5 digit hex} or higher is used, it's going to render as [?] every time (in FM8-FM2019, and FM2020 to date), even if the font does populate the glyph, because FM doesn't yet support Unicode SMP. Curiously, the actual Unicode character is preserved in the FM binary source, so might work someday.

I've had to deal with this quite a bit on a current project, and use Variables to handle it. For BMP codepoints not in my Body font, the var def invokes a character format calling for a font that does. For SMP codepoints, the var has to invoke a legacy/overlay font that populates the desired glyph at some low order (usually ISO 8859-1) hex value.

For example:

For [∧]
Name: U+2227 LOGICAL AND
Definition: <SegoeUISymbol>\u2227

For [🌢]
Name: U+1F322 BLACK DROPLET
Definition: <Wingdings>\x53

(\x53 is "S")
(<Strings> are Character Formats, declaring only the needed Font Family)

I use the U+ notation to float the Names down to a self-sorting block at the bottom of the var catalog, and use the exact Unicode formal codepoint name for consistency, de-duplication, and ease of later revision as fonts & FM itself evolve.

After creating such a variable, FM collapses the \u and \x notation to the actual characters (∧ and S). This can be annoying where the character is whitespace or otherwise visually ambiguous.

LinSims

Community Expert

You're not the only one for whom the Character Palette/Hex Input won't stay open. It's definitely a bug, and I reported it as such for FM 2019, athough it was also an issue in FM 2017. Vote for the bug; Adobe prioritizes fixes, and the more people who vote for a bug, the more likely they are to take a serious look at it.

W

Winfried Reng

Community Expert

Hi Whlotus,

Which font do you use? I checked this, and I found this character only in Cambria Math. However, only when I use BabelMap. In the Windows Character Map I could not find this character. And even in BabelMap, which has the reputation to be much more reliable than the Windows Character Map, I could not select this character.

When you have a font which supports this character, you can use this also in FrameMaker. E.g. create a variable with this unicode character and inser this variable in your text.

https://community.adobe.com/t5/framemaker/enter-unicode-u-21cc-code-point-in-dialogs/m-p/4372050?page=1#M28725

Best regards

Winfried

W

whlotusAuthor

Known Participant

I was using the font Source Sans Pro. The character shows up in Windows Character Map, but I don't think the font contains the code point glyph. If I load it into fontdrop.info, the glyph doesn't seem to exist in that font. So Character Map must be applying a font fall back for some fonts, but I'm not sure. However, the code point does exist in SegoeUISymbol.

Regards

Bob_Niland

Community Expert

I've switched to using Bablestone's BabelMap app. Both FM CP and MS CM have issues.

Bob_Niland

Correct answer

Community Expert

If U+2227 [∧] renders as [?], it's because your font simply doesn't populate that codepoint, and FM doesn't yet do fallback. The font would have been invoked by Para tag, Char tag, variable or override; and the override could come in with the paste. U+FFFF and below are Unicode BMP, and FM supports the BMP range.

If U+10000 {5 digit hex} or higher is used, it's going to render as [?] every time (in FM8-FM2019, and FM2020 to date), even if the font does populate the glyph, because FM doesn't yet support Unicode SMP. Curiously, the actual Unicode character is preserved in the FM binary source, so might work someday.

I've had to deal with this quite a bit on a current project, and use Variables to handle it. For BMP codepoints not in my Body font, the var def invokes a character format calling for a font that does. For SMP codepoints, the var has to invoke a legacy/overlay font that populates the desired glyph at some low order (usually ISO 8859-1) hex value.

For example:

For [∧]
Name: U+2227 LOGICAL AND
Definition: <SegoeUISymbol>\u2227

For [🌢]
Name: U+1F322 BLACK DROPLET
Definition: <Wingdings>\x53

(\x53 is "S")
(<Strings> are Character Formats, declaring only the needed Font Family)

I use the U+ notation to float the Names down to a self-sorting block at the bottom of the var catalog, and use the exact Unicode formal codepoint name for consistency, de-duplication, and ease of later revision as fonts & FM itself evolve.

After creating such a variable, FM collapses the \u and \x notation to the actual characters (∧ and S). This can be annoying where the character is whitespace or otherwise visually ambiguous.

W

whlotusAuthor

Known Participant

Bob, thanks so much for your elegant solution. Yes, indeed the font I was using does not have that code point. I didn't think the font mattered because it seemed to work in other programs. Also the Windows text rendering APIs automatically use a fall back font to render code points correctly even if a font does not contain it. I guess it makes some sense that FM would strictly adhere to the font specification and not allow arbitrary system-dependent fall backs that authors would not expect.

It's ironic that the ∧ character displays correctly in the Variables pod GUI, but doesn't when you insert it into the document (unless you add a font format to the definition).

Bob_Niland

Community Expert

whlotus: I didn't think the font mattered because it seemed to work in other programs.

Many apps nowadays do fallback (recent FM chatter here). Whether or not to do it may be both a theological and an API matter. FM doesn't, yet, but I won't be surprised if it happens as part of a text rendering engine update (needed for SMP, OTF variable fonts, Unicode combining and likely other things). Other Adobe apps do fallback, so in addition to the Windows APIs, there's probably some library of corporate code that can be leveraged.

re: I guess it makes some sense that FM would strictly adhere to the font specification and not allow arbitrary system-dependent fall backs that authors would not expect.

Switching FB on and off would seem to be a requirement, as would being able to control the cascade.

re: It's ironic that the ∧ character displays correctly in the Variables pod GUI, but doesn't when you insert it into the document (unless you add a font format to the definition).

FM seems, internally, to know about all of Unicode. The rendering engine doesn't (nor does the Char Palette). We await developments.

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded