How To Add Language Tags For Amazon KDP eBook?

[om.ranade]

Hi, I'm writing a ebook in Marathi language. I'm using Nirmala UI font.

While uploading my ebook on amazon I understood that I need to set my book content's Language tags as Marathi (aka. Select language as Marathi in adobe indesign properties window) But after doing that, a few letters in my language gets disconnected like "ल्ल" >> "ल् ल". I've noticed that by selecting Language tag as "Hindi" the text does not disconnect letters. Altough the tag needs to be Marathi. 

How do I overcome this issue? 

[James Gifford—NitroPress]

I assume you're exporting to EPUB — fixed-layout or reflowable? The latter is greatly preferable.

Language gets a little complicated in ID/EPUB/KDP, and I don't know what your overall familiarity level is, so this will be long.

The only places language is addressed by InDesign is in the overall version/UI selection, which may set a few defaults in actual document settings (not sure), and in the language tag assigned to each Paragraph and Character Style:

The first thing to do, if you haven't already, is review every Paragraph and Character Style to make sure the correct language is selected. (This is assuming you have a meticulously-organized ID document, with a style assigned to every element. Spot formatting, always bad practice, compounds problems in EPUB export.) You may find one or more styles have a wrong language setting, and that will be your problem.

You can verify that tags are being used by opening the exported (reflowable) EPUB and looking at the main XHTML content file. These settings cause a language tag to be inserted into each content file of an EPUB export. Adobe seems to have changed this up a little, so there are two possibilities depending on which version of ID you're using:

• Standard practice, and what was done a few versions back and is now done again, is to insert one language tag for the entire document, in the heading information:

  

• If there is only one language setting in the doc, no further ones will be used. If there are individual settings, such as a French paragraph in an otherwise English doc, there will be one general setting and then a lang tag in each different element (paragraph or span).
• For a time in the last year, EPUB export was including a lang statement in every text element, which some validators and services did not like. If you're running a slightly older version, you might see that instead.

To look at this content, open the EPUB export using a ZIP archive tool. Under the OEPBS folder, you'll find one or more XHTML files, relatively large proportional to content. Extract one and open it in any browser, then switch to code mode. You should see heading lines like the above, with a lang statement for "mr-IN". Otherwise, you'll find a lang statement in every text paragraph.

If everything seems correct from your end — you are selecting the right language and it's specified in every style and you've done a quick check to make sure it's being exported — then the issue is with KDP. I haven't done any non-English projects with them, but if you have a language setting and it's correctly set to Marathi, and you still get these word/alphabet issues... it's on them. You'll have to work it through with their (really lousy, all-but-nonresponsive) tech support.

I don't find myself surprised that non-English, or at least non-Western languages like the Indic ones have bugs in the conversion details. But good luck, and with better luck it's just a simple mistake on your part, which is at least easy to correct. 🙂

Check back if none of this fixes your problem or there are other details, or if KDP's response doesn't seem correct.

[om.ranade]

I'm sorry if my query was not clear but heres a little clarification,

I'm facing the problem when I select the language to Marathi (from Charecter/Paragraph Style) - few letters are not rendering properly. Although the language tags are correct as "mr-IN" and KDP is not giving me a issue. Its just that when I select the language as Marathi - few letters are not rendering as they do when the language is English-US.
Side Node : I do not face this problem when I select the language as Hindi. I tied to play with OpenType settings but I don't exactly know how they work. (PS: Could this issue even be solved?)

When language Is Set To English-US (This is the correct form)

When Language Is Set To Marathi (This is where the letters are separated)

[James Gifford—NitroPress]

I think I had a clear picture of the problem, but you did not give any technical details or indication of your skill level (with EPUB export etc.) in your first post, so I tried to cover the major points. If you're up to speed with things like HTML/CSS in the export files, we can focus a little more.

The key question is where this problem is occurring in the conversion chain. If it's correct in your InDesign document, and printing or export to PDF, then it's not likely to be anything at the ID level.

You don't say whether the EPUB export is to fixed pages or reflowable. The next step would be to open the EPUB itself, preferably in a "neutral" reader like Calibre, Thorium or EPUBreader, and see if and when the letter variations happen there. If you get the letter separation with Marathi, but not with others, it may be a general implementation problem with the language settings.

And then it's up to how KDP handles the various export versions. I am not completely clear on where the examples you give are from (which level of this, that is), but if Kindle is showing these variations and will not show 'Marathi' correctly when it is handled properly at all intermediate levels... then the problem is very much KDP's to either fix, or work with you to solve.

Do as much variation and testing at each of these three levels — four, if you include looking at the XHTML export code — to isolate what conditions (what language settings, etc.) create and solve the problem at each level.

[James Gifford—NitroPress]

Another thought: is this letter behavior controlled, to any extent by 'ligature' settings? If you turn ligatures on and off for a style, does it affect how these letters connect and group? Or is that a more fundamental aspect of the language (Marathi vs. Hindi)?

[Joel Cherney]

(PS: Could this issue even be solved?)

Well, the best solution for this would be updates to Nirmala UI, which is the source of the issue. A solution for you, personally, might be to figure out how to insert a zero-width joiner there. I don't know any Marathi at all, and have no idea what challenges there are in typing Marathi, but in similar cases in Nepali when a given Devanagari font fails to render a Nepali-specific glyph, I've found that the fastest answers are "pick a different font" and "insert a zero-width joiner when you're typing the word."

Your screenshot is in InDesign; are you experiencing the same glyph shaping issue when the text is marked as Marathi in your ebook after export? That would be another indicator that it's the font that is causing the issue. 

[Joel Cherney]

Here's another thread where I discuss cases where you can rely on language-setting glyph shaping, and cases where you have to enter a Unicode joiner, or non-joiner, as appropriate. 

[James Gifford—NitroPress]

I am not at all clear where or why (== under what settings) the problem is occurring. I would hope that Adobe understands language needs well enough to correctly define the language settings; it seems that if Marathi had this grievous flaw it would have long ago been reported and corrected. (Emphasis on 'seems'...) And if the font has all the correct letterforms, separate and conjoined, I'm not sure where it would play in unless it was corrupted.

It does look as if the fault is right in InDesign, which would thus have nothing to do with EPUB or KDP renditions. But AFAIK thinks like ligatures are 'phantom' elements, imposed only by the app itself on a realtime basis — that is, a text export would not show ligature glyphs. I wonder if something similar is happening here.

Only orderly testing of settings and exports and checks at each level can really isolate this.

[om.ranade]

It seems that the issue is with adobe indesign itself.

Indesign is not rendering the text correctly when the language is set to Marathi, It is displaying the text correctly while the language is set to English-US / Hindi. I don't understand what would be the reason for this?

[Joel Cherney]

Well, there are lots of moving parts here, which I'd be perfectly happy to post about (at great length!) if you're interested, but the basic message is that Nirmala UI doesn't support Marathi, not really. Tiro Typeworks, the foundry responsible for Nirmala, wasn't asked for Marathi support when they made Nirmala for Microsoft. Page 3 of this PDF tells you exactly what languages were targeted for support in Nirmala, and Marathi is not in the list. 

It's somewhat analagous to being asked to typeset something in French in a font that doesn't have ê (e-with-circumflex) in it. You can work around it, if you need to; you can kern accents into place manually, but the font just doesn't support the language in question. The solution I'm suggesting here for Marathi typesetting in Nirmala is actually rather less hacky than kerning the ^ into place over the e, which I'd honestly never do if I had a target format that was anything other than print. 

The OpenType specification has lots of features that can be used by type designers. I haven't done any type design in years and don't currently have any tools installed that would allow me to figure out exactly what is going on under the hood in Nirmala, but it clearly has language-aware substitution of alternates, or language-aware glyph shaping, which is demonstrated by the OP's observation that the glyph rendered correctly when marked as Hindi. 

But the Unicode spec fortunately has a set of character shaping glyphs that can allow the OP to access glyph shapes that aren't available with a Marathi language setting. 

[James Gifford—NitroPress]

So I ass/u/me that there are fonts that are fully Marathi-compliant... and that switching to one of these would solve the ID issue, if not necessarily anything downstream (given that EPUB and Kindle fonts are subject to swapping and substitution)?

[om.ranade]

Well I changed the language to Marathi & exported in EPUB,

The letters when viewed in KindlePreviewer appear as they should (connected "मूल्य"). The only issue is within adobe indesign.

Side note : I did try to play with OpenType settings and Stylistic Sets settings, but I don't exactly know how they work.

[James Gifford—NitroPress]

That would indicate to me that apps handle this issue much like ligatures — on a realtime basis using some set of rules. The final check would be to look at the XHTML code and see if individual letters are exported, or the connected 'ligatures.'

Since your end goal is correct, that would seem to solve the problem for the short term. But if InDesign is not correctly handling the language details, that needs to be corrected. Or, as Joel has already noted, it could be any of several font issues — poor 'ligature' mapping, corruption, missing cojoined character glyphs, etc. The solution there would be to try several Marathi fonts and see if they all behave the same way.

So is the issue resolved, for this project, or are there are other points that need some kind of fix?

[om.ranade]

I tried switching the fonts (Adobe devanagri) and it does work properly inside indesign. I was still wondering if there would be a way to make Nirmala UI font render text properly when language is set to Marathi, as it does while language is English-US? 

[James Gifford—NitroPress]

You can experiment, but I think Joel has clearly outlined that the font is not 'programmed' to work seamlessly in Marathi, and so you may run into this problem at every variation and iteration of work using it. Short of getting the foundry to encode all the correct cues into the font set so that glyphs conform to Marathi use, it will always be something of a gamble for anything except printed (and thus unchanging) publications.

Even in English, there are many fonts that have incomplete character sets or poorly optimized kerning and ligature data. There's simply nothing that can be done with these except accept any faults (using workarounds and hacks as needed) or use another font altogether.

[Joel Cherney]

Every piece of software will have their own ways to handle glyph rendering. Most Windows apps use the Unicode glyph-rendering systems provided by the OS. I don't have any idea what's going on under the hood on a Kindle, or in a Kindle-viewer installed on a Windows machine, but it's not related at all to how InDesign is going to handle the font. InDesign's own unique font rendering system is notoriously picky.

So e.g. your browser, if it tries to render text marked with a language-and-locale pair doesn't appear in your font's OT method, will fail over to another language in its attempt to render the font. So, if the Marathi language code doesn't appear in Nirmala's guts, your browser (your OS, your Kindle) will just render it with, I'm guessing, an en_US lanaguage and locale, which we know works with this particular glyph shaping challenge. (This is common among complex-script fonts that one gets from Microsoft.) InDesign doesn't do this; it throws up its hands saying "But I don't know what to do with Marathi!" 

So, if you wanted this behavior changed, I suppose you could head over to indesign.uservoice.com with a feature request. But if you tried to post it as a bug report, you might your bug report closed, because it's not a bug. InDesign is actually respecting the intention of the type designer! 

[James Gifford—NitroPress]

I don't have any idea what's going on under the hood on a Kindle

No one does. That's the first problem. 😄

[om.ranade]

Yes I understand, I'll work by using a different font or try to get around this issue somehow, for this project.

Thank you Joel & James for your help, I appriciate it!

[Joel Cherney]

Here is how to get around the issue - I included a link to a previous thread where I've already solved a very similar issue in Nepali. What you'd need to do if you want to use Nirmala UI in Marathi in InDesign is to: 

1) retype that word one time, but this time use a zero-width joiner right after the halant

   (zero-width joiner can be found in Type -> Insert Special Character -> Other -> Joiner

2) copy that retyped word to your clipboard

3) open the Find/Change dialog

4) paste your retyped word with joiner into the "Change to:" field

5) copy and paste an example of your unchanged मूल्य word into the "Find what:" field

6) click "Change all"

[James Gifford—NitroPress]

In something book length, it might take an extraordinary amount of effort to find all such broken ligatures. I have no idea how many combinations there might be, and that could be some long, close proofreading. Or if there are 'negative cases' — situations where A and E are not meant to be Æ, etc.

I'd also wonder how they will be handled in EPUB export. Joiners/nonjoiners, in my experience, are 'trustworthy' only for print output. Any specific experience with exporting them to HMTL or EPUB?

[Joel Cherney]

I'd also wonder how they will be handled in EPUB export. Joiners/nonjoiners, in my experience, are 'trustworthy' only for print output. Any specific experience with exporting them to HMTL or EPUB?

Been a while since I posted the phrase "Wow, just wow." I understand that lots of stuff doesn't make it into EPUB export but pulling Unicode control characters of of text? That... simply didn't occur to me. I mean, I'm going to test it tonight, for sure, but that would make it completely impossible to export EPUB in, er, at least Malayalam, Sinhala, and Tamil, just off the top of my head. Actually, when I think about it, non-joiners are necessary in Persian, so if non-joiners were being stripped out of EPUB export from InDesign, then one of the three Farsi proofers who went over my work with a fine-toothed comb would have certainly raised a flag at one point or another. Still, it's worth testing out. 

I would also never suggest that a long complex-script document rely on a visual proof to ensure proper glyph composition; that's the kind of thing that GREP can solve quite easily. There are only so many characters in Devanagari that can possibly follow a halant, so I imagine that a complete test would take maybe four to six minutes to put together. 

[James Gifford—NitroPress]

I suspect the various export processes manage these alphabetic issues, over and above how browsers and readers do — at least, any such tools that aren't completely Western- if not English-centric. I don't really know one way or the other but seem to recall at least some publication flows having problems with such embedded characters. (After all, ligatures are almost universallly a phantom object created by the display device, not actual glyphs in the text strings or dependent on joiners.)

And knowing next to zip about the Indic languages at a technical level, maybe you're right that there's a limited number of combinations that could be GREPped into compliance.

But I can't but help think the simple, obvious, bulletproof solution here is to use a fully compatible font. 🙂

[Joel Cherney]

(After all, ligatures are almost universallly a phantom object created by the display device, not actual glyphs in the text strings or dependent on joiners.)

I think I'm actually frothing at the mouth over here. But never mind that, we are in complete agreement on the only issue that actually matters:

But I can't but help think the simple, obvious, bulletproof solution here is to use a fully compatible font. 🙂