• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
0

How can you get text including Emojis into a data-merge document?

Explorer ,
Jul 30, 2019 Jul 30, 2019

Copy link to clipboard

Copied

Hi,

I'm using InDesign to create a memory book. Many people get one page each to fill with content. As past projects have shown, people are really keen about using Emojis, which I'd like to support. But I just can't get Emojis into InDesign via Data Merge, only when I copy&paste them manually.

My workflow so far:

  • In InDesign, I create a layout with sample texts, including Emojis.
  • I create a character style for Emojis, using the EmojiOne font.
  • In my paragraph style, I define a GREP Style to detect Emojis in any text and apply the character style created in the above step.

This works. But as for the data merge part:

  • I provide a sample page to help people get an idea how to fill it and what it will look like.
  • I provide a Google form. People fill it with text and upload images.
  • Google spreadsheet magic: I'm using a few formulas to prepare the texts for InDesign, and a script to determine the file names of the uploaded images.
  • I download the resulting spreadsheet (tried several formats here).
  • I use LibreOffice to convert the downloaded spreadsheet into .csv with UTF16-encoding.
  • In InDesign, I import the .csv using the Data Merge panel.

But the problem is: No matter how I format it and how I convert it, I won't get Emojis into the InDesign document. Only one of the two cases happens:

  • Emojis (as well as other non-English characters) turn into cryptic symbols (when interpreting as ASCII - that's the expected behavior).
  • Special characters survive, but Emojis are just stripped out of the text (when interpreting as Unicode, and formatting the source as UTF-16).
  • But if I open the UTF16-formatted .csv with a text editor or with LibreOffice, select the text including the Emojis, copy it and paste it into my InDesign document, it works.

Does anyone know a solution? What am I overlooking? Is there any other character encoding than UTF16 that InDesign supports?

Views

3.1K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct answer

Explorer , Jul 31, 2019 Jul 31, 2019

Ha, I have found a solution!

 

How to get line breaks and Emojis into InDesign via Data Merge

 

InDesign doesn't support line breaks or Emojis from data sources that you import via Data Merge. With this workaround, you can encode any special characters in your spreadsheet and decode them in InDesign again.

 

In your spreadsheet:

  • Prepare all your data as usual (including Emojis and line breaks)
  • Add a second sheet to your spreadsheet
  • Use formulas to fill it, and use ENCODEURL() to encode every special char
...

Votes

Translate

Translate
Explorer ,
Jul 31, 2019 Jul 31, 2019

Copy link to clipboard

Copied

Ha, I have found a solution!

 

How to get line breaks and Emojis into InDesign via Data Merge

 

InDesign doesn't support line breaks or Emojis from data sources that you import via Data Merge. With this workaround, you can encode any special characters in your spreadsheet and decode them in InDesign again.

 

In your spreadsheet:

  • Prepare all your data as usual (including Emojis and line breaks)
  • Add a second sheet to your spreadsheet
  • Use formulas to fill it, and use ENCODEURL() to encode every special character in URL notation (spaces will turn into %20)
  • Export the result as .csv and make sure to convert it to UTF-16 character encoding for InDesign (using a text editor or LibreOffice)

Capture.PNG

 

In InDesign:

  • Open a text editor, copy the below script into it, and save it as URL-decode.jsx

  • Move the script file to your scripts directory - on Windows this is: %AppData%\Adobe\InDesign\Version 14.0\en_GB\Scripts\Scripts Panel

    (you can type %AppData% into the Windows search box, press enter and navigate to the folder from there)

  • Use Data Merge as usual and import the .csv as data source
  • In the preview, you will see the URL-encoded characters
  • Click the "Create Merged Document" button
  • The resulting file will still contain all the URL-encoded characters
  • From the scripts panel (Window -> Utility -> Scripts), run the URL-decode.jsx script
  • Voilà, all your special characters are back!

 

URL-decode.jsx:

//DESCRIPTION:Decodes and replaces all URL-encoded strings in the active InDesign document.
//AUTHOR:Johannes Eichberger
//
//Example:
//    This%20is%20URL-encoded%20text!
//    This is URL-encoded text!
//
//How to use:
//    Open an InDesign document with URL-encoded texts.
//    Run the script.
//    It replaces every (!) text in every text frame.
//
//Limitations:
//    Works with text frames that contain text only. Any non-text
//    content, such as tables and linked images, will get lost.
//    If an encoded text exceeds its container (overset text),
//    only the visible part will be replaced. You can run the script
//    multiple times to solve this issue.
//
//When to use:
//    InDesign's Data Merge has only poor support of character encodings.
//    If you want to feed InDesign with Emojis, you can encode them in the
//    spreadsheet via =ENCODEURL() and decode them using this script
//    after running Data Merge.
//

#target indesign;

function main() {
    try {
        var myDocument = app.activeDocument;
        var myTextFrames = myDocument.textFrames.everyItem();
        var myContents = myTextFrames.contents;
        for (var i=0; i<myContents.length; i++) {
            var thisTextFrame = myDocument.textFrames.item(i);
            var myDecoded = decodeURI(myContents[i]);
            thisTextFrame.contents = myDecoded;
        }
        if (i==0) {
            throw "No texts found!";
        }
        alert("Texts have been replaced.");
    }
    catch(err) {
        alert(err);
    }
}
main();

 

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Jul 31, 2019 Jul 31, 2019

Copy link to clipboard

Copied

Hi joheiat ,

FWIW: be aware that your script could change the contents of text frames.

Before, text frame with one anchored object, the yellow rectangle:

BeforeRunningScript-ANCHORED-OBJECT-PRESENT.PNG

After running URL-decode.jsx:

AfterRunningScript-ANCHORED-OBJECT-REMOVED.PNG

Also a problem. Tables. You'll remove tables.

Before:

BeforeRunningScript-TABLE-PRESENT.PNG

After running URL-decode.jsx:

AfterRunningScript-TABLE-REMOVED.PNG

Regards,
Uwe

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jul 31, 2019 Jul 31, 2019

Copy link to clipboard

Copied

Hi Uwe / Laubender,

Yes, you're right. The script is on a very basic level and only works for text frames which contain nothing else but text. I might think about extending it to keep additional content placed in text frames too. Thanks for pointing me to this!

If anyone has a demand for this, please leave a reply to nudge me.

Also, feel free to extend the script.

Regards,

Johannes

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Oct 18, 2019 Oct 18, 2019

Copy link to clipboard

Copied

Hi Joheiat,

 

I'm actually in need of exactly this — a better process for data-merging with emojis. However, when I run your script to decode the url-encoded text, I get an error message. Is the script you posted here accurate and up-to-date? Thanks!

 

Steven

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jul 23, 2020 Jul 23, 2020

Copy link to clipboard

Copied

Hi Steven,

 

So sorry, I happened to see your reply only now. Are you still in the need of this? I tested my script again with the current version of InDesign with a minimalistic test file and it still works. Which error message do you / did you see?

 

Regards,

Johannes

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Feb 01, 2024 Feb 01, 2024

Copy link to clipboard

Copied

Hi Joheiat,

 

For me, the script operated as intended. However, I occasionally have content that is emoji-heavy. It then displays the error URL-Error: Bad or Invalid URL.

Is there a way to solve this? Thanks!

 

Niven

 

Screenshot (10).png

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Feb 05, 2024 Feb 05, 2024

Copy link to clipboard

Copied

Hi Niven,

 

I'm not sure what happened in your case, but as you state it happens only occasionally, it might be a cropped URL. Is there probably a percent sign in your encoded string which is NOT followed by two hex-numbers? Your error message ends with a % sign. Probably this is just cropped?

 

My script is very limited, it decodes only the visible text in a frame. Surely a script geek can find out how to decode the full text instead. What you could try as a work-around is: add another text frame outside the printing area, and connect the original frame to let overset text flow into that new frame. My assumption is that this would make overset text less likely to get cropped just after a % sign.

 

Regards,

Johannes

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Feb 05, 2024 Feb 05, 2024

Copy link to clipboard

Copied

Hi Joheiat,

 

My text box was cropped. It showed the same error after extending the text-frame. The script used to work just fine but sometimes content loaded with emojis results in this error.

Are you aware of any resources I could use to upgrade the script?

 

Niven

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Feb 05, 2024 Feb 05, 2024

Copy link to clipboard

Copied

Hi Niven,

 

Could you share a concrete example of a text that doesn't work, especially in its encoded form with the percent signs?

 

As for scripting resources, I would start with reading the documentation here, but depending on your scripting/programming experiences, this might be time-consuming. The issue in my script is probably that I'm accessing the text frame content via .contents which obviously only allows editing of the visible part. One would probably need to find methods or properties instead which allow access to the whole text.

 

Good Luck,

Johannes

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Feb 05, 2024 Feb 05, 2024

Copy link to clipboard

Copied

Hello Joheiat,

 

I've attached two files to illustrate the error that occurred in my case because I was unable to express it in words.
The URL-Decoding Script is applied to one file, but not the other.
The script operated fine up until Page 19, at which point the previously mentioned issue occurred.

Since I am aware that this is asking for too much, I sincerely appreciate your time.

 

Regards,
Niven

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Feb 05, 2024 Feb 05, 2024

Copy link to clipboard

Copied

LATEST

Hi Niven,

 

I see in your first file (Script not applied) on page 19 that there is a break within an encoded sequence. One special character encodes to %A4, but is split into two text frames, the first one ending with % and the second one starting with A4. The script reads and writes textFrames.item(i).content which is obviously only the visible content, therefore this is a limitation (besides many others, such as ignoring mixed styles within a text frame). The first frame only contains a % sign, not followed by two hex-digits, therefore it throws an error.

 

joheiat_0-1707152925154.png

 

To fix this, the script would need adaption to access the actual text of the frame instead. You're right, this is too much to demand here, however I'm leaving this here in case someone more experienced wants to update the script. For the time being, you can get into scripting yourself, or you can deal with work-arounds (such as disabling hyphenation), or you can go the tedious path and copy&paste all texts manually.

 

Let me also suggest to vote for these feature requests, which would make some of the datamerge issues easier to overcome:

 

Best regards,

Johannes

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Jul 23, 2020 Jul 23, 2020

Copy link to clipboard

Copied

Hi Johannes,

the script code in your correct answer was damaged last year when this thread was moved to the new InDesign forum.

That could be one cause of an error message. Look into the for loop. At first glance I see that the iterator [i] was removed on myContents. Best correct your answer and post the original code again.

 

Thanks,
Uwe Laubender

( ACP )

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jul 23, 2020 Jul 23, 2020

Copy link to clipboard

Copied

Hi Uwe,

 

Thanks for the hint. I've updated my answer.

 

Best,

Johannes

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
May 21, 2021 May 21, 2021

Copy link to clipboard

Copied

Hello Joheiat,

Since you allready solved the problem how to convert image url to file names: could you share the method/script you used to do so? 

Thanks!

Regards,

Pascal

 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
May 21, 2021 May 21, 2021

Copy link to clipboard

Copied

Hello Pascal,

Do you mean the Google spreadsheet script that I used to get the file names of images that were uploaded via Google forms? Let me take a look, if I find it I will post it here.

Johannes

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
May 21, 2021 May 21, 2021

Copy link to clipboard

Copied

I'm sorry, I've lost access to the file where I solved this problem. I used a Google Apps script which I had to run manually. If I recall correctly, it did pretty much the same as the one in this StackOverflow answer (check all URLs in one column, and output the file names in another column):

https://stackoverflow.com/a/51398404

Hope this helps you.

Johannes

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
May 22, 2021 May 22, 2021

Copy link to clipboard

Copied

No problem. Thanks for the reply! I will look into it.

Cheers

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines