Multiple record data merge into paragraph styles-applies the wrong style

[LinaD36]

Hi, I've been working on this project for sometime and everytime I manage to get one part of the workflow to work another seems to break. My agency publishes catalogs in multiple formats: large-print, audio, braille, and HTML. I've been trying to redesing our work process so that the catalogs will be laid out from merged data out of comma-separated file. The data merges have worked fairly well in Word, but InDesign is a challenge. I'm merging multiple records on a page, like a mailing label. The paragraphs need to be formatted and I'm trying to apply a paragraph style to them. After much work, I think I've finally got the data merge to work correctly, but the wrong paragraph styles are applied. I'm going to apply a new master page to the data once the data is merge that uses the paragraph styles for text variable running headers and I need to build a table of contents based on the paragraph styles so I need this to work. Attached are some screenshots.

Master page set up for data merge with paragraph styles:

Here is the merged document with the wrong paragraph styles applied:

I have only a few weeks to get this process ironed out to keep to our rigorous production schedule. If anyone can help I would really appreciate it.

thanks,

Lina

[Laubender]

@Peter – That's great.

But let's wait what the OP, Lina, will have to say…

Sometimes I smell complications 😉

Uwe

[LinaD36]

I've been watching your progress and am very impressed. I can't wait 'til Monday to try this out. I can't thank all of you enough for all the time and effort you have put into this. It's so marvelous that I can get help from all around the world. However because I'm not technical, I'll need more images and step-by-step directions on exactly what to do. Not really sure how to run the script or where in the workflow I should put it. Should I just run the script or search after I've run the data merge with the empty fields in, and do nothing else before or after? Sorry to have to task you with this, but I'm not a programmer and am not sure what to do with code.

best,

Lina

[Laubender]

@Lina – this requires a 3-step process:

1. Do the merge (and remember: DO NOT remove blank lines during the merge! Very good advice by Peter!)

2. Do a TEXT search/replace
Search for (without the quotes): "<FEFF>"

Replace with nothing (= will delete all the special characters, that will prevent the next step!)

3. Do a GREP search/replace
Search for (without the quotes): "^\s+"

Replace with nothing

The best thing is: you do not need to use my script. Just work with TEXT and GREP search/replace. Important: don't do step 3 before step 2!

Step 2 and 3 combined are represented by my script. So you could do both changes in one step. Just double click the script in the Scripts Panel.

To make the code lines into a ExtenScript file, just copy/paste the published code in answer #58 to a text editor file (e.g. TextEdit.app on a Mac OS), and further on with TextEdit, hit cmd+t to make the file a text only file without formatting and save it with a name like "Remove_AllMergeFieldMarkers_PlusWhitespaceAtParaBegin.jsx" (maybe come up with a better, shorter name 😉 ) to:

Applications/Adobe InDesign CS6/Scripts/Scripts Panel

If you are not using CS6 use the Scripts Panel in your version of InDesign in the applications folder or directory.

Optional: In the Scripts Panel folder you could make a special sub folder like "ScriptsForDataMerge" or something and save it to that one.

NOT OPTIONAL: give the text file the suffix ".jsx"

AND ANOTHER ADVICE: Save the three or two steps in different InDesign files, so you could go back easily, if something goes wrong.

Uwe

[LinaD36]

Thanks  so much Uwe. Sounds fabulous. Will give you more feedback on Monday.

Lina

[Peter Spier]

Lina,

I can't speak for Mike or Uwe, but this has actually been one of the more interesting problems of the past year for me, and I'm actually excited to have the solution. I do a number of data merge projects myself and this will be a real boon.

[LinaD36]

Peter,

My fingers are crossed, I'm overwhelmed by the depth and amount of help the three of you have given me. I really hope it will work. I work for a small, nonprofit agency that provides large-print, audio, and braille publications for blind and visually readers in the United States and American citizens living abroad. This year has been especially hard as Congress shut us down for 17 days and we already had to make due with not being paid for 4 days of work this year, with more budget cuts and days without pay on the horizon next year. But my publication goes out to more than half a million people in the U.S. and around the world, and I hope that you know that these are the people you are really helping. Readers write me all the time to let me know how our program is a lifeline for them or their loved ones who have lost their sight. Thank you for all that you have done. I'll definitely let you know on Monday.

Lina

[MW Design]

This has been a productive thread for me as well.

I went back through several merges and realized why this problem was never apparent. In every case, there was never blank data in any of the fields I would key off of for headers/footers, etc. But the issue was there when the merges removed blank lines. As noted in this thread, the apperance was correct, but not the paragraph style. Had I needed to use those styles for variables, I *might* have faced this issue in the past unless I simply cursed ID and manually fixed it mumbling all the while.

So thank you, Lina, for bringing this to our attention. And thank you again, Uwe, for the script.

Mike

[Laubender]

@Lina, Peter and Mike – let me point out, that the script has a special scope:

the whole document.

There might be situations where you would like to apply the GREP search/replace part only on the merged text frames and not on the regular ones. Or to a limited amount of text inside the merged text frames.

In cases like that, you need to use the regular TEXT search/replace action first (scope: the whole document) and restrict your GREP search/replace to some paragraphs one could identify with their applied paragraph styles. Or by other means. It will depend on the special case…

When I go back to the first screenshot Lina provided, I can see both:

1. Let's call them "Merge Field Markers" (still don't know the offical term for it)

Ok. I looked it up: it's Unicode for the character ZERO WIDTH NO-BREAK SPACE

And GREP in InDesign obviously does not count that as "whitespace" ( the GREP for that is: \s )

Here a take out from Lina's second screenshot:

In my search about <FEFF> I found the following entry in Wikipedia:

http://en.wikipedia.org/wiki/Zero-width_no-break_space

Let me quote it here (emphasis is mine):

Zero-width no-break space (ZWNBSP) is a deprecated use of the Unicode character at code point U+FEFF. Character U+FEFF is intended for use as a Byte Order Mark at the start of a file. However, if encountered elsewhere it should, according to Unicode, be treated as a "zero-width non-breaking space". The deliberate use of U+FEFF for this purpose is now deprecated, with U+2060 word joiner (HTML: &#8288;) strongly preferred.

2. There are also other special characters present in Lina's screenshot:

XML tags I would say:

But there is no indication in the master text frame for the data merge for XML tags (here another take out from Lina's first screenshot):

So, this question goes to Lina:

How and at what stage of your workflow did you add those XML tags?

After the merge?

If so, first run the search/replace actions and only after that do the tagging.

And another thing, he, he!
We already had that whole issue in the forum.

In April 2011. I missed it then:

Michael Gianino

Data Merge Invisibles

http://forums.adobe.com/message/3614180#3614180

Answer #6 by John Hawkinson (with a script snippet to eliminate all <FEFF> characters):

http://forums.adobe.com/message/3615031#3615031

I came across that when looking up "Zero-width no-break space" as a forum search 😉

Uwe

[Laubender]

Just a follow up to the script by John Hawkinson:

I think it's better to use TEXT search/replace for that purpose, because it's scope is wider. I foresee cases like tables inside tables with "<FEFF>" characters, that cannot be reached other than that.

And, surprisingly, it seems to be faster doing the TEXT search/replace…
(I ran John's script from the ESTK (ExtendScript Toolkit), all CS5.5; maybe it's faster doing so in InDesign)

And, when tested with the material Peter provided, at one point on it missed some occasions of "<FEFF>" and also from that point on it will start a wrongly applied paragraph style: "Genre" where "Title" should be…

Uwe

Message was edited by: Laubender

[Peter Spier]

Hah!

And here I thought I had a good memory for the arcane stuff.

Thanks, yet again.

[Peter Spier]

Another suggestion that Lina (or others) might find useful. In line with what Uwe said above, I saved the text query Find <FEFF> and Change to nothing as a custom named Find/Change Query so I can select it from the dorpdown list without having to remember the code, or that it won't work in GREP.

Cheers.

[LinaD36]

Uwe,

Honestly, I couldn't tell you when the XML tags were added. I never added them. I'm thinking this must be a default process that's been added in InDesign CS 6.0. My goals were simple: merge data from a comma-separated file into paragraph styles in InDesign. When Mike and Peter pointed out the XML tags, I wasn't sure when they came in or how they got there. I just tagged my template with paragraph styles and ran the data merge utility that's available in 6.0. I have no need to tag the data for XML.  Our database exports to XML directly.  Sorry not sure I can really help you.

Lina

[Laubender]

@Lina – seems we have another case for unwanted XML tags after a data merge:

jetman88

Crazy colored brackets in InDesign data merge document

http://forums.adobe.com/thread/1347560?tstart=0

I provided a scripting solution to get rid of the tags:

http://forums.adobe.com/message/5888732#5888732

That leaves the question: are these cases an coincident?
I don't think so…

Of course I could add the untagging to my script in answer #58. Just in case the tags will disrupt the GREP search/replace part.

Uwe

[LinaD36]

Thanks Uwe! So much to try out tomorrow. I'll have to give it a spin.

Lina

[LinaD36]

Uwe,

Am I doing something wrong? I've run both searches. I've manage to delete the empty subject code paragraph style fields, but not the empty genre paragraph style fields. I even tried running both searches twice, to see if this would rid me of the empty genre paragraph codes. Here are pictures of what I did:

1. Data merge with blank lines included:

2. Merged document with empty genre paragraph style fields and empty subject code paragraph style fields:

3. First text search:

4. Text Search completed:

4. Now the GREP Search:

6. GREP completed:

7. Notice that the Subject Code data merge markers are gone, but the Genre data merge markers remain:

Still hoping that it's user error on my part. Hope to hear from you.

Lina

[Peter Spier]

Uwe will probablyhave more to say, but it looks to me like the problem is still the XML tags.

Could you show us the Story Editor view of the template frame before the merge?

[LinaD36]

Template master in Story Editor view:

[Peter Spier]

OK, that's really strange. The template shows ordinary merge tags, the merged file shows XML tags. I think we need to figure out where the change is happening.

Can you take us step by step from the template to the merged document?

[LinaD36]

Sorry, will do this first thing tomorrow, have to go pick up my daughter

from day care. Today was a catch up day so, I got to the software project

late. But will try to take pictures of everything and walk you through it

first thing in the morning.

Lina

[Peter Spier]

No problem. I think finding where the XML is coming from is going to be the key.

[MW Design]

After the merge when the XML tags show up, would you take a screen shot of the Tag panel?

I am wondering if at this point there is document corruption that just won't go away. Perhaps pre-merge, save a copy out as IDML, then open it, run the merge and see what happens.

It might just be expeditious to take a look at the ID document if this continues.

[Laubender]

@lina, Mike, Peter – in the other thread I provided a script to get rid of the XML tags after the merge.
THEN the two search/replace actions should be no problem.

I gave a link to my answer and the script in answer #72 here:

jetman88

Crazy colored brackets in InDesign data merge document

http://forums.adobe.com/thread/1347560?tstart=0

I provided a scripting solution to get rid of the tags:

http://forums.adobe.com/message/5888732#5888732

jetman88 did not answer yet, if the script does work for him.

So here is the script again:

@lina: copy the code to a pure text file, name it (without quotes): "EN_UntagAllXMLElements_ACTIVE_DOCUMENT.jsx" and put it into your "Scripts Panel" folder under: Applications/Adobe InDesign CSx/Scripts/Scripts Panel.

Once the merge is done, open the "Scripts Panel" (the panel, not the folder!) in InDesign and run the script by a double click. Then save the document with a different name. Then procede with 1. TEXT search/replace, 2. GREP search/replace. Or use my script in answer #58 for these two steps.

//EN_UntagAllXMLElements_ACTIVE_DOCUMENT.jsx
//Authors:
//Hans Haesler
//Gregor Fellenz
//Comments by Uwe Laubender

/**
* @@@BUILDINFO@@@ EN_UntagAllXMLElements_ACTIVE_DOCUMENT.jsx !Version! Sun Dec 01 2013 09:38:21 GMT+0100
*/


//See: http://www.hilfdirselbst.ch/gforum/gforum.cgi?post=506568#506568
//Data for testing: http://www.hilfdirselbst.ch/gforum/gforum.cgi?do=post_attachment;postatt_id=6654;

//Green4ever
//"Untag Frame" menu action is invalid [CS5 Win XP]


//The active document:
var d=app.documents[0];

//Hans Haesler:
//Untag all XML-Tags with exception of the root element:
d.xmlElements[0].xmlElements.everyItem().untag();

//TO GET RID OF THE ROOT ELEMENT:
//Workaround by Gregor Fellenz:

//Add a new rectangle to the document:
var tempRect = d.rectangles.add();
//Tag the rectangle with the "Root" element:
tempRect.markup(d.xmlElements[0]);

//AND HERE IS TRICK:
//Remove the rectangle:
tempRect.remove();

Please provide some screenshots for the whole process:

1. The master document (yes, I know, we already seen it, but do it anyway for sake of completion)

2. The merged document saved with a unique name

3. The merged document after running the script saved to a unique name

If the XML tags are gone, you could procede.
Otherwise give me a personal message so you could send me the InDesign file or better an exported IDML file for further testing.

I have no idea where the XML tags are coming from. So it would be good to send all the files from 1-3 as exported IDML.

Regards,
Uwe Laubender
( Adobe Community Professional )

EDIT: Reposted code in code tags of the forum editor

Could be, that the links in this post will not work anymore, because the old InDesign forum was moved to a new forum software by the end of 2019.

[Laubender]

@Lina – just to get a more clear description of your workflow:

1. What is your operation system?

2. What is your exact InDesign version?

3. Are some 3rd party plugins (not Adobe) installed?
(Maybe one of the 3rd party plugins provide additional XML features and intercept the merge adding the XML tags?)

Uwe

Message was edited by: Laubender

[LinaD36]

Uwe,

I am under strict security regulations at work and am restricted from

saving your data to the Applications Data folder. But I did see yesterday

in the InDesign Script panel that perhaps I could save the script to the

AppData folder in the Users folder and it might show up in InDesign. Will

try that today when I get in. I'm running Windows 7 on a PC with InDesign

CS 6. I will try to provide screenshots of the whole process today and I

thought also that may be I will start over with a new document and do

things from the beginning.

Lina

[Peter Spier]

Yes, scripts can be stored in your user profile scripts panel folder, and they will be availble only to you. That should be C:\Users\<your username>\AppData\Roaming\Adobe\InDesign\Version 8.0\<your language>\Scripts\Scripts Panel by default.

Starting fresh would be a really good idea (you can copy the curent template frame into a new document to save time).

It would be worth going to Help > Manage Extensions... to open the extensions manager and scan the list for anything not Adobe. I also have a feeling there may be some third-party plugin working here, perhaps one of the catalog plugins.