Multiple record data merge into paragraph styles-applies the wrong style

[LinaD36]

Hi, I've been working on this project for sometime and everytime I manage to get one part of the workflow to work another seems to break. My agency publishes catalogs in multiple formats: large-print, audio, braille, and HTML. I've been trying to redesing our work process so that the catalogs will be laid out from merged data out of comma-separated file. The data merges have worked fairly well in Word, but InDesign is a challenge. I'm merging multiple records on a page, like a mailing label. The paragraphs need to be formatted and I'm trying to apply a paragraph style to them. After much work, I think I've finally got the data merge to work correctly, but the wrong paragraph styles are applied. I'm going to apply a new master page to the data once the data is merge that uses the paragraph styles for text variable running headers and I need to build a table of contents based on the paragraph styles so I need this to work. Attached are some screenshots.

Master page set up for data merge with paragraph styles:

Here is the merged document with the wrong paragraph styles applied:

I have only a few weeks to get this process ironed out to keep to our rigorous production schedule. If anyone can help I would really appreciate it.

thanks,

Lina

[LinaD36]

Sorry my whole family has the flu. I'll have to deal with this when I'm not sick. Happy Holidays!

Lina

[Colin Flashman]

I have been following this post from a distance and so far the OP has been getting great advice from Peter Spier, MWDesign and Uwe Laubender, so there's been no need for me to add my "buck o'five". I notice that scripts have been mentioned to thread many frames into one story, but I think this one was missed, and it is quite a good script for the purpose of threading unlinked text frames together:

http://ajarproductions.com/blog/2008/11/28/merge-textframes-extension-for-adobe-indesign/

But what is of more interest to me is why the OP is going from an XML based solution to create the catalogue to a Data Merge solution. I say this because it appears from the OP's first post that the original design had XML tags, suggesting that once upon a time the art WAS an XML workflow.

I am a regular user of Data Merge to produce VDP campaigns for my employer, and have a blog that amongst other things, looks at interesting ways to use Data Merge in InDesign to do things that users may not be aware of. But in this case, I feel that Data Merge is a less efficient tool for the job. Yes, it CAN do it, but SHOULD it do it? It is akin to hammering in a nail with a screwdriver - it can do it, but its not efficient and its not what the screwdriver was designed to do.

XML would hold a few advantages in this task, such as:

• The ability to preview a single record that is hundreds of entries into the database without having to create a finished document first;
• The ability to store carriage returns in a field of data, something that data merge can't do unless the carriage return is swapped out with another delimiter such as a pipe symbol (|) and then post-processed using a find/change to replace the pipe symbol with carriage returns;
• XML data can be repurposed for other media, or that the XML is a result of repurposing from another media already (meaning no double-handling of data)
• Would not require the amount of post-processing that the revised solution requires, especially if the text at the centre of the data merge changes. Any post processing could be handled by the DTD or XLST upon import.

There are not many resources (in Print) for using XML with InDesign but the two go-to books that I have are "A Designer's Guide to Adobe InDesign and XML" by James Maivald; and "InDesign CS5 Automation Using XML & Javascript" by Grant Gamble.

That said, XML is also a steep learning curve... really steep. The rewards are worth the effort if it will be used a lot.

Also, I am not a user of any paid third-party catalogue plugins but would imagine that there would be an array of solutions on the market already that would make this task much easier. XML and Data Merge are the ways that InDesign "off of the shelf" allows to automate repeating data en masse into an InDesign project, but are not necessarily the only ways.

I certainly do not want to undermine the work and effort of the fellow posters who have gone above and beyond the call to assist the OP. Ultimately it is what the OP is more comfortable using and the thread is certainly of use to other readers with similar projects.

[LinaD36]

Hi CDflash,

Thank you for your interesting insights. I will look into the XML solution further once I have the current production workflow nailed down. Our catalog goes out to about half a million blind and visually impaired subscribers in four formats: Large-print, braille, audio recording, and HTML. We had previously been exporting the data as XML and importing it into Corel Ventura. Then about 5 years ago Corel stopped supporting Ventura. Almost 3 years ago we hired a contractor to help us migrate the workflow from Ventura to InDesign. After 3 years of hard work from our staff, the contractor failed to deliver anything. So I fell back on to using a function that I knew would work well in Word, because 3 of the formats require the data be delivered in Microsoft Word. The large-print version is the only version that requires that we use an enterprise publishing solution like InDesign, so I've tried to implement the data merge in InDesign. I'm trying to improve on many inefficiencies with the old workflow as well, which had us export the data, layout it out in the four formats, and edit it four times. I'm trying to frontload the editing while the data is still in the comma-separated format. Five years ago a number of our staff, including me, took some introductory courses in XML to be more conversant in it, since we are all writers and editors. I find it fairly straight forward. Javascript, however, seems very hard for me to learn. Any way, I do appreciate your advice and will try to find some time to look into in the near future.

best,

Lina

[LinaD36]

Okay so I finally got a chance to try the "save as IDML" fix, and I'm sad to say it does work for me. I still get overset text:

I diligently did all these steps: 1) added numbered column to CSV file:

Included Number in Data Merge:

Performed data merge with blank lines:

Generated Merge document

Here is the merged document:

Ran script to remove XML tags:

Here is the document without XML tags:

Ran text search:

Ran GREP search:

Saved as IDML:

Opened IDML:

Ran stitch together script:

Really curious to know what you did differently.

Lina

[MW Design]

In the first screen shot, it looks mucked up. But the later ones look correct.

Once the Stitch script has been run like in the last screen shot, you can go to the pages panel, delete all the pages. Delete all but the first text frame.

Drag out the only text frame from margin to margin, side to side, top to bottom. Make it two columns. Create a second page. Click on the out port at the bottom of the frame on page one. Hold down shift, go to the top left of the second page and click (while hold shift down) at the intersection of the top and left margins. ID should add pages and frames until the end of the document.

[LinaD36]

Hallelujah! You found the answer. More in a bit. But I think you figured it out! The text frames have been merged twice, one on top of the other. When I delete all the text frames on top, the ones on bottom seem to be correctly linked. I will go through and double check in a minute, I think this is a tied to the problem of InDesign generating an extra text frame where there was not one in the master page. I will check back with you once I've checked all the pages.

Lina

[LinaD36]

So delete the extra text frame on the top does reveal a 566 text frames that are correctly linked in order on the bottom! Hurray!

At the end it got kind of strange and I had to reflow the text and apply a the "none" master page to it, but it will still work for me.:

[LinaD36]

As I go through the text now, there are still some places were the text frames seem to be double placed, but it does seem to work in general.

[LinaD36]

I realized I had introduced an error into the whole process and saw that the book annotations were one off, but when I repeated the whole process again, the annotations are now slightly out of order:

I think this is the same problem Peter was having before.

[LinaD36]

I figured out that where the book annotations are out of order the text box threading is in the wrong order, so if you rethread those boxes that are out of order the rest of the document is alright.

[Laubender]

@Lina – the idea of having a unique number per text frame is a very good one. But I think, that number should be on its own paragraph at the beginning of every text frame.

A script that is stitching together the single frames could then fetch that number easily and doing the stitching number by number instead of trying to do it in geometrical order or in order of the (invisible) ID numbers. I did not write that script yet.

What I do not understand: why did you populate a master page with a text frame in the basic document for your data merge?

I did all my testing around a one page document as the master and a single text frame on that page to do the merge. No other elements. Just that one text frame in the whole document. Keep it simple as possible.

Uwe

Message was edited by: Laubender

[Peter Spier]

LinaD36 wrote:

I figured out that where the book annotations are out of order the text box threading is in the wrong order, so if you rethread those boxes that are out of order the rest of the document is alright.

That does sound like the problem I was seeing, and it should be easy to spot with the numbers where the order goes wonky. Cut/paste to rearrange a set should work fine.

[MW Design]

I finally had it go completely wonky. I don't know whether to cheer or cry in my beer. Once it went wrong, it goes wrong every time. Not even IDML of the merge document opened, saved then the merge occur would produce a correct order.

I cobbled together three scripts others wrote which run one after the other. It produces a correctly threaded file. It only works in the scenario above where I mentioned a full, left margin to right margin frame. First one to run is the Ajar script cdflash mentions. I have used it before for other types of threading. Then Uwe's delete script for the markers, lastly Uwe's threading script fires off.

I think the reason it works is that at some point, ID's internal frame IDs get mucked up and Uwe's stitching script depends on ID getting it correct. I see no other posibility.

@Uwe, if given the scenario of a full-width (margin to margin frame) merge frame, how much work is involved to make the stitching proceed top to bottom regardless of internal frame ID? For what amounts to a book produced from a merge, one is going to delete all the pages and all but a single frame anyway and then reflow the first frame. I think it would be just the ticket.

Mike

[LinaD36]

Hi Uwe,

I'm okay with the number paragraph being it's own style and paragraph, I'm deleting them out at the end any way.

I set up the data merge as part of a master page, I think because of some advice I got here on this forum. But I can try it as a stand alone page and see if it works better or causes problems. Only problem is I've run out of time. I've got to work on getting the actual publication out. So not sure how much time I'll be able to devote to testing for the next week or so. But I will keep checking back on your progress, I'm trying to put two separate catalogs out using this workflow, so I'm motivated to get it all hammered out in an efficient way. Thanks so much for your help.

Lina

[Laubender]

@Lina & Mike – I think the best we can do is preparing the numbering of the units in a fixed position. Best would be the first paragraph containing the number and nothing more.

I will redo my stitching script that it is able to read out the numbers and do the stitching according to the numbers.

Other methods like stitching after ID numbering of the text frames or in geometrical order seem not be reliable (tests by Mike and Peter do suggest that).

I think I can have a new stitching script for extacly that purpose by Tuesday next week.

Uwe

[MW Design]

Uwe--I may have miscommunicated.

The Ajar script processing in logical order, top to bottom (if set that way in the dialog box). It however does not thread from page to page. But it does perfectly succeed in I think what you are terming, geometrical order.

So the combined script I have that first executes the top-bottom logical order runs, then yours runs and threads page to page. It works perfectly when combined.

[LinaD36]

Sounds great Uwe, look forward to it.

Lina

[Laubender]

@Mike – yes. But the Ajar script would fail, if the order of merged frames is not logical (one of Peter's examples here suggested that it could happen this way).

So it's better to rely on data like the consecutive numbers  the basic  CSV or TXT file is providing.

Uwe

[MW Design]

The Ajar script succeeds. From reading down through it and trying to understand it, I think the Ajar script is actually replacing the current text frames (and progressing logically down the page) and the result must be that they are in ID's internal order correctly. Then when your's fires off, it is able to connect the pages (I think yours is skipping already threaded frames, dunno).

In any case, the combination does work. I have used it over a dozen times last night and today, going through page by page making sure the numbers are in order. I can post on dropbox if you would like and send a PM. That way it doesn't clutter the thread and it can be reviewed for accuracy.

Mike

[Laubender]

@Mike – ok. Send me a PM with a link. InDesign file(s) together with their IDML representation, the source CSV or TXT (whatever you are using) zipped together.

It's always good having some material for tests. 🙂

Uwe

[Colin Flashman]

@MW

The Ajar Script v1.2.0 by justin pudney DOES allow threading from page to page - it's a choice from the dropdown field at the top of the dialog box. Make sure that you have the latest version from the ajar productions website. (it was an amendment a year or so ago i think)

[MW Design]

Hi CD--I have the latest version.

It has a choice for doing all pages of a document, but it does not thread from page to page:

Running Uwe's script afterwards does connect the frames across the pages:

Because Justin's script replaces the frames in logical order (here from top down, all pages), the script from Uwe then connects "the dots" so to speak.

Take care, Mike

[Laubender]

Lina wrote:

Sounds great Uwe, look forward to it.

@Lina – a new script that would stitch the text frames by numbers (single unique integer number present in the first paragraph of every text frame after the data merge) is in a testing phase. You'll hear from me at Monday.

Have a nice weekend!

Uwe

[Laubender]

@Lina – I sent you a private message (PM).

Uwe

[LinaD36]

So the great news is that the new script from Uwe works! I can post screenshots of it working, if people want proof. I am so relieved and thankful to everyone who has helped so much with this process. Anyone who wants the stitch together script should contact Uwe through a private message.

I am deeply indebted to all of you.

All the best,

Lina