Skip to main content
Participant
July 11, 2018
Question

InDesign data merge record number limitation?

  • July 11, 2018
  • 4 replies
  • 2986 views

One of my clients are a printing company. They routinely get orders to print things like coupons or cards with individual numbers on each. So once the graphics gets approved they place it in InDesign, then merge it with a csv that contains desired numbers.

Problem is they often get orders for quantities like 20000-50000 and InDesign seems to be incapable of handling that many as it will lock up when trying to merge. So they have to break it up into pieces of about 5000 numbers each, which adds extra work as someone has to keep up with individual pieces and also then multiple merged files cause confusion on the printing side.

So they have asked me to look what may be causing the issue. I have used 2 separate PCs for testing:

PC1: i7 8GB RAM, Windows 8.1x64

PC2: i5 16GB RAM, Windows 7x64

InDesign CC 13.1 x64 on both

Both PCs behave approximately the same:

i7 CPU averages 15-20% utilization by indesign.exe. i5 CPU averages 25-30%. In both cases system still very much responsive

On both systems memory usage of indesign.exe generally floats between 2 and 4 GB. Can reduce itself to as low as about 500M, may spike closer to 4 but still never quite runs out of memory.

Disk usage on the i7 system averaged 50-60% with occasional spikes to 100%. On the i5 machine averaged 20-30%, sometimes would spike to 80-90%, but wouldn't stay there long.

Both PCs would progress rather well through "merging records" and when they would get towards the end the progress would just stop forever.

I was running task manager and resource monitor on a separate screen on both PCs. It would seem as if indesign.exe was not totally locked as the numbers for its CPU, memory and disk usage would still constantly change. However, I left both PCs running the task overnight and in the morning it would still be running.

So it doesn't seem like I am running out of system resources or have insufficient hardware. If csv is broken up and separate pieces are merged all works well, so I know my data has no issues. Thus it seems to me that InDesign is simply incapable of merging more than a certain number of records unless someone can advice me what I need to do.

Thank you

Vassili

This topic has been closed for replies.

4 replies

VassiliTNAuthor
Participant
July 11, 2018

Thank you everyone for your answers, that's kind of what I thought.

Colin, I will give your script a try, not sure if I will be able to make heads or tails from it as I am not a programmer nor an InDesign expert, but doesn't hurt to try May be a week or two before I get an opportunity to do so

Colin Flashman
Community Expert
Community Expert
July 11, 2018

Hi there Vassili

Speaking from my own experiences, my employer routinely does data merges of 45,000 every quarter, but the data is not calling in images and has about 20 fields. On a mac running High Sierra it takes about 90 minutes to merge. The item in question is a two page direct mail item with an address and other data.

We also run a larger data merge that does call images in (barcodes), each record is 3pp long and is 75,000 records, and that takes MUCH longer, but on a dedicated machine with nothing else running and all power savers turned off, it will do it within an afternoon.

Obviously, the larger the base file is and the more graphics/data that have to be processed, the longer it will take, so one place to start is to optimise the merge to its bare minimum. For example, if pictures are higher resolution than they need to be, reduce the resolution to something more appropriate.

May I suggest a javascript that I have written that will break the merge into smaller chunks WITHOUT the client having to stop and start the merge themselves? It can be found on my website https://colecandoo.com/scripts/  (it is called Data Merge to Batches) as well as some other data merge scripts that you and your client may find handy.

If the answer wasn't in my post, perhaps it might be on my blog at colecandoo!
Steve Werner
Community Expert
Community Expert
July 11, 2018

David Blatner did a video tutorial on Lynda.com on Data Base and Database Publishing which includes some recommendations of products beyond Data Merge. You can a free trial on Lynda.com to view it:

InDesign: Data Merge and Database Publishing

Steve Werner
Community Expert
Community Expert
July 11, 2018

The Data Merge feature was never designed for projects as large as that.

You need to pay for a third-party database or variable data application instead.

You can ask for recommendations for those on the forum who use such products.