Data Merge slow. To much data

[Sergi15A7]

Hi, I'm having an issue with data merge.

I need to merge the data of 200.000 barcodes. They are 14 KB .bmp each. I'm doing it in 6 columns of 10.000 barcodes each. In the end, it means 1648 InDesign pages. After 4 hours I canceled it half done.

I am trying now 412 pages but it is taking an entire hour already.

Is there a way to make it faster by software? I know my MAC is a bit old, 3,06 GHz Intel Core 2 duo, 4 GB RAM. from 2009, but I want to know if is there I am missing from the program.

Thank you very much

[Colin Flashman]

Can you please go into a little more detail? I'm trying to picture this in my mind's eye but the maths isn't working out.

There are 200000 barcodes to be processed using data merge, each of them is a link to a .bmp file that is 14kb each.

What is meant when you say "doing it in 6 columns of 10,000 barcodes each meaning 1648 pages" - do you mean the database has six fields of 10,000 records? If so, that would only yield 60,000 barcodes. I must be misunderstanding.

Are you able to clarify some of the information in the post?

What kind of barcode is being merged - is it one that could be replaced using a font such as code 3 of 9, or is it something that has a check digit that has been generated elsewhere?

[Sergi15A7]

You're right, sorry. I'm doing 60.000 barcodes just to give it a try and it's being so troublesome that I don't dare to do the 200.000 that I need in one go. I've done the code 128 barcodes with BarcodeStudio because it gives me control over how are they done and it generates the .csv needed.

[patMorita · Accepted solution]

The reason why it gets so slow is simple. InDesign builds the merged files in the RAM and once it reaches its limits it starts using hard disk space. The more pages you have in the merged document (and the more data) the slower it gets.

The solution is simple. Break it down into chunks. Try to merge ranges. You will be faster then doing all at once.

If you are on macOS 10.13 or newer you may try MyDataMerge which does that for you automatically: it chuncks the file and (if you want to) merges all generated files back into an indesign book. We use it regularly for large merges.

[Sergi15A7]

Thank you very much. That was very useful.

[tonyrush]

Pat, can I ask a clarifying question on this?  

We do a lot of postal mail campaigns in which we're sending an 8-page letter that has the order form pre-filled with the customer's information.  We do the mail merge entirely inside InDesign.  

What's the advantage of using MyDataMerge instead of just using InDesign's native functionality?  InDesign allows you to specify ranges of the CSV so you can manage the final size of the outputted document.  

I'd love some faster speed and if MyDataMerge can do that, that's great.  I guess I'm not clear on what MyDataMerge is doing that I can't already do.  

Thanks in advance for your help.

[patMorita]

Hi Tony 

Of course, even if I don't know the specifications of your project. So let me tell you what it helped me within our mail merge projects.

1.  Handling cases: InDesign throws an error if a field that is used in the merge is empty. With MyDataMerge you may specify simple or complex rules: if this is empty then output that. If that is also empty then e.g. delete the whole line or output an empty value or ... or ... or... think you'll get it
2.  You can merge to single documents with names dynamically defined from your database. Or merge in chunks of X records per file (in one run). You can directly merge PDF files or interactive PDFs (!) keeping links etc. You can merge to single files but also create an InDesign book file for easy post processing e.g. printing.
3.  Apply different styles based on rules
4.  Save everything including all rules in a project file. So the next time you need it just open the project update your datasource or drag n drop a new one and straight go for export
5. You may also append an automator workflow to post process each merged file
6. You may write custom scripts if all the rules and conditions don't match your needs.
7. Pre sort/filter the records automatically without changing your datasource
8. Dynamically merge QR Codes from your database. Even colored ones.
9. There's much more. But it right know feels like an advertising post lol so I will stop here 😄

Side note: it will never merge faster than InDesign itself cause it uses InDesign. But in many cases the gain of speed is in the process all around the merge. Except the specific case discussed here where InDesigns merge lacks in Design cause it creates too large documents for memory in one run merges.