Skip to main content
S
Sergi15A7
Participant
September 25, 2020
Answered

Data Merge slow. To much data

  • September 25, 2020
  • 2 replies
  • 3840 views

Hi, I'm having an issue with data merge.

I need to merge the data of 200.000 barcodes. They are 14 KB .bmp each. I'm doing it in 6 columns of 10.000 barcodes each. In the end, it means 1648 InDesign pages. After 4 hours I canceled it half done.

I am trying now 412 pages but it is taking an entire hour already.

Is there a way to make it faster by software? I know my MAC is a bit old, 3,06 GHz Intel Core 2 duo, 4 GB RAM. from 2009, but I want to know if is there I am missing from the program.

 

Thank you very much

This topic has been closed for replies.
Correct answer patMorita

The reason why it gets so slow is simple. InDesign builds the merged files in the RAM and once it reaches its limits it starts using hard disk space. The more pages you have in the merged document (and the more data) the slower it gets.

 

The solution is simple. Break it down into chunks. Try to merge ranges. You will be faster then doing all at once.

 

If you are on macOS 10.13 or newer you may try MyDataMerge which does that for you automatically: it chuncks the file and (if you want to) merges all generated files back into an indesign book. We use it regularly for large merges.

2 replies

P
patMoritaCorrect answer
Inspiring
September 26, 2020

The reason why it gets so slow is simple. InDesign builds the merged files in the RAM and once it reaches its limits it starts using hard disk space. The more pages you have in the merged document (and the more data) the slower it gets.

 

The solution is simple. Break it down into chunks. Try to merge ranges. You will be faster then doing all at once.

 

If you are on macOS 10.13 or newer you may try MyDataMerge which does that for you automatically: it chuncks the file and (if you want to) merges all generated files back into an indesign book. We use it regularly for large merges.

S
Sergi15A7Author
Participant
September 28, 2020

Thank you very much. That was very useful.

Colin Flashman
Community Expert
Colin FlashmanCommunity Expert
Community Expert
September 26, 2020

Can you please go into a little more detail? I'm trying to picture this in my mind's eye but the maths isn't working out.

There are 200000 barcodes to be processed using data merge, each of them is a link to a .bmp file that is 14kb each.

What is meant when you say "doing it in 6 columns of 10,000 barcodes each meaning 1648 pages" - do you mean the database has six fields of 10,000 records? If so, that would only yield 60,000 barcodes. I must be misunderstanding.

Are you able to clarify some of the information in the post?

What kind of barcode is being merged - is it one that could be replaced using a font such as code 3 of 9, or is it something that has a check digit that has been generated elsewhere?

If the answer wasn't in my post, perhaps it might be on my blog at colecandoo!
S
Sergi15A7Author
Participant
September 28, 2020

You're right, sorry. I'm doing 60.000 barcodes just to give it a try and it's being so troublesome that I don't dare to do the 200.000 that I need in one go. I've done the code 128 barcodes with BarcodeStudio because it gives me control over how are they done and it generates the .csv needed.

Terms & ConditionsAccessibility statement
Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices