Highlighted

Data Merge slow. To much data

New Here ,
Sep 25, 2020

Copy link to clipboard

Copied

Hi, I'm having an issue with data merge.

I need to merge the data of 200.000 barcodes. They are 14 KB .bmp each. I'm doing it in 6 columns of 10.000 barcodes each. In the end, it means 1648 InDesign pages. After 4 hours I canceled it half done.

I am trying now 412 pages but it is taking an entire hour already.

Is there a way to make it faster by software? I know my MAC is a bit old, 3,06 GHz Intel Core 2 duo, 4 GB RAM. from 2009, but I want to know if is there I am missing from the program.

 

Thank you very much

The reason why it gets so slow is simple. InDesign builds the merged files in the RAM and once it reaches its limits it starts using hard disk space. The more pages you have in the merged document (and the more data) the slower it gets.

 

The solution is simple. Break it down into chunks. Try to merge ranges. You will be faster then doing all at once.

 

If you are on macOS 10.13 or newer you may try MyDataMerge which does that for you automatically: it chuncks the file and (if you want to) merges all generated files back into an indesign book. We use it regularly for large merges.

TOPICS
Performance

Views

66

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

Data Merge slow. To much data

New Here ,
Sep 25, 2020

Copy link to clipboard

Copied

Hi, I'm having an issue with data merge.

I need to merge the data of 200.000 barcodes. They are 14 KB .bmp each. I'm doing it in 6 columns of 10.000 barcodes each. In the end, it means 1648 InDesign pages. After 4 hours I canceled it half done.

I am trying now 412 pages but it is taking an entire hour already.

Is there a way to make it faster by software? I know my MAC is a bit old, 3,06 GHz Intel Core 2 duo, 4 GB RAM. from 2009, but I want to know if is there I am missing from the program.

 

Thank you very much

The reason why it gets so slow is simple. InDesign builds the merged files in the RAM and once it reaches its limits it starts using hard disk space. The more pages you have in the merged document (and the more data) the slower it gets.

 

The solution is simple. Break it down into chunks. Try to merge ranges. You will be faster then doing all at once.

 

If you are on macOS 10.13 or newer you may try MyDataMerge which does that for you automatically: it chuncks the file and (if you want to) merges all generated files back into an indesign book. We use it regularly for large merges.

TOPICS
Performance

Views

67

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Sep 25, 2020 0
Adobe Community Professional ,
Sep 26, 2020

Copy link to clipboard

Copied

Can you please go into a little more detail? I'm trying to picture this in my mind's eye but the maths isn't working out.

There are 200000 barcodes to be processed using data merge, each of them is a link to a .bmp file that is 14kb each.

What is meant when you say "doing it in 6 columns of 10,000 barcodes each meaning 1648 pages" - do you mean the database has six fields of 10,000 records? If so, that would only yield 60,000 barcodes. I must be misunderstanding.

Are you able to clarify some of the information in the post?

What kind of barcode is being merged - is it one that could be replaced using a font such as code 3 of 9, or is it something that has a check digit that has been generated elsewhere?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Sep 26, 2020 0
New Here ,
Sep 27, 2020

Copy link to clipboard

Copied

You're right, sorry. I'm doing 60.000 barcodes just to give it a try and it's being so troublesome that I don't dare to do the 200.000 that I need in one go. I've done the code 128 barcodes with BarcodeStudio because it gives me control over how are they done and it generates the .csv needed.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Sep 27, 2020 0
Participant ,
Sep 26, 2020

Copy link to clipboard

Copied

The reason why it gets so slow is simple. InDesign builds the merged files in the RAM and once it reaches its limits it starts using hard disk space. The more pages you have in the merged document (and the more data) the slower it gets.

 

The solution is simple. Break it down into chunks. Try to merge ranges. You will be faster then doing all at once.

 

If you are on macOS 10.13 or newer you may try MyDataMerge which does that for you automatically: it chuncks the file and (if you want to) merges all generated files back into an indesign book. We use it regularly for large merges.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Sep 26, 2020 0
New Here ,
Sep 27, 2020

Copy link to clipboard

Copied

Thank you very much. That was very useful.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Sep 27, 2020 0