How to speed up PDF generation in InDesign

[tonyrush]

One of the main tasks that we do in our office is:

A.  Merging an original InDesign file (approx 7.5Mb) with a CSV file;

B.  Exporting that resulting merged document into a PDF file

During export, we've tried different Adobe PDF Presets:  (High Quality Print, Press Quality, Smallest File Size, PDF/X-4, etc.)   The "Smallest File Size" preset seems to be the fastest although not by much.  It generally takes 3-5 minutes to create the merged document which seems a bit slow.  

But the real sluggishness is when we export that merged document into PDF format.  It takes anywhere from 4-8 minutes and the resulting PDF is only about 2.3Mb in size. 

We've tried this on multiple machines.  One is an iMac 3.5Ghz Quad Core i7 processor with 32Gb of 1600Mhz DDR3 RAM.  The other is a Macbook Pro with 2.6 Ghz 6-core i7 processor and 16Gb 2400 Mhz DDR4 RAM.  There's not really thimble's worth of difference in the export time.  They pretty much run neck and neck and, when they don't, either one might outperform the other.

Anyway, I have two questions:

A.  Is there anything I can do to speed up this process?  It takes HOURS of our day for one staff member to create these PDFs.  And most of that time is spent just waiting for the machine.

B.  What part of the computer is most integral to this kind of task?  In other words, if I choose to get a new Mac before the end of the year -- and had to choose -- would I want to go with the fastest processor...or the most RAM?  Which would give us the biggest boost in speed for this kind of work?

Thanks in advance.  I appreciate your help and would be happy to provide more information if it will help give a better answer.

[Eugene Tyson]

It depends on the content of the data being created into a PDF.

A plain text file of 2 pages could take nanoseconds. A 1000 pages could take minutes.

A file with 2 pages of vector heavy content could take 10 minute. A file with 1000 pages of raster images could take the same time. 

How is the merge taking place?
Are you creating an InDesign first from the Merge resulting in a 1000's of pages of an InDesign File?
Are you going directly to PDF from the Data Merge Panel?

Does it work faster if you split the merge in 2 or into 4 or into 6 or 10 different merges?

It's very hard to direct without any visuals or a clear workflow of what you're doing 🙂

Any more clues for us 😄 

[James Gifford—NitroPress]

I don't know enough about the merge process, but is what you see in ID a finished merge, or is the merge operation repeated at the time of print or export? If ID is having to update or re/generate those pages on the fly, several minutes to export is to be expected.

Is it possible to save a merged document as a static one, export from that, and thus bypass that second merging/validation process?

[Sherry @ Lacy & Par]

I believe the result of a data merge in InDesign is always a static piece. There are no live merge options. It acts as though it was a normal InDesign document.

[Loic.Aigon]

To second @Eugene Tyson PDF generation delays take the time it takes depending both the complexity of the file and the capacity of the machine. If there was a "magical" way to speed up the pdf generation, it's likely Adobe would have implemented it by default.

A possibility though is avoiding network ressources, if you have links on your network and a low bandwith, it may be that InDesign struggles to retrieve data from the network. if so, you may try to use local resources and it should reduce generation delays.

Once that said, you may try though scripting with an invisible document and no refresh on screen (see enableRedraw property of script preferences) or through InDesign Server but I don't think you will save that much.

A last resort would either be to sequence PDF exports (page ranges) or to use a Plugin such as DesignMerge from Meadows Publishing. They propose PDF/VT export options which can be helpful for large merging jobs.

FWIW

Loic

[Robert at ID-Tasker]

Like already suggested - if you link your files over network - move to local drive.

Memory - never enough 😉

Local drive - do you have SSD or HDD? With enough extra RAM you could create RamDisk - 100s times quicker than fastest SSD 😉 

If your data is prepared perfectly - maybe you could do it overnight or during the day on a separate machine - batch processing with scripting? 

[Sherry @ Lacy & Par]

I don't know how much more perfectly it could be set up when it's just a single address on every other sheet, but open to any and all suggestions. 🙂

I've done the overnight runs and have a script that will save the INDD files to PDFs in batches. The problem I'm having recently is with either the script crashing or the program crashing, mid-save. 

[Robert at ID-Tasker]

I don't know how much more perfectly it could be set up when it's just a single address on every other sheet, but open to any and all suggestions. 🙂 

By Sherry @ Lacy & Par

If those are just texts - you could just import a TXT / RTF / DOC(X) file and reflow it on pages? 

Will be WAY quicker than DataMerge.

And even if it's a plain text in Excel - it can be formatted as InDesign Tagged Text - and imported to InDesign extremely quickly. 

Or imported as a table, each column formatted as desired - then converted to text. 

I've done the overnight runs and have a script that will save the INDD files to PDFs in batches. The problem I'm having recently is with either the script crashing or the program crashing, mid-save. 

---

Maybe you just need a better tool - but if it's just a text and a simple layout - you don't really need DataMerge. 

[Sherry @ Lacy & Par]

Some background: We are a mail house and these mail pieces are barcoded and sorted, so I am afraid we really require the variable data features as each address doesn't take up the same amount of space. The background image of each post card is saved as a 2 page (front and back) master file. The variable data is set up as an overprint and the two files are "married" on the digital press. 

What I don't understand is having done these as data merges with larger and even more complex jobs for years, how it's only been recently we have been running into problems. Our quantities haven't increased. We aren't doing VDP on graphics, just the text (even the barcode is a typeface) -- literally nothing has changed but the amount of time it takes to create the PDF.

40+ minutes to save a PDF that is only 5.2 mb in size seems a bit much, when I can impose 8 of these files into cutstacks, 4up with crops and color bars, in less than 5 minutes. 

I get that it may be time to shell out some bucks for a more robust VDP solution; it's just that it is going to be a hard-sell for my employer when up until now the built in Data Merge feature has been working fine.

I just want to know what changed... because it's not our files.

[James Gifford—NitroPress]

40+ minutes to save a PDF that is only 5.2 mb in size seems a bit much

I don't think it has anything to do with saving the PDF, which from a static file should take 10 seconds to a minute at most.

It's in the data-shuffling for each page. Either ID has changed something to make this less efficent, your data or methods have changed in a way that slows things down, or there's a system problem. If you say the (same in all respects) process used to be faster, then you need to figure out what's changed. I'd put changes in ID low on that list — there haven't been any updates to DM in a very long time, AFAIK — and wonder if you have system, network or data-format issues that are making the process deadly slow.

Can you use some wholly different machine that runs ID well, and attempt this operation with entirely local components and data, to a local and reasonably fast destination drive?

[Sherry @ Lacy & Par]

See, that's what I thought. It SHOULD only take at most a few minutes, right? 🙂

I'm using the same iMac. Saving the PDF to local hard drive from a static INDD file created by data merge.
Not sure what could have changed on my end other than standard OS upgrades. Could that be it? 

Maybe bad RAM? Could it be as simple as needing new hardware? 

[Robert at ID-Tasker]

[...] OS upgrades. [...]

By @Sherry @ Lacy & Par

What OS version?

[...]
Maybe bad RAM? Could it be as simple as needing new hardware? 

By Sherry @ Lacy & Par

There is never enough RAM - but a bad RAM will crash your system constantly.

A new hardware can be a solution - maybe a PC this time?

[Sherry @ Lacy & Par]

I won't say I'm not open to a PC in general, but as a replacement for THIS particular workflow, there are a lot of things to consider. We have to take compatibility with our clients as well as coworkers into account, and that there is a lot of information that would be unable to make the system jump. Other proprietary software, licenses, configurations, and calibrations with our digital presses being a concern. 

Good point on the RAM. You're right. If it was corrupt, I'd be seeing other issues. Maybe 40 GB isn't enough?

I'm running Ventura 13.6.6 -- but this is a 7 year old machine. Maybe it just couldn't handle the last update and is having to throttle certain operations.

I'm running everything local to my internal hard drive (that I keep fairly cleaned off for this purpose) so I'm thinking the network isn't part of the equation.

[Robert at ID-Tasker]

@sherry @ Lacy & Par

40GB should be enough. 

Can you install v18 - if that's the one that was OK? 

About a PC - you would have to use it just to generate PDFs - or INDD files - then you can continue your workflow on Mac. 

One more thing - do you have "background tasks" enabled? 

[Sherry @ Lacy & Par]

I'd have to save down client files to the previous version and if any used features of the latest version, we'd be in trouble. Their art still needs to look as they intended.

While the PC could open the InDesign files, paying for another seat on the software might be an issue. I'm already using my two licensed uses between my office and my work from home Mac. With a new Mac, I could transfer the license because I'd be using it instead of my current office Mac.

And yes... I do have background tasks enabled. 

[James Gifford—NitroPress]

I completely understand the need to keep shop costs in line, for both hardware and software — and to keep things simple.

But I'm a bit puzzled how a shop so small as to use one shared license allows that to hamper what sound like industrial-sized projects. Maybe it's 'nunny,' as my Southie cousin use to put it... "nunnya [expletive] bidness!" But I do believe a shop should be equipped to do its jobs efficiently, not by nickel and diming the platform doing it.

[Robert at ID-Tasker]

@sherry @ Lacy & Par

I know it's too late - but why no one tested new version of InDesign before upgrading? 

I'm pretty sure you can use the same license on a PC and Mac - but not on a 3x machines... 

I don't think there has been any groundbreaking new feature in the v19 vs v18. 

Maybe try disabling those "background tasks"? 

https://creativepro.com/a-new-workaround-for-that-pesky-background-export-issue/

[Sherry @ Lacy & Par]

"I know it's too late - but why no one tested new version of InDesign before upgrading?"


Variable data isn't our only business. A lot of our clients are designers with the latest and greatest... if we can't open their files, we're useless to them.



"Maybe try disabling those "background tasks"?"


The only background task that was running was the creation of the PDF 🙂 but that said... turning it off DOES seems to help a bit, just nothing earth shattering. Looks like I'm going to be looking for a new machine.


And honestly? I'm not mad about it. My coworkers have gone through several PCs in the past 7 years. This has been my first real issue with my Mac. I think I have more than got my money's worth.

[Robert at ID-Tasker]

Variable data isn't our only business. A lot of our clients are designers with the latest and greatest... if we can't open their files, we're useless to them..
By @Sherry @ Lacy & Par

Then why not keep old version anyway? 

You can have more than one major version installed on your machine - and when new one is available - just uncheck "uninstall old version" so the old one won't be removed. 

You can still install v18 and have it together with v19. 

[James Gifford—NitroPress]

Without getting into the Win-v-Mac debate, I'd say that machine has aged out for the purposes. 40GB is generous and probably not the issue, but a newer CPU, OS, 64GB and a modern SSD for at least the data would probably be money well invested.

[Robert at ID-Tasker]

[...] but a newer CPU, OS, 64GB and a modern SSD for at least the data would probably be money well invested.

By @James Gifford—NitroPress

But for the price of a Mac with this configuration - you can buy two similarly configured PCs - or one way more powerful and still have change... 

[James Gifford—NitroPress]

I wouldn't argue any of that — my fundamental position, long divorced from the nuclear era of the hardware wars, is that I am puzzled by anyone who buys another Mac these days. Even Apple seems embarrassed that they still sell "com-pew-turs."

[Robert at ID-Tasker]

Sherry @ Lacy & Par 

Can't you go back to the older  - faster - version of InDesign?

[...] (even the barcode is a typeface) --

By @Sherry @ Lacy & Par

Then you don't need DataMerge - just properly formatted text.

[...] 40+ minutes to save a PDF that is only 5.2 mb in size seems a bit muc[...]

By @Sherry @ Lacy & Par

I don't use DataMerge - prefer my own solutions - but are you talking about generating PDF - instead of generating INDD file and then saving it as PDF?

[...]

I just want to know what changed... because it's not our files.

By @Sherry @ Lacy & Par

What is the location of your files? From v19, InDesign is extremely picky / sensitive to the network speed / latency times - so if you work over the network...

[James Gifford—NitroPress]

I think this is just a case of greatly overloading a feature not really meant to handle thousands of iterations.

I agree with Robert T: I'd flow preformatted text into a master page frame so that all the pages are semi-static, then dump that to print. I doubt you're going to find a solution that lets you use data merge and not have it take an hour per K or more.

And that some simpler/more optimal tool might also be a better solution. Even Word might be worth exploring; AFAIK its mail-merge feature is... efficient.

[Sherry @ Lacy & Par]

Up until now, I had been data merging certificates with a 15 digit Alpha-Numeric sequence, AND QR code in quantities of 100,000 records at a time with no issues. It's only been recently that saving the PDFs started taking so long.

We are already creating a static Master file separate from the overprint. The files are then combined on the digital press.