Combining or Merging Indesign Files (IDML/INDT)

Report · Jun 29, 2015

Greetings,

I would like to inquire about a possible performance problem regarding combining multiple IDML / INDT files or copying pages via Indesign Server Scripting API.

Note that we are using Indesign Server CS6 (with latest 8.0.2 update).

Currently, we are using the following line to merge multiple indesign documents (IDML/INDT)

document.pages.everyItem().duplicate(LocationOptions.AFTER, baseDocument.pages[lastPageIndex])

Above line performs what is needed (textframes are threaded, etc), however there is a possible issue with its performance which degrades horribly especially after several merging operations and if there are several pages in the base document or the document to be copied (though the performance is more affected when the document to be copied has many pages)

Testing done:

#1 For the below test, Notice that the merging is not constant and degrades per minute (performance may possible degrade when using a larger document)

start	end	time elapsed for pages.everyItem.duplicate	operation done
with a base of 42 page INDT document
12:13:33	12:15:28	0:01:55	add 42 page INDT file
12:15:29	12:18:35	0:03:06	add another 42 page INDT file
12:18:37	12:22:53	0:04:16	add another 42 page INDT file
12:22:54	12:28:15	0:05:21	add another 42 page INDT file
12:28:17	12:34:30	0:06:13	add another 42 page INDT file
		TOTAL
		0:20:51

with a base of 42 page IDML document
12:13:58	12:16:44	0:02:46	add 42 page IDML file
12:16:56	12:20:59	0:04:03	add another 42 page IDML file
12:21:11	12:26:21	0:05:10	add another 42 page IDML file
12:26:35	12:32:39	0:06:04	add another 42 page IDML file
12:32:50	12:40:06	0:07:16	add another 42 page IDML file
		TOTAL
		0:25:19

Time elapsed is 20 minutes for INDT and 25 minutes for IDML. it adds ~1 minute for each document added (for a 42 page, this duration may increase when no of pages increase).

#2 For below test, we noticed that INDT seems to perform faster than IDML (though it is larger by 10x and is a binary file)


with a base of 838 page IDML document
14:30:44	14:55:40	0:24:56	add 42 page IDML file


with a base of 838 page INDT document
15:14:55	15:23:40	0:08:45	add 42 page INDT file

#3 For below test, notice that the merging degrades by several minutes even if you just increase the number of pages.

Notice that we added the documents according to increasing no of pages as we found out that it performs significantly faster than not (saved several hours)

I also incorporated the doc.save and gc calls but it still doesn't help reduce the whole merging process to several minutes. (still takes 11 hours just to merge 33 documents with total of 1046 pages)

Note that we also checked binary merging (merging small files first but the resulting time is much larger than above)

with a base of 1 page IDML document
15:58:45	15:58:52	0:00:07	add 1 page IDML file
15:58:53	15:59:00	0:00:07	add 1 page IDML file
15:59:01	15:59:08	0:00:07	add 2 page IDML file
15:59:09	15:59:18	0:00:09	add 2 page IDML file
15:59:19	15:59:29	0:00:10	add 3 page IDML file
			perform $.gc and doc.save into temp IDML
15:59:35	15:59:42	0:00:07	add 4 page IDML file
15:59:44	15:59:52	0:00:08	add 6 page IDML file
15:59:54	16:00:05	0:00:11	add 6 page IDML file
16:00:08	16:00:21	0:00:13	add 6 page IDML file
16:00:23	16:00:49	0:00:26	add 7 page IDML file
			perform $.gc and doc.save into temp IDML
16:00:53	16:01:03	0:00:10	add 7 page IDML file
16:01:05	16:01:19	0:00:14	add 8 page IDML file
16:01:22	16:02:18	0:00:56	add 10 page IDML file
16:02:22	16:04:12	0:01:50	add 17 page IDML file
16:04:17	16:06:41	0:02:24	add 19 page IDML file
			perform $.gc and doc.save into temp IDML
16:06:50	16:09:14	0:02:24	add 20 page IDML file
16:09:19	16:12:31	0:03:12	add 20 page IDML file
16:12:37	16:15:53	0:03:16	add 21 page IDML file
16:15:59	16:21:00	0:05:01	add 26 page IDML file
16:21:07	16:27:25	0:06:18	add 27 page IDML file
			perform $.gc and doc.save into temp IDML
16:27:40	16:38:28	0:10:48	add 37 page IDML file
16:38:38	16:50:42	0:12:04	add 39 page IDML file
16:50:55	17:08:09	0:17:14	add 42 page IDML file
17:08:21	17:28:39	0:20:18	add 44 page IDML file
17:28:55	17:58:32	0:29:37	add 52 page IDML file
			perform $.gc and doc.save into temp IDML
17:58:58	18:26:26	0:27:28	add 52 page IDML file
18:26:42	19:11:57	0:45:15	add 63 page IDML file
19:12:16	20:08:49	0:56:33	add 64 page IDML file
20:09:09	21:13:22	1:04:13	add 67 page IDML file
21:13:59	22:39:28	1:25:29	add 67 page IDML file
			perform $.gc and doc.save into temp IDML
22:40:15	0:35:17	1:55:02	add 84 page IDML file
0:35:48	3:30:53	2:55:05	add 98 page IDML file
		TOTAL
		11:26:36

Notice that I have already added $.gc and doc.save per 5 documents however the effect is lost/not significant around the 4th/5th time.

Please confirm if there is something that can be done to resolve above Indesign's performance for merging / copying pages from one document to another.

or suggest alternative code or some setting that we may have missed.

Report · Jun 30, 2015

$.gc rarely makes a significant difference.

Are you doing doc.save() or doc.save(docFile)?

The second would make a (sometimes significant) performance difference because it dumps all the undo data, while the improvement by the first would be minimal.

Harbs

Report · Jun 30, 2015

Another thing you can try (I don't know that it will make a difference):

document.pages.everyItem().duplicate(LocationOptions.AT_END, baseDocument)

Additionally, IIRC, duplicating the pages one at a time and re-linking the text threads manually can help performance.

You also might want to try duplicating spreads instead of pages. (Not sure if that will work for your documents.)

Report · Jun 30, 2015

Hi Harbs,

@doc.save/doc.save(file), I'm already doing doc.save(file).

@LocationOptions.AT_END, I remembered that we tried this earlier and there isn't any difference.

@duplicating pages one at a time and relinking, thanks for the suggestion. I'll try this and get back later.

@duplicating spreads, will also look into this but not sure if applicable.

Thanks

Report · Jul 01, 2015

Another thing worth trying:

InDesign does not clear memory/swap files until a script finishes running. I’ve seen some incredibly large swap files from the scripting engine. (like 2GB+)

If that’s your problem (highly likely), one thing that can really help is to chunk your script and process the pieces in IDLE events. When InDesign goes idle it normalizes memory.

One last suggestion: I don’t know what’s in you files, but I’d guess it’s an awful lot of text. Lots of style overrides in text greatly effects performance. If you have control of the styling, the more that can go into styles, the better.

Please report back if something helps…

Harbs

Report · Jul 01, 2015

For below test, we noticed that INDT seems to perform faster than IDML (though it is larger by 10x and is a binary file)

The reason is because the IDML needs to be parsed, while the InDesign file is all native (it's really a database file). IDML will ALWAYS take more time to open and it'll always be smaller.