Why is manual link folder replacement faster than script in InDesign?

[Aprking]

Hello everyone,

I have a question to ask. When I use this script to change the target folder of links, if there are many linked files, the entire script takes a long time to complete. However, if I rename the source directory first and then use InDesign's Links panel to replace the target folder for finding missing files, this method is much faster, only a fraction of the time it takes for the script to run.

What could be the reason for this significant difference in speed?

Is there any way to optimize this script?
Thank you all.

var originalFolderPath = "F:/source";
var targetFolderPath = "e:/target";

var doc = app.activeDocument;
var links = doc.links;
var updatedLinksCount = 0;
for (var i = 0; i < links.length; i++) {
    var link = links[i];
    var originalLinkPath = link.filePath;
    if (originalLinkPath.indexOf(originalFolderPath) === 0 || originalLinkPath.indexOf(originalFolderPath.replace(/\//g, "\\")) === 0) {
        var originalFileName = originalLinkPath.substring(originalLinkPath.lastIndexOf('/') + 1);
        if (originalFileName == originalLinkPath) {
            originalFileName = originalLinkPath.substring(originalLinkPath.lastIndexOf('\\') + 1);
        }    
        var targetFilePath = targetFolderPath + "/" + originalFileName; 
        //$.writeln("Target File Path: " + targetFilePath);
        var targetFile = new File(targetFilePath);
        if (targetFile.exists) {
            link.relink(targetFile);
            updatedLinksCount++;
        }
    }
}

if (updatedLinksCount > 0) {
    alert("Changed: " + updatedLinksCount + " links to files with the same names in the target folder!");
} else {
    alert("No links to update found!");
}

[Aprking]

There is also a possibility that the issue is unrelated to the UNDO history. Processing the segmented array would release memory as it processes, and this is the reason for the speed improvement.

[Aprking]

The code provided by @rob day:

links = app.activeDocument.links.itemByRange(si,li).getElements();

Took 9724.496 seconds to execute, and I nearly went crazy during testing. After analyzing it, I came to the following conclusions:

This segmented retrieval method indeed reduces memory usage because it only extracts a small portion of link objects each time, which is relatively more efficient. However, in practice, each retrieval operation for link objects involves interaction with the InDesign engine, introducing certain overheads including execution time and computational resources. On the other hand, the previous method, although retrieving all link objects at once, may consume more memory but reduces the number of interactions with the engine. Therefore, in most cases, the overall execution efficiency might be higher. Additionally, when processing links in bulk, the number of links in each batch is not the only factor affecting performance; other factors such as nested loops and specific operations performed on links will also impact performance.

[Robert at ID-Tasker]

Final statistical report: Document size 752,016KB, 365 pages, with 40 links per page, totaling 14,595 links. As per guidance from @Peter Kahrel , among these, 729 are actual links, while the remaining 13,866 are pointers.

[...]

By @Aprking

OK. I always try to ask basic questions at the beginning - but this time - I failed...

So let's start from the basics - what exactly are your files?

Can you show us some screenshots?

First suggestion - any chance you could use Master Spreads and eliminate duplicates by shifting links there?

Why do you have to relink to new folder?

How often do you have to do this?

Maybe you wouldn't need script at all - just a different workflow...

[Aprking]

@Robert at ID-Tasker 

I apologize, but due to confidentiality reasons, I cannot share screenshots of the files provided by the client. Please accept my apologies for this limitation.

However, I can provide a detailed explanation of the content. The document in question is a catalogue spanning several hundred pages, containing a significant amount of text and graphics. The graphics include various formats such as TIFF, PDF, EPS, etc. Typically, each page contains dozens or even hundreds of linked graphics. To cater to different publishing requirements, these catalogues often include links in the same name but different color modes, such as Gray, CMYK, and RGB. Consequently, three separate PDFs are generated: one in RGB mode, one in Gray mode, and one in CMYK mode. After meticulously proofing and publishing the PDF in RGB mode, the next step is to update the links to generate PDFs in Gray and CMYK modes. Additionally, each mode has its subfolders, further increasing the complexity of the process. The total number of links often exceeds ten thousand, hence my pursuit of efficiency.

And furthermore, I find myself caught in a peculiar cycle: whenever I encounter tasks that require repetitive completion, I invest a significant amount of time in writing scripts or plugins to reduce the workload. However, once the workload is reduced, it grants me ample free time. Subsequently, I use this free time to write more scripts or plugins to further reduce the workload.

Well, I admit it, I am the Ouroboros of productivity...

Once again, I apologize for the inability to share screenshots of the document. I am deeply grateful for your assistance and understanding.

[Robert at ID-Tasker]

@Robert at ID-Tasker 

I apologize, but due to confidentiality reasons, I cannot share screenshots of the files provided by the client. Please accept my apologies for this limitation.

[...]

By @Aprking

Of course. No problem.

But I'm pretty sure there would be a way to reduce number of links...

[Pickory]

Calling your script from another script with an appropriate undo mode might be worth a test.

P.

===

It doesn't need to another script, just the function.

[m1b]

Hi @Aprking and everyone,

Well, I'll mention this because it hasn't been discussed, and I'm not sure if it's sensible or not, but I just did a proof-of-concept test of editing the links as plain text via .idml file. I'm using MacOS so you will have to replace command line tools with Windows equivalents (eg. 7zip?).

Disclaimer: I know hardly anything about IDML files! What worked for me may not work in your case for reasons I didn't consider, so, for example, in step 4 your files might be differently-named or -structured. Thanks to @Peter Kahrel for teaching me recently that IDML files were .zip files.

What I did:

1. export demo.idml

2. make a folder called "expanded" in the same directory as 1.

3. decompress idml contents into the "expanded" folder—in a terminal: unzip demo.idml -d expanded

4. opened "expanded/Spreads/Spread_uce.xml" in VSCode (is this always the same file?)

5. performed find/replace targeting the link file paths (this would probably match the string replacement you did in your script).

6. save and close Spread_uce.xml file.

7. re-compress the .idml file—in terminal again: cd expanded; ls | zip -@ -r ../demo2.idml (be careful with this step, the re-zipped file must have the exact same structure as the original, eg. the default behaviour of zip is to compress with a root directory—which we don't want, and Indesign won't open an IDML with an unexpected structure).

8. open demo2.idml (created in step 7).

9. the links paths will be now changed but they won't be updated (similar to using reinitLink), so you must ask Indesign to do that in one operation.

So that's the idea—good or bad. It rests on the fact that VSCode is very fast at performing the find/replace. However step 9 (updating modified links) will be slow no matter what, so ultimately this may not be so fast for you. Also if this is something you want to fully automate, then this is probably too much of a stretch. Still it adds to the discussion. 🙂

- Mark

[Aprking]

I tried using the method of editing .idml files five or six years ago, and it has two issues: Firstly, saving as an idml file takes a long time, and secondly, opening the idml file after editing takes even longer. So, this method might not be the best approach for modifying links.

Editing idml files is typically applied in cases where third-party plugins are used, and some hidden characters or links cannot be modified. In such cases, editing idml files is a good method.

Additionally, if the code we're discussing includes:

app.doScript(main, ScriptLanguage.JAVASCRIPT, undefined, UndoModes.FAST_ENTIRE_SCRIPT, "Update Links");

This code will also significantly increase the processing time. Therefore, when dealing with a large number of link changes, I tend to avoid using UndoModes.

[m1b]

Ah yes, of course. I should have considered the huge amount of time to generate the idml, etc. etc. Oh well. Thanks for letting me know @Aprking.

I'm quite curious about your comment regarding avoiding using UndoModes. Are you actually saying that the script is faster by avoiding using app.doScript? That seems plausible. Or are you specifically saying that UndoModes.FAST_ENTIRE_SCRIPT is slower than UndoModes.ENTIRE_SCRIPT? That would surprise me!

- Mark

[Robert at ID-Tasker]

I tried using the method of editing .idml files five or six years ago, and it has two issues: Firstly, saving as an idml file takes a long time, and secondly, opening the idml file after editing takes even longer. So, this method might not be the best approach for modifying links.

[...]

By @Aprking

Exactly. It will always take more time than just relinking to new links - because the rest of the file needs to be re-created from scratch as well...

[Peter Kahrel]

Nice one, Mark. Just a comment: 7Zip is not a command-line tool, it's an application. It sits in my file manager. I right-click an IDML file, click 7-Zip, then I can choose to unzip the  thing or just show its content. It's a brilliant program, available on Windows, Mac, and Linux.

There's Spread_xxxxx.xml file for every spread, so no, Spread_uce.xml is not the only possibility. If you're looking for some text in a big file you'll need a file manager that lets you search in many files in a directory.

P.

[m1b]

Thanks Peter. Well, it was an interesting little exercise.

> If you're looking for some text in a big file you'll need a file manager that lets you search in many files in a directory.

Yes, VSCode is excellent and incredibly fast at find/replace in multiple files at once.

- Mark

[m1b]

@Aprking this is nothing you don't know, but I realised we have not explicitly answered your main question:

> Why is manual link folder replacement faster than script in InDesign?

The unsatisfying answer is: native features are always faster, in terms of raw speed, than going via the scripting API, which introduces performance overhead, sometimes quite significantly. The "manual link folder replacement" you mention relies on a native update links feature so it is faster.

We have given the topic a good workout on this thread though.

- Mark

[Aprking]

From my testing, I found that using UndoModes, regardless of the parameter value, significantly increases processing time. Therefore, I have no choice but to abandon its usage.

Regarding the topic discussed by @m1b , "Why is manual link folder replacement faster than script in InDesign?", we have successfully reduced the processing time of the script compared to the native functionality. So, we should express our gratitude for your participation and assistance.

[m1b]

Hi @Aprking, 

> From my testing, I found that using UndoModes, regardless of the parameter value, significantly increases processing time.

Do you mean that you don't use app.doScript? Sorry that I'm not understanding.

> we have successfully reduced the processing time of the script compared to the native functionality

Oh that's great news! After you post about it taking 9724.496 seconds to execute, I was worried that on the real file it still wasn't fast enough.

- Mark

[Aprking]

I am planning to use the following code to decide whether to use app.doScript:

if (undoCheckbox) {
    app.doScript(main, ScriptLanguage.JAVASCRIPT, undefined, UndoModes.FAST_ENTIRE_SCRIPT, "Update Links");
} else {
    main();
}

[m1b]

Ah, thank you. Please let us know when you have finished what you finally decided. 🙂

On one hand I would expect app.doScript to be slower because it seems like introducing another code scope/context, but also that it may be faster due to combining Undos into one. We will see. Thanks again.

[Robert at ID-Tasker]

[...] native features are always faster, in terms of raw speed, than going via the scripting API, which introduces performance overhead, sometimes quite significantly. The "manual link folder replacement" you mention relies on a native update links feature so it is faster.

By @m1b

Yes, It has been pointed out already...