Why is manual link folder replacement faster than script in InDesign?

[Aprking]

Hello everyone,

I have a question to ask. When I use this script to change the target folder of links, if there are many linked files, the entire script takes a long time to complete. However, if I rename the source directory first and then use InDesign's Links panel to replace the target folder for finding missing files, this method is much faster, only a fraction of the time it takes for the script to run.

What could be the reason for this significant difference in speed?

Is there any way to optimize this script?
Thank you all.

var originalFolderPath = "F:/source";
var targetFolderPath = "e:/target";

var doc = app.activeDocument;
var links = doc.links;
var updatedLinksCount = 0;
for (var i = 0; i < links.length; i++) {
    var link = links[i];
    var originalLinkPath = link.filePath;
    if (originalLinkPath.indexOf(originalFolderPath) === 0 || originalLinkPath.indexOf(originalFolderPath.replace(/\//g, "\\")) === 0) {
        var originalFileName = originalLinkPath.substring(originalLinkPath.lastIndexOf('/') + 1);
        if (originalFileName == originalLinkPath) {
            originalFileName = originalLinkPath.substring(originalLinkPath.lastIndexOf('\\') + 1);
        }    
        var targetFilePath = targetFolderPath + "/" + originalFileName; 
        //$.writeln("Target File Path: " + targetFilePath);
        var targetFile = new File(targetFilePath);
        if (targetFile.exists) {
            link.relink(targetFile);
            updatedLinksCount++;
        }
    }
}

if (updatedLinksCount > 0) {
    alert("Changed: " + updatedLinksCount + " links to files with the same names in the target folder!");
} else {
    alert("No links to update found!");
}

[rob day]

Here it is with a timer added—multiple tests are plus or minus 3 secs, so available memory might be a factor:

var st = new Date().getTime ()
var targetFolderPath = "~/Desktop/target/" 
var links = app.activeDocument.links.everyItem().getElements();
var updatedLinksCount = 0;
var targetFile;
for (var i = 0; i < links.length; i++) {
    targetFile = File (targetFolderPath +  links[i].name);
    if (targetFile.exists) {
        //links[i].parent.place(targetFile)
        links[i].relink(targetFile);
        updatedLinksCount++;
    }
}  
var et = new Date().getTime ()
alert((et-st)/1000 + ": Seconds")

[rob day]

Can you tell me how many links are in the documentation?

320 links, 40 are unique. Are you testing a target on your startup drive?

[Aprking]

Local hard drive, of course。

This is my test data: 20page×40links/4.52s

50pagex40links/14.752s

100pagex40links / 32.159s

365pagex40links/835.581s

The key is the number of links, but the processing time increases exponentially when it reaches a threshold.

[rob day · Accepted solution]

Maybe try upating the links by page so you avoid such large arrays. This was 6x faster for me on a doc with 40 links per page:

vs. all doc links

var st = new Date().getTime ()
var targetFolderPath = "~/Desktop/target/"

var p = app.activeDocument.pages.everyItem().getElements()
var updatedLinksCount = 0;
var pg, targetFile;
for (var i = 0; i < p.length; i++){
    pg = p[i].allGraphics
    for (var j = 0; j < pg.length; j++){
        if (pg[j].itemLink != null) {
            targetFile = File (targetFolderPath +  pg[j].itemLink.name);
            if (targetFile.exists) {
                pg[j].itemLink.relink(targetFile);
                updatedLinksCount++;
            }
        } 
    };   
};   
var et = new Date().getTime () 
alert((et-st)/1000 + ": Seconds")

[m1b]

Hi @rob day, at first I thought this was going to be a sure winner, but I noticed a difference between using doc.links and doc.pages[i].allGraphics[j].itemLink ... when I do the pages[i].allGraphics way (see my test code below) the script updates *every link to that one file* in one go.

This wrecks my test setup, because while my test document has 500 placed graphics per page and 30 pages (doc.links.length == 15000) it only has 71 *unique* linked files (duplicated). So the `updatedLinksCount` reports as 71—not 15000—and the running time goes down proportionally—6 seconds vs 350-600 seconds. So, to clarify, this does work, and blazingly fast, but I'd like to know how fast it is on a document with 15000 unique linked files. Maybe no better than the other methods.

@Aprking if your document has many instances of the same linked file, then this would definitely be the way to go.

All very curious.

- Mark

function main() {

    // TESTING VERSION - PER PAGE

    var results = [];
    var st = new Date().getTime()

    var originalFolderURI = "file:/Users/mark/Library/CloudStorage/Dropbox/TEST/MY%20FOLDER";
    var targetFolderURI = "file:/Users/mark/Desktop/MY%20FOLDER";

    var doc = app.activeDocument;
    var everyGraphic = doc.pages.everyItem().allGraphics;

    var updatedLinksCount = 0;

    for (var g = everyGraphic.length - 1, pageGraphics, link, originalLinkURI, originalFileName; g >= 0; g--) {

        pageGraphics = everyGraphic[g];

        for (var i = pageGraphics.length - 1; i >= 0; i--) {

            link = pageGraphics[i].itemLink;
            originalLinkURI = link.linkResourceURI;
            originalFileName = link.name;

            // $.writeln('originalLinkPath = ' + originalLinkURI);

            if (0 !== originalLinkURI.indexOf(originalFolderURI))
                // no match
                continue;

            link.reinitLink(originalLinkURI.replace(originalFolderURI, targetFolderURI));

            updatedLinksCount++;

        }

        results.push('Page ' + g + ': ' + Math.round(((new Date().getTime() - st) / 1000)) + ' seconds elapsed');

    }

    results.push('Total time: ' + (Math.round((new Date().getTime() - st) / 1000)) + ' seconds elapsed');

    alert('Results:\n' + results.join('\n'));

    if (updatedLinksCount > 0) {
        alert("Changed: " + updatedLinksCount + " links to files with the same names in the target folder!");
    } else {
        alert("No links to updated!");
    }

};

app.doScript(main, ScriptLanguage.JAVASCRIPT, undefined, UndoModes.ENTIRE_SCRIPT, "Update Links");

[Robert at ID-Tasker]

@m1b 

So this line:

var everyGraphic = doc.pages.everyItem().allGraphics;

somehow filters out duplicates???

[m1b]

@Robert at ID-Tasker, I cannot explain it. When I dump the Link objects in the debugger and compare them, there is no difference at all. But the reinitLink method has a different effect on the allGraphics version. Also weird is that link.specifier() returns identical specifier string for both ways!

Maybe I've messed up my testing and confused the issue—it would best to see if anyone else can confirm.

- Mark

[Aprking]

Due to being on a Windows system, I changed the paths like this:

var originalFolderURI = "file:/F:/test";
var targetFolderURI = "file:/F:/test01";

But the replacement wasn't successful.

[m1b]

Hi @Aprking, I would expect that URIs are cross-platform. Can you uncomment the following line in the script:

// $.writeln('originalLinkPath = ' + originalLinkURI);

This should show you what originalFolderURI and targetFolderURI should look like. Bear in mind that special characters must be encoded,—eg. space is %20 and you can use encodeURI(myPath) to do it for you—but you shouldn't need to if you just copy the URI from one of the originalLinkURI.

- Mark

[Robert at ID-Tasker]

Hi @Aprking, I would expect that URIs are cross-platform.

By @m1b

Looks like they are not:

First - PC - the rest - Mac.

[m1b]

@Robert at ID-Tasker those URIs look cross-platform to me: protocol is "file:", delimiter is "/". Same as on Mac. The only thing that looks different is the "C:" but I guess that is part of the URI.

@Aprking, based on Robert's screenshot, you URIs might be:

var originalFolderURI = "file:F:/source";
var targetFolderURI = "file:e:/target";

[Robert at ID-Tasker]

@Robert at ID-Tasker those URIs look cross-platform to me: protocol is "file:", delimiter is "/". Same as on Mac. The only thing that looks different is the "C:" but I guess that is part of the URI.

By @m1b

For me cross-platform would mean identical - the fact that there is "C" - makes it completely different.

And if you take a look at "normal" links names - they are the same as URI - less the "file:" and "\" instead of "/".

@Aprking, based on Robert's screenshot, you URIs might be:

var originalFolderURI = "file:F:/source";
var targetFolderURI = "file:e:/target";

By @m1b

That's what I've suggested already.

And it should be capital "E".

[Robert at ID-Tasker]

@Robert at ID-Tasker those URIs look cross-platform to me: protocol is "file:", delimiter is "/". Same as on Mac. The only thing that looks different is the "C:" but I guess that is part of the URI.

By @m1b

For me cross-platform would mean identical - the fact that there is "C" - makes it completely different.

And if you take a look at "normal" links names - they are the same as URI - less the "file:" and "\" instead of "/".

@Aprking, based on Robert's screenshot, you URIs might be:

var originalFolderURI = "file:F:/source";
var targetFolderURI = "file:e:/target";

By @m1b

That's what I've suggested already.

And it should be capital "E".

By @Robert at ID-Tasker

And that's why I always use UCase() when comparing links - part of - supplied by user. 

[Aprking]

for (var i = pageGraphics.length - 1; i >= 0; i--) {

link = pageGraphics[i].itemLink;
originalLinkURI = decodeURI(link.linkResourceURI); // Decode the link path
originalFileName = link.name;

if (originalLinkURI.indexOf(originalFolderURI) === 0) { // Check if the link is located in the original folder
var targetLinkURI = originalLinkURI.replace(originalFolderURI, targetFolderURI);
targetLinkURI = encodeURI(targetLinkURI); // Encode the target link path
link.reinitLink(targetLinkURI); // Update the link path
updatedLinksCount++;
}

}
Successfully changed the link paths, but the files whose paths were changed weren't updated. You still need to manually click "Update All Links" in the Links panel.

[m1b]

Hi @Aprking, If the links are out of date add in the extra line:

link.update();

You can check link.status also if you need to.

The big question was, how fast was it on your massive document? In my test I didn't have to update the links—I was just re-locating the identical files—so that will take longer I guess.

- Mark.

[Aprking]

I added

link.update();

but it still doesn't work. I also included:

for (var i = pageGraphics.length - 1; i >= 0; i--) {

    link = pageGraphics[i].itemLink;
    originalLinkURI = decodeURI(link.linkResourceURI); 
    originalFileName = link.name;

    if (0 !== originalLinkURI.indexOf(originalFolderURI))
        // no match
        continue;

    var targetLinkURI = originalLinkURI.replace(originalFolderURI, targetFolderURI);
    targetLinkURI = encodeURI(targetLinkURI); 
    link.reinitLink(targetLinkURI); 
    if (link.status === LinkStatus.LINK_OUT_OF_DATE) {
        $.writeln('is me');
        link.update();
        updatedLinksCount++;
    }
}

Still ineffective, cannot update immediately.
However, I think this method might be the fastest way.

[Robert at ID-Tasker]

Due to being on a Windows system, I changed the paths like this:

var originalFolderURI = "file:/F:/test";
var targetFolderURI = "file:/F:/test01";

But the replacement wasn't successful.

By @Aprking

Have you checked how URI should "look like"?

First line on the bottom list - is file placed from my local drive - there is no "/" after "file:" - the rest are missing links from Mac.

[Robert at ID-Tasker]

I have tested it countless times. When the document is under 100 pages, the processing time still increases linearly, typically ranging from a few seconds to a few tens of seconds. However, when the page count exceeds 300 pages and the number of links exceeds 12,000, the processing time increases exponentially. It usually takes several tens of minutes or even over an hour. The script optimizations provided by everyone can only reduce the processing time by a few seconds. I still haven't found the reason for this. Nevertheless, I appreciate everyone's help.

By @Aprking

12k links in a single INDD file??

You either need to "optimise" your links - maybe combine them - or split your file into smaller chunks - and use Book feature.

What kind of documents are you creating?

[Peter Kahrel]

> the processing time increases exponentially when it reaches a threshold

I've seen that in other areas as well, e.g. adding hyperlinks. After creating about 500 links InDesign starts to slow down dramatically. There's not really very much that you can do about it.

[Robert at ID-Tasker]

> the processing time increases exponentially when it reaches a threshold

I've seen that in other areas as well, e.g. adding hyperlinks. After creating about 500 links InDesign starts to slow down dramatically. There's not really very much that you can do about it.

By @Peter Kahrel

That's because of the UNDO history.

But there is a way - save to a new file after processing a set number of operations - let's say after every 200 operations - or close to whatever bottleneck you'll discover.

@Aprking 

Modify your code to save your INDD file to a new file with a new name every page or few pages - when you switch to processing links by pages - you'll purge UNDO history and you won't have exponential slowdown.

[Robert at ID-Tasker]

@Aprking 

Modify your code to save your INDD file to a new file with a new name every page or few pages - when you switch to processing links by pages - you'll purge UNDO history and you won't have exponential slowdown.

Extra bonus - if it crashed on say page 9 - you don't have to process again first 8 pages - you can just open 8th file and start proccessing from page 9.

[Peter Kahrel]

Saving the file on to itself and clear its undo history sometimes helps, but in the case of the hyperlinks I mentioned, removing the undo history didn't help. It was the sheer number of hyperlinks that was the problem. It looked as if InDesign allocates a certain amount of memory to hyperlinks, and when that memory fills up there are problems. It does continue to add links, but very slowly. I've no idea reaaly how it works, but memory allocation seems to be involved. It looks as if you have the same problem with links to files.

When you consider the number of links, you should factor out duplicates. I the same file is linked 10 times, the file has 1 link: the other 9 are pointers, they're not really links. @Aprking has 12K unique links, which is not the same as 300 links duplicated 40 times. The latter file still has 300 real links, the remainder are pointers.

[Robert at ID-Tasker]

Saving the file on to itself and clear its undo history sometimes helps, but in the case of the hyperlinks I mentioned, removing the undo history didn't help.[...]

By @Peter Kahrel

I would NEVER suggest "Saving the file on to itself " - only to a new file with a new name - with a counter.

And what kind of hyperlinks? To places inside the file - different files - or websites / e-mails?

[...]It looked as if InDesign allocates a certain amount of memory to hyperlinks, and when that memory fills up there are problems. It does continue to add links, but very slowly. I've no idea reaaly how it works, but memory allocation seems to be involved. It looks as if you have the same problem with links to files.

You might be right... after InDesign creates object - its properties need to be saved - and collection updated and re-indexed.

The more objects in the collection - the more updates to the whole collection and re-indexing - at some point - with not enough memory - InDesgin starts swapping info between physical RAM and drive... and even fastest SSD is 1000x times slower than RAM...

I need to finally finish my benchmark tool and run it on the beast... with extra test on RAMdisk...

[Peter Kahrel]

> I would NEVER suggest "Saving the file on to itself " - only to a new file with a new name - with a counter.

I've been saving files on themselves for more than 20 years and never had a problem. In fact, it's my default way of saving files so I never have undo histories.

> And what kind of hyperlinks? To places inside the file - different files - or websites / e-mails?

Doesn't matter. I think most of them were URLs.

[Robert at ID-Tasker]

> I would NEVER suggest "Saving the file on to itself " - only to a new file with a new name - with a counter.

I've been saving files on themselves for more than 20 years and never had a problem. In fact, it's my default way of saving files so I never have undo histories.

[...]

By @Peter Kahrel

I prefer to be safe than sorry.

[...]

> And what kind of hyperlinks? To places inside the file - different files - or websites / e-mails?

Doesn't matter. I think most of them were URLs.

I think it might.

What was the status of the HYPERLINKS pallet? Hidden or visible?

Like with other pallets - InDesign is constantly trying to check / upate the status - so if pallet was open - it might be the reason why it was so slow... and if your network connection was slow / intermittent / off...