Serious InDesign Server slow down with Catalina

[Jostify]

We are stuck with serious performance issues with our scripts for InDesign Server (mostly Applescripts) after upgrading  from Mojave (macOS 10.14.6) to Catalina (macOS 10.15.7). The problems are alike with Indesign Server CC 2019 and CC 2020.

Our current project aims at a Indesign Server solution that, when taken in production (very soon), must be able to run on a machine that demands Catalina or later. And it will rely heavily on high performance.  But I’m afraid the these performance problems are deep rooted and can only be solved by Adobe and/or Apple.

We haven’t encountered any resembling performance issues with other applications than Indesign Server, or with scripts working with tasks that does not involve InDesign server on these machines or other macs running Catalina. The slow down with InDesign Server are abnormal.

The problems manifests themselves in several ways:

1. The time it takes for a script to do its tasks have raised to unacceptable levels. In the very best cases, it has more than tripled.

2. Processing time also varies a lot between executions of the same script – for no apparent reasons.

3. Processing time is often even higher when initiated by a soap call than when the same script is run from a script editor.

For evaluation I have used one and the same script working with one and the same document. The test document (relatively light in our context) is opened from a template with around 350 empty text- and image frames. The script first analyses the document, recording all frames with a certain label content and makes a setting record for those frames based on the label content. Then it imports text and place images according to those settings.

Under Mojave it takes just above one minute for the script to do the job – both when started with a soap call and directly from an editor. Processing time is varying between 1:02 and 1:12, which I consider to be a ”natural” variation.

After the upgrade, the scores for editor executions have varied between 2:08 and 3:44. Started with soap the time varies between 3:30 and above 5 minutes

In short: The increase in process time –  at least 300 to 500 percent with soap calls – is of course unacceptable. Also, the large variations is also a problem in itself. 

Notes:

• The process time scores above referes to the time amount from when the script starts to work with the document until it’s finished with the document (so any delays outside of the document processing is not included).

• We cannot pick out certain actions that has slowed more than other. The recording of document content is heavily decelerated – taking between 40 seconds and 2 minutes and more, compared to taking a few seconds with Mojave. Likewise, the process to place text and images, fit images to the frame and so on have slowed down to the same extent.

• Our experiences of the performance problems comes from a Mac Mini used for development, where the tests I refer to is done. To confirm the problems, we have also done the same tests on a fresh machine delivered from Apple with Catalina pre-installed, a lot of RAM and no other installations than the ones necessary for doing these tests – with the same disappointing results.

[Jostify]

There is one very puzzling thing, that we have observed after I wrote this post :
At some, very rare occasions the script suddenly runs as quick as it used to do before the upgrade – just above 1 minute.
This have happened two times – out of around one hundred tries with the same script. In both cases it was executed by a soap call (with otherwise has become the slowest method with Catalina).
Then, running the same script several times directly after such an occasion, gave us same disappointing results as usual: around 3:30 and sometimes more, from soap.
This tells me there is a potential to run InDesign Server scripts as fast with Catalina as with Mojave – but, in most cases, the script is heavily disturbed by something...

[Test Screen Name]

I have no experience with this specific issue, but if I may I will share some thoughts. Possibly, it is one specific method or action that is taking a very long time, and if so, you may be able to focus on solving a better-defined problem. If it were me, I would change the script to optionally write a detailed, timestamped, log file as it does the task. By looking at the timestamps, and perhaps comparing with a log from the best timing or another system, you will be able to identify a specific problem area. It may be quite unexpected; for example, I have seen the suggestion that running certain executables under Catalina causes the macOS system to contact Apple to check signatures, causing wild unexpected variations in run time, which Apple refused to consider a problem.

[Jostify]

Thanks very much for your input, Test Screen Name.

From the timestamped logging the script already do, one can se there are no eye catching time gaps between specific operations. 

I have also, by logging, separated parts that contains script commands solely targeted to Indesign Server from other parts doing other stuff (like doing calculations, running shell script commands, using Objective-C code, doing file system stuff and so on). 

From that I can conclude that the Indesign commands at least take their fair part in the slowdown, if not most of it. One bunch of code, for example, just collects data from the documents frames, such as object reference, object class and label content, storing it in a record to be processed later. In my test document there are 340 such text frames and rectangles – with no content yet when this data collection is done.

In Mojave that part took 1 or 2 seconds. In Catalina it takes 12-15 seconds. 

Later in the process there is a long part were the script fill those frames with (mostly) pictures and (some) text, then doing som adjustments like fitting the picture to the frame. Also that part of the process consists mainly of Indesign script commands – and takes much longer time after the upgrade

I could do more work to make the script log every command, with millisecond timestamps, to see if certain Indesign commands have slowed down more than others. But anyway, it is obvious that pure indesign commands in general are affected – both those for just collecting data from the document and those for doing stuff with it (which is not to say non-indesign commands are not affected). 

The delays are on the millisecond level, and everywhere. The slowdown is only shown in the sum of them.

Apples signature checking was one of the first suspicions we had (after ruling out that Catalinas new network volume security measures had something to do with that).  There are ways to totally deactivate gatekeeper and something called amfi that are involved in signature checking. We tried that in the course of our troubleshooting – but it made no difference. Well, running scripts from an editor became a little bit faster, but not running them from soap calls. 

[Dirk Becker]

Applications >> Utilities >> Activity Monitor

Choose your InDesign Server

Launch a script

From the third button / gear icon dropdown, choose "Sample Process" while the script is still executing

This should give you an idea where the Server is spending most of its time during the sample period, with some similarities to a crash log. Save the file and upload to an external service such as pastebin or dropbox - then share a link here.

[Jostify]

Here you go, Dirk:

https://www.dropbox.com/s/sev1hl6154kw2zr/Sample%20process%20InDesignServer.zip?dl=0

Thank you for taking a look at this!

[Dirk Becker]

At the very top you see how your SOAP request trickles thru to OSA (Open Scripting Architecture) and UASExecute. The high number suggest this is entered once and real activity below. Most of the time spent in UASRemoteSend, and there BAHandleEventInternal (my guess: handling the response as AppleEvent). Most of the activity is with very basic operations AEGetAttributePtr, AEGetDescDataSize, AEGetDescData and others. All of them performing an enormous amount of steps in LSCopyApplicationInformation. This is all the AppleScript runtime, not InDesign.

When you search far down for more "Support for AppleScript" you see one poor CScriptEngine::CreateScriptRequestData (the InDesign equivalent structure to the AppleEvent), some more AE copy back and forth to fill it in and something happening on a Spline. That's a random step of your code at work within InDesign itself.

Back to all those LSCopyApplicationInformation. Long ago I have given up Mac programming and left it to the folks at Adobe, so I can only speculate. When I google for that name, it looks like either an accessor to the process manager itself - get some metadata from one process info - as in search of a targeted process, or more likely LS (LaunchServices) is used as inter application / process communication vehicle. All those asString operations and stream operations sound like the latter.

So apparently (again: just a guess!) your AppleScript runtime is sending AppleEvents across process boundaries (=slow) to itself. You can maybe verify that with your faster comparison system where the same Sample Process output should have far fewer LSCopy, asString, ostream and so forth.

This marshalling across process borders should not happen as you are working within the same process. So the big question is how you address your server, and whether there alternatives so that all the AE* methods would bypass the slow path.

What are the first few lines of your script? Is it a traditional [ tell application "InDesign Server nn" ], something using the version independet approach [ tell application id "com.adobe…" ] or something else - e.g. by process ID? Is this a compiled script or a plain text? Do you use nested tell statements? Some things to try/change, I don't know whether any leads to success. Also, do you have multiple InDesign Server processes running on that machine, multiple versions installed? May your script be cross-talking to a different instance?

[Jostify]

Thanks for your expert answer, Dirk!

I don’t have the same kind of deep computer understanding at all. It is interesting to read your analysis, but I don’t have the ability to draw clear conclusions from all of it. 
I am just a self learned, but very experienced, applescripter. Automating work flows has been one of my main work tasks for almost 20 years, often with Indesign in the center, and I have learned a lot about stuff like effective script structuring along the way.

The script I use for performance test in this case – the one the sample process is taken from – is used in a real production environment. The script are executed from a soap call, but does not work alone by itself. It loads other ”child scripts” and define properties for them in turn. The reason for that (besides avoiding the main script from being too large) is that there are other such ”parent scripts” (I.e. scripts started from soap) are sharing the same code. Also, some of child scripts are (in a modified version) used in our client environment that that it has to be backward compatible with for a while. 

I believe the scripts included are well structured, but as a whole the runtime process are quite large and complicated for the above reasons. There are almost no targeting of other applications besides InDesign Server. But there are quite many simple shell script commands for things like file copying – done outside of InDesign tell blocks of course. Likewise the data calculations is done by Applescript, not InDesign. A few script libraries are also used, some of them containing Applescript Objective-C code. 

I’m sure the scripts and the process structure can be optimized further, and the number of if-statements can be reduced when backwards compatibility is not needed anymore. 

However, despite this complexity, everything worked perfectly fine with macOS Mojave. So my main question – and problem – is: Why does exactly the same processes performs so bad on Catalina? (We have invested in a new Mac Pro to be able to run many instances simultaneously on one machine, which is essential to our project, and therefore the scripts MUST work with Catalina.)

To answer the specific questions you ask (although I understand you gave me them as suggestions of what to try):
• I call applications by their id: [ tell application id "com.adobe.”InDesignServer ], not [ tell application ”InDesignServer” ]. 
• The first lines of the script? After stating some properties, the run handler begins by loading som child scripts (without any tell block, I.e. using Applescript), thereafter asking Indesign server for the script arguments used in the further process. 
• I save the scripts as compiled scpt-files. However I have tried with changing them all to plain text – not resulting in any performance gain.
• I avoid nested tell statements. Besides the fact that ”InDesignServer” is always the parent application when executing scripts from soap calls, there are no targeting of applications inside tell application-blocks.
• In the production context there are multiple InDesign Server processes running, but just one version. Right now there are two versions installed (CC2019 and CC 2020, because of our troubleshooting) but they are never launched at the same time. I will uninstall one of them, just to see if there is any difference – but have little hope that will help. 
However, when the sample was taken (the one uploaded for you), there were only one InDesign Server instance running. Actually, the time it took for the process to run at that time was even longer than it use to be (due to the sampling?)
• I’m pretty sure there is no cross-talking between instances. Avoiding that has been a main concern. Also, in the testing context, usually only one script is ran at the same time.  And, as mentioned above, running the script with only one running instance doesn’t make any difference in performance.

Anyway, as mentioned before, everything works so fine as long as the macOS is Mojave. Do you have any idea about why everything starts performing so bad (at least in 88 cases out of 100) when just one parameter – the MacOS version – is changed?

[Test Screen Name]

1. Is this your post? https://forum.latenightsw.com/t/indesign-server-very-slow-on-catalina/2770 If not, it's someone else with similar issues.

2. One more thought, following the other replies. I believe scripts can be in AppleScript or JavaScript. Maybe it is the AppleScript engine which is giving the performance issues. You could try taking a short critical section of the scripts and converting to JavaScript, and including timing as before. This would tell you whether a complete conversion might re-deliver the required performance. See also http://www.zzpxyx.com/posts/compile-applescript-for-performace/ "... worked for me un­til I up­graded to ma­cOS Catalina, where the per­for­mance of the scripts was no­tice­ably de­graded. I had to wait for a few sec­onds"

[Jostify]

1. That's me!

2. Thanks! I will read the post at zzpxyx. It lokks like an interesting lead.

I have been considering doing comparisons with javascript doing the same tasks, although it's not very easily done. I myself have no skills in javascript, and the javascript-guys at my workplace have little experience in indesign scripting. The applescript code used in this project contains years of development, and convering everything javascript cannot be done quickly. But some critical parts can be, as a first step and experiment. 

Anyway, this is an interesteing suggestion. Just getting a grip of the reasons for our problems is very important in our current situatio,n so that we can make desicions on how to move on. 

[Jostify]

Sorry for the misspellings. Ther seems to be no way of editing a post in this forum. 

[Jostify]

However, there is one thing that I think contradicts the suggestion that the AppleScript engine have been weakened in Catalina:

That is the fact that we sporadic recods if the script running as quick as everwith Mojave. As I wrote avove we had two such records. Yestarday we had another one, were the script took 1 minute and 3 seconds. I have no record of a better score than that, even with Mojave. But after that, that the scores went back to "normal", taking at least three and a half minute.

This tells me that the ability to run Applescripts fast are still there, but most of the time the process are disturbed by somesting. Or what do you think @Dirk Becker ?

[Dirk Becker]

Please perform a similar Sample Process on your Mojave machine.

No need to post, just tell whether there are the same dominant sections of AE* calls going thru LSCopy* calls. If not, that would confirm my suspicion that Apple has changed that "com.apple.AE" module. The AppleScript language component - those "(in AppleScript)" calls - heavily relies on AppleEvents, implemented by the "com.apple.AE" module - the "(in AE)" calls. Nothing bad, but these presumed basic and fast operations now burn all your CPU time in "(in LaunchServices)" converting data back and forth into strings and streams. If the other Process Sample has only few string/stream operations, Apple likely has converted binary transfer into that string based approach. The occasional "outputSafeString" suggests that might be considered a security "improvement".

For another test, try to use the [ tell application "InDesignServer" ] statement rather than the … id - eventually there is a check to allow local, fast (in-program) execution based on the resulting addressing mode. When I asked for the first few lines of your script, I was thinking of that, not any more complicated artifacts of your structuring large scripts.

For completeness sake I'd also try some more ways - e.g. call via process ID - there are some fragments on the web starting from SystemEvents … .

Another one, if I remember that correctly: your script can have a property "parent" used for inheritance. When you point it to the application the script will pass unhandled requests to that parent. Eventually you'll need to combine that with "using terms from application" statement.

As you suggested on LateNightSW, execution via the Apple Script Editor is also faster. This is something an Adobe engineer could follow – watch what ScriptEditor is doing differently and adjust InDesign's "Support for AppleScript" plug-in accordingly. Or they could ask Apple … Assuming anybody from Adobe watches this thread at all. So ring a few bells - Uservoice etc.

Could you also test the terminal / command line osascript? If it works the same way as AppleScriptEditor, your soap call could eventually invoke the AppleScript via command line, or you bypass SOAP and use some web server to get there …

[Dirk Becker · Accepted solution]

Another idea - what is your front process? Any changes when you tell the server to activate?

[Jostify]

That's an interesting hypothesis, and I will try all the things you suggest.

I apprecieate your help very much! 

[Jostify]

Hurray!

Telling InDesign Server to activate did the trick!
I also made Applescript frontmost when doing som calculations, but don't now yet if that have any significance.
The process time scores are now – consistently – as low as they were with Mojave.

What's your conclution from this, @Dirk Becker  ?

For us, this means we now have a workaround that saves us for the time being. And I suppose it gives Adobe a good indication of what to do. 

[Jostify]

*conclusion* 🙂

[Jostify]

It seems [property parent : application id "com.adobe.IndesignServer"] have the same healing effect!

That's an even better workaround.

For the record: What didn't make any difference was calling Indesign by name or path.

[Dirk Becker]

Given your "In the production context there are multiple InDesign Server processes running", it can only help one of them to activate. I'd also be careful with the parent approach to not accidentally cross processes - never tried that with any application that allows for multiple instances on the same machine. Eventually the phrase "current application" works here too?

With the limited analysis (no own experiments, no stepping around in assembler), it appears that Apple has tied a slow/secure path of script execution (involving LSCopy) to some criteria that is not always met by Adobe. Best approach would be Adobe engineering does their own confirmation of my theory and files a radar (Apple incident system) to end up with a solution - e.g. change the AppleScript plug-in or document our current findings.

[Jostify]

1. The activate approach actually works even when the script is run simultaneously over all the five active ports.

The process times are then 5-40 percent longer, but that's expected. And it's far better than running only one process before that change was done. So it actually seems to be enough for the scripts to tell "InDesignServer" to activate (not a specific instance of it). 

2. The parent approach wasn't stable anyway. The good effect lasted just a handul of runtimes. Or maybe it did't have any effect at all – the good scores just beeing som strange overspill from the spreceding script executions with the activate approach?

[Jostify]

And it seems to be enough to activate InDesign Server just once, when the main script starts running.

I have not been able to disturb it by clicking aroud with other apps on the server, och by running a script that activates other apps every half second. The performance stays at the same good level.

[Dirk Becker]

Have you also tried to launch the script via command line, using the osascript command?

Kris has reminded me of App Nap - ESTK had some trouble with it back in 2017. Introduced with 10.9 https://developer.apple.com/library/mac/releasenotes/MacOSX/WhatsNewInOSX/Articles/MacOSX10_9.html , but maybe Apple has recently changed a detail? To confirm, please open Activity Monitor, CPU view. With a right click at a table column you can show an extra column "App Nap". Then watch your Server during slow execution.

To change the App Nap policy, following that page you'd use the command

defaults write "com.adobe.InDesignServer" NSAppSleepDisabled -bool YES

[Jostify]

I didn't try using the osascript command, since it would demand some work to simulate script arguments. And I don't have much time over because om the deley the problems have caused. 

I did som tests this morning by inactivating the activate-workaround to bing back the slow performance. But the "No" in the App Nap column hav never turned to "yes" during these tests. 

[Jostify]

@Dirk Becker. Thank you so much for your expert help in this case! Without it I doubt it would be solved in time for us, and in any case the costs in time and effort would have been much higher. 

Now we have a workaround – that's a fantastic relief for me.

And, as you write, this propably gives Adobe a very good starting point for solving the problem permanently.