Hijacking the Learning Panel

[Silly-V]

I have done some experiments and I have great news. I have put the Application.

showLearnPanelWithContent() to the test and gained some pretty interesting results. The verdict is, we actually have the power to do very cheap and easy web-data communications using just scripting and we can also create a 'poor-man's' CEP extension by hijacking the Learning panel using a minimally-invasive but in-need-of-evaluation technique.

First , let me say that at first I did not know they would let us have so much power using this function. I thought maybe showing a public URL could be the limit. It was a test to even see if a local HTML file could be displayed. It was!

To display an interactive web-connected HTML page which can push Illustrator data to the web but not interact with Illustrator automatically, just use an empty string "" as the first argument 'manifest' to showLearnPanelWithContent().

Then, after confirming javascript is enabled and works inside the local page, I decided to see if CSInterface could work there just like in CEP extension javascript. So I copied my CSInterface.js file into the same place as my html file so it could be easily included as a script tag. To my satisfaction, it worked indeed and I was able to use CSInterface functions.

So now it's easily possible to get web data, do user interaction and send the art processing data from Illustrator back out to the web. But could we also use the web data to Illustrator itself rather than just this panel's embedded front-end client?

The next step was to see if this already promising web technique could also go all the way and function as a real CEP panel by execution an action on the back-end! This did not work though, my evalScript calls only produced "EvalScript error". However, this is where the manifest path, the 1st argument comes into play.

When set to a manifest.xml file that Illustrator somehow considers valid (I tried my own, and they did not work, maybe its their location?), the evalScript calls begin to work. I knew that this panel was powered by an extension, so I went looking for it all through the filesystem.

I finally located the Learning panel extension in the application's folder (in Windows inside of Program Files) and under Support Files/.... When I set the function's 1st argument to this extensions' manifest, the font of the displayed custom HTML page changed but I still go the eval script error. I noticed in the Learning panel (called OnBoardingForLearningPanel) extension's manifest there was a reference to the jsx file called "OnBoarding.jsx", however this file was not present in the extension folder.

By adding a file called "OnBoarding.jsx" to its expected location, the eval script error was fixed! The jsx file I created only had a sample function with an alert, but it seems like its content would not really matter because once jsx execution is enabled, all kinds of arbitrary scripts could be injected to run from the front-end to Illustrator by using evalScript with lines that eval an #include line in JSX. Such lines could be wrapped in a function wrapper and supplied dynamic arguments as well.

So there you have it:

1. It's possible to do web data fetching and pushing data from Illustrator to the web as well as rich interactivity using only JSX and simple HTML skills with no additional setup.
2. Do all the web data work and user interactivity and also automatically manipulate Illustrator all in one-go, but you have to put the "OnBoarding.jsx" file into its place first and this will require an administrator prompt.
   

 

Here is the JSX code which launches the panel into action:

#target illustrator
function test () {

	var manifestPath = app.path.fsName.replace(/\\/g, "/") + "/Support Files/Required/CEP/extensions/com.adobe.illustrator.OnBoardingForLearnPanel/CSXS/manifest.xml"
	
	var testHtmlLocation = "file:///" + encodeURI("C:/Users/.../Simple HTML Test.html");

	app.showLearnPanelWithContent(manifestPath, testHtmlLocation);

};
test();

 Note using app.path gives this technique longevity as this path may change and "Adobe Illustrator 2021" may become "2022" one day and so on.

Here is the HTML file I've made:

<html>
	<head>
		<title>TEST</title>
		<script src="CSInterface.js"></script>
		<style>
			body {
				display: flex;
				flex-direction: column;
				align-items: center;
				font-family: Arial, Helvetica, sans-serif;
			}
			body > * {
				margin: 5px;
			}
		</style>
	</head>
	<body>
		<h1>Simple Extension Test</h1>
		<input type="text" value="" id="test-input" placeholder="Enter a string"/>
		<button id="test-doc-action">Test Document Action</button>
		<p id="result-message"></p>
		<script>

			const cs = new CSInterface();
			const scriptTemplate = `function testScript (strArg) {
				app.documents.add();
				alert('Your new document is made.\\n' + strArg);
				return 'SUCCESS!';
			}
			testScript('STRING_ARGUMENT');`;

			function runScriptFromLearnPanel (scriptString, strArg) {
				const finalizedTemplate = scriptString.replace("STRING_ARGUMENT", strArg).replace(/^\t\t\t/g, "");
				return new Promise(resolve => {
					cs.evalScript(finalizedTemplate, (res) => resolve(res));
				});
			}

			function main () {
				var btn = document.getElementById("test-doc-action");
				var testInput = document.getElementById("test-input");
				var resultDisplay = document.getElementById("result-message");
				btn.addEventListener("click", function () {
					runScriptFromLearnPanel(scriptTemplate, testInput.value)
					.then((res) => {
						resultDisplay.textContent = res;
					});
				});
			}
			window.onload = main();
		</script>
	</body>
</html>​

The CSInterface.js file is next to it in its folder, it enables using evalScript among other functions. If the OnBoarding.jsx file is not there, the evalScript portion will not work. However it's still got some other functions which may be of use. Also, window.cep.fs appears to work - this suggests that we may be able to read and write or maybe even execute files from the Learn panel. All of this needs more experimentation.

On web communication: in my experience there have been only several ponderous ways to facilitate getting data to and from the internet so far. We have had Bridge and its http connection method which was working OK until a couple of years into it, it broke and after they fixed it, it broke again. There's the InDesign http object, which everyone says is unreliable and depricated or doesn't do https. And then there's writing OS-level scripts like VBS or AppleScript to do curls, and finally there's making one's own full-fledged CEP extension with all the work that goes into it. This new technique may solve a lot of these issues. Even without sending fetched data to Illustrator automatically, a copy-pasted JSON packet from the Learning panel could launch a batch of many many files. If having this ready-made data and Illustrator process would save many minutes of time of users having to go to many browser tabs to get the different bits of their data, that is a win! Being able to post process results to Google Sheets or Zapier, any API could be very useful.

Conclusion: using the Learn panel, we can enable web-data-based workflows quicker than ever and empower all these entrepreneurs who can make highly personalized and sophisticated art using this minimal yet very powerful automation. They could rig up their art process to their business process and hit the ground running. Potential benefits over genuine CEP other than satisfaction in laziness include the 'stability' as in, the Learn panel will likely be stable and updated by Adobe without any effort on your part compared to having an updated CEP extension. However, they could also change this panel any time and eventually there's going to be UXP anyway. 

[CarlosCanto]

wow!! great article Vasily thansk for sharing!

[m1b]

Yes, wow! Thanks for taking us through your research and experiment. It's full of possibilities.

For anyone following along on Mac OS, the path to the manifest is inside the .app bundle:

/Applications/Adobe\ Illustrator\ 2021/Adobe\ Illustrator.app/Contents/Required/CEP/extensions/com.adobe.illustrator.OnBoardingForLearnPanel/CSXS/manifest.xml

[Monika Gause]

Awesome!

Now we will have to lobby for enabling the Learn panel for everyone.

[GNDGN]

Thanks for sharing this experiment.

[Silly-V]

CSXS events work!

HTML Javascript:

				// set up CSXS event test:
				cs.addEventListener("LearnPanelJSXScriptEvent", function (event) {
					resultDisplay.textContent = "type=" + event.type + ", data=" + event.data;
				});

JSX event-emitting script:

#target illustrator
function test () {

	function eventTest () {
		try {
			var xLib = new ExternalObject("lib:\PlugPlugExternalObject");
		} catch (e) { alert(e) }
		if (xLib) {
			var eventObj = new CSXSEvent();
			eventObj.type = "LearnPanelJSXScriptEvent";
			eventObj.data = "some payload data...";
			eventObj.dispatch();
		}
	};

	eventTest();

};
test();

This means that once an event listener is attached inside the Learn panel's html, it does not necessarily have to obtain data from Illustrator by having to have the user push a script-executing button in the panel and have to have the OnBoarding.jsx file in place. At first, I was thinking of the most brutal way by dynamically writing the html file or some contents of it via JSX script to push that Illustrator data to the panel. Now it is evident that the CEP event listener can be set up to parse the payload string and not only get new data from Illustrator that way but also get more UI instructions such as a prompt dialog or entirely different UI to dynamically replace the existing UI.

Imagine, a batch script can run and invoke different child process scripts that are in different files and all of them can be sending data to this front-end listener to have it update a progress bar and also send the data to a remote url.

[rcraighead]

This sounds intriguing, @Silly-V ! Our AI automation solution interacts with our website constently, pulling variables from a JSON file and transfering image/vector assets. I wonder if this is something we should look at. What version of AI is required?

[Silly-V]

It has to be at least Ai2020.

Yes, with much more work than in this example, it may be possible to upload and download files. This could be a grease needed for ultimate satisfaction for some DAMs.

[Kurt Gold]

Very interesting. Congratulations on that investigation.

I don't like to be some grumping grinch, but I'm afraid that this adventure won't last very long.

Hopefully I'm wrong!

[Silly-V]

You are implying the man might come and shut the party down?

On one hand it's a very convenient way to go "CEP-lite" and it kind of implies that the Learn panel is a generic window to show stuff in, such as information about a script. On the other hand however the Learn panel may be for internal stuff only and this was an oversight. Still, and I am not sure if this will work this way, but even without the user adding in administratively the OnBoarding.jsx file, if they go to a website that has bad javascript injected or existing purposefully in its source, and it has code that does window.cep.fs they could have files written and processes executed that they didn't want. So the users who are executing JSX that sets the Learn panel will have to watch out for the contents of their JSX and execute jsxbin files only from trusted sources and developers will have to watch out for not putting in urls to bad websites in their JSX code and being careful to not put in urls to places which have been hacked or are susceptible to hacking. I guess the risk is same as making a CEP extension that replaces its UI with content from some URL - as a basic control you don't want users to put arbitrary strings in there. Hmm let's watch this space.

[Kurt Gold]

Yes, the party can be burst off sooner as you may wish today.

Let's see what may happen in the future.

[Silly-V]

Update: new findings regarding jsx/fs.

Although the most preferred and convenient way to manipulate Illustrator from the Learn panel would be through the use of JSX directly, this can only be done by putting a file into a location using elevated access. However, it is possible to do much of almost exactly the very same thing using window.cep.fs.writeFile().

I learned CSXS events are one-way messages sent from JSX to CEP, allowing us to quickly push data up from JSX scripts without any extra work. However they are only one-way and the JSX script can't wait on any response from the panel, which would have still been good for all sorts of things. So the next route is to write down text files and wait on them.
Using a CSXS event to trigger the panel into writing down a file works.. sort of - it's a twisted sort of asynchronous event whereby the code process of the panel seems to run after the JSX script is completed. So while I was using $.sleeps to try and wait on a file, it was sleeping and then writing the file, defeating the purpose.

Can there be a way around it? Probably!
It looks like the CSInterface event that is used to detect whether the panel was open or not can be leveraged to write down and keep a state of things. It was also necessary to mind this as the Learn panel does not automatically open when new content is processed within it via the receiving javascript that sets new UI content in my project -

requestOpenExtension did not appear to work for me to open it up when it's minimized to a tab. However, the menu item "Adobe Learn Panel Menu Item" in the app.executeMenuCommand fame has the ability to toggle the minimized state. It doesn't know or care which is which, so that's why keeping track of the panel open state in a text file becomes necessary.
The issue still prevailed as to how to interrupt the synchronous script to allow the Learn panel to write the text file and then to continue the script. And the work-around for this, I saw through adding a mousedown and a mouseup event in a ScriptUI dialog. Now it worked!
While this method does not guarantee success without usage of a BridgeTalk panel script or a mouseup/down modal dialog like this one, putting up a small dialog isn't that big of an issue - plus I foresee documentation help being called from the UI dialogs of scripts anyway, as a lot of explanation may deal with the script's UI.
So, while all this looks like it works so far, none of this would be any issue if the OnBoarding.jsx file is put into place and Panel-to-JSX return messages can be leveraged to show the panel when it needs to be shown.
However, documentation is fun and all - but the special treat within the treat is there is an ability to go ahead and also do JSX execution provided that a JSX script is invoking the process. A JSX can call to the panel via event to get a remotely fetched script and it can write it down and the calling script can execute it. As with all JSX scripts, it's best not to run something that you don't trust - same as Excel macros. The only way this process can be enabled is through the use of JSX, which can do a lot even faster without using the Learn panel.

Back to the powers of good, potentially (within the lifespan this technique has left- but principles of which are basic and would extend to whatever takes place of CEP), this could be used to create a quick web-fetching method for ScriptUI scripts as they can not only trigger the fetch, (which could be copy-pasted in worst case for still good results), it could write down the data for the active ScriptUI script to read a moment later and do something with.
I like this reduced CEP approach 🙂
But if it gets shut down, then I'll have some nice off-the-wall experience and will recreate it in normal CEP or later UXP. (Hopefully UXP will allow for a lot of possibilities like these :))

[Davil24]

Thanks 👍.