Hi,
So as per the contract of the doScript method, the first argument can be a File, String or a JavascriptFunction. See the details at
https://www.indesignjs.de/extendscriptAPI/indesign-latest/#Application.html#d1e42173__d1e45616
Now lets take your first use case
app.doScript(String(main()),ScriptLanguage.JAVASCRIPT,[],UndoModes.ENTIRE_SCRIPT,'main');
In the above you have called the method main by using the statement main(), so it will be called and the return value of it would be passed to the String constructor and the resulting value is then used to call the doScript method. Seems your method does not return anything in which case it would be undefined and the above call would be equivalent to the following when the doScript method is called
app.doScript(String(undefined),ScriptLanguage.JAVASCRIPT,[],UndoModes.ENTIRE_SCRIPT,'main');
The String(undefined) would return a string undefined and thus no code was actually called by doScript and hence you don't see any undo entry in the Edit menu
Now in your second use case
app.doScript(main(),ScriptLanguage.JAVASCRIPT,[],UndoModes.ENTIRE_SCRIPT,'main');
The main method is called and its return type is then used to call the doScript method. This is equivalent to the following when the doScript method is called
app.doScript(undefined,ScriptLanguage.JAVASCRIPT,[],UndoModes.ENTIRE_SCRIPT,'main');
Thus the error that pops up makes sense.
Correct way of calling the doScript method. We generally just use the method name without the invocation braces. So the following should work good
app.doScript(main,ScriptLanguage.JAVASCRIPT,[],UndoModes.ENTIRE_SCRIPT,'main');
If we need to pass the arguments, we can also use the string form as follows
app.doScript("main()",ScriptLanguage.JAVASCRIPT,[],UndoModes.ENTIRE_SCRIPT,'main');
Hope this clears out your confusions.
-Manan