How can I convert a layer to Smart Layer in Photoshop using AppleScript?

I haven't checked the AS scripting reference, however, both AS and JS DOM coding is limited. What the DOM doesn't cover, one can usually achieve using AM code (Action Manager).  The most accessible form of this is recorded by the ScriptingListener plug-in.

AppleScript can call "inline" code or reference an external script file to perform tasks not directly covered in the AS DOM.

The AM code to convert the active layer to an Embedded Smart Object:

executeAction(stringIDToTypeID("newPlacedLayer"), undefined, DialogModes.NO);

The AppleScript code:

tell application "Adobe Photoshop 2024"
	
	set js to "executeAction(stringIDToTypeID(\"newPlacedLayer\"), undefined, DialogModes.NO);"
	
	do javascript js
	
end tell

Notice how the " double-quotes in the original code have been escaped \" with a backslash as the JS code is contained within open/close double-quotes in the AS code.

Another option rather than escaping the double-quotes, is to use single-quotes:

tell application "Adobe Photoshop 2024"
	
	set js to "executeAction(stringIDToTypeID('newPlacedLayer'), undefined, DialogModes.NO);"
	
	do javascript js
	
end tell

If you wanted to reference an external JSX file:

tell application "Adobe Photoshop 2024"
	activate
	do javascript of file "~/Desktop/createSO.jsx"
end tell

or

tell application "Adobe Photoshop 2024"
	activate
	do javascript (file "Users:username:Desktop:createSO.jsx")
end tell