detect if we are editing an opacity mask via scripting?

There is likely no direct way to determine whether the user is currently editing an opacity mask.

However, while not perfect, the window title or activeLayer name may change to something specific to opacity mask editing, allowing you to infer this state.

-- Pattern 1. Guessing from the window title
on run
	tell application "Adobe Illustrator" to activate
	
	tell application "System Events"
		tell application process "Adobe Illustrator"
			set window_title to title of window -1
			--> `${document_name} @ ${zoom} % (${path_name}/${opacity_mask_in_user_language})`
		end tell
	end tell
end run

// Pattern 2. Guessing from activeLayer.name
var doc = app.documents[0] ;
var layerName = doc.activeLayer.name ;
// --> `<${default_opacity_mask_name}>`