Here's an issue I've been having for way too long now, and my long searches for an answer have been futile.
Let's say I draw a mask on a layer. If I need to move around the vertices of that mask for either tuning or keyframing, sometimes when I simply click the vertex, the mask mode will change. If I want that mask mode set to "Subtract," for instance, and I click the vertex, suddenly the mask mode switches to "Lighten" or "Intersect" or any other mask mode rather than staying put.
I've tried restarting AE, restarting my computer, uninstall/reinstall, cleaning caches - I've run the whole gamut and nothing has permanently solved this horrendously frustrating issue. If anyone else out there has come across this issue and solved it, I will be forever in your debt!!
I tried reproducing it with the current AE version but I don't see any mask mode changes. Any chance you can capture a video of it or put an AEP project file up some where I can download to try to repro it?
Here's a GIF that I just captured that shows you what I've been dealing with.
I can't see exactly what you're doing except that you're moving the mask while it's in Mask Transform mode PLUS you've got the Toggle Pixel Aspect Ratio correction enabled - chances are quite high that this should be left disabled as it's more of a legacy feature than something most users use today.
Back to your issue, I tried reproducing the issue and I can't reproduce it. If trashing your AE Prefs doesn't help then you need someone from Adobe Support to have a hands-on look at your issue.
I made a mask around a layer, and when I tried to adjust any part of the mask's path, the mask mode switched.
Unfortunately this issue is completely spontaneous and does not happen to only specific project files - If I restart my machine, the issue may not occur for an hour or a day or even ever again, or it may happen repeatedly.
Maybe it is related to the mouse/tablet driver. Could you try with another mouse or update the driver for your graphics tablet? I know that while creating a mask, you can change the modes using the S, N, F and A keys. But it has no sense that clicking on a vertex, the mode changes automatically.
Have you tried trashing AE's Prefs files? Uninstalling may still retain corrupted Pref Files. And, installing may also retain corrupted Pref Files. So, trash the Pref Files and update us on this issue.
This did not work, unfortunately. No changes at all.
Standby, currently dealing with a separate AE issue ("unable to allocate memory" alert that' not allowing me to save).
Definitely something is happening with your After, I tried to replicate it but by my side everything is normal, it does not change itself. You could try uninstalling Ae with the Creative Cloud Cleaner Tool and then reinstalling it.
If this does not work for you, contact Adobe Support as suggested by Roland.
This is unexpected behaviour. Your best bet is to file a bug report. Please take the time to report this as a bug. You will need to supply full system specs and steps to reproduce. Screen captures/recordings are always very useful too.
You can file the bug here: After Effects: Hot (1662 ideas) – Adobe video & audio apps
I'm having the same issue. Except it just decided to start happening for no reason. Anytime I make ANY adjustment to the mask, the mask will change by itself from subtract to intersect. This is maddening that I have to switch it back EVERY single time I touch this mask.