When applying preset with masks, masks are added, not replaced

This is as designed. Now that masks are kind of universal (you do not add a graduated filter any more, you add a mask and the first component of that mask defines what it is), it makes sense that adding a new mask by using a preset should not delete any existing masks. In fact, this was also an old feature request: many people asked Adobe not to erase existing filters when adding a new one with a preset. You could file a feature request under 'Ideas' to have this as an choice when you create the preset.

I understand how to use the latest masking tools.  I'm asking specifically about presets.  In previous versions of Lightroom Classic, applying a preset with a mask would replace the existing presets.  In version 11, the masks in the preset are added to the list of masks that already exist.  This feels like a big to me unless there's an option somewhere to revert to the old behavior.

Note that when syncing Develop settting between two photos that both have masks, Lightroom brings up a dialog to ask if the masks should be replaced or merged.  Why does that option not exist for applying presets to photos that already have a mask?

From your statements, what I suspect is you are creating a mask, selecting a brush for it, then  adding additional brushes treating them as separate masks, as opposed to building on the mask. What you may be missing, is creating a new mask.

And some tutorials, first one is long but inclusive and probably the best:

https://www.youtube.com/watch?v=Y7Fld8eEHsE

https://www.youtube.com/watch?v=tzPWzJQsYTI

https://www.youtube.com/watch?v=69MMxR3MGW8

At the Adobe Support Community, in Lightroom Classic, Under Discussion, right near the top, notice the document link that is ALWAYS provided when updates/upgrades are released. Here:

I have encircled the link in the screenshot above, go in the actual web page, and click on that link

Notice the link within that document that I encircled, on the actual web page, click on it

That brings up a document Adobe ALWAYS posts at updates and upgrades. Several articles may be included. Especially at upgrades. Notice that I circled two, you will want/need to bring up and read both. For now, for the subject at hand, on the actual web page, click on that one circled in red, the one about Masking.