Hello, I've run into this a number of times and I haven't been able to narrow down the issue.
Essentially, I'm attempting to use the pen tool to add a mask to a layer from the composition window. On first click, the pen tool creates two vertices on top of each other and completes the mask. When I try to click on the next point I want my new vertex to exist at on the mask, I get the "pen tool add vertex" icon. This tips me off to the weirdly completed mask that was just created.
I am able to create masks inside the layer window, but for me, it is easier to just do them inside the comp window.
This is a consistent problem I've run into, but it doesn't happen on all of my projects, only randomly.
I am running Windows 10 and the latest version of AE (and have had this issue on previous AE installations too).