Pen Tool not creating Masks correctly

[kspangen]

I'm seeing a strange issue where I try to draw a layer mask with the pen tool, and instead of adding a path point where my mouse is, the new point is on top of the first one, and it's given handles that reach where the mouse is.  I have a layer selected, I definitely know how to use the pen tool, and I can create shape layers just fine. The issue is drawing a mask on an image layer.  I can use the polygon tool and make a square, and add and move points, to get the desired mask but that's just an awful solution.  This problem has occurred over many projects, and I've restarted my computer and the application several times.  I just updated AE as well.

I have no idea why it's acting crazy or really even what's happening and would LOVE some assistance.(the cursor wasn't captured but it was above the first letter on the right, where the handle is)

[Rick Gerard · Accepted solution]

I am assuming that your OS and AE version are up to date and nothing changed between the working mode and the problem mode except something in your comp.

Try adding a new comp with only a new solid. If it does not work follow the reboot, relaunch and try again in a new project with nothing but a solid in the comp. If that works, the problem is probably solved, if not try trashing preferences. If the problem still exists then check your mouse drivers, GPU and everything else.

If the new comp new solid works but you are still having problems with the problem comp then reveal all modified properties of all layers in the comp and start turning things off until you find the culprit.

[kspangen]

Thanks Rick. This is across multiple comps/projects. It seems to be only on image layers, my video layer masking is ok.  I'll try dumping my preferences.  I can double-click the layer and edit the mask just fine in the layer panel, so there's something weird going on in the comp panel

[Rick Gerard]

just for fun, you might want to try toggling the pen tool to rotobezier and see if that works. Anything common about the image files that are giving you problems?

[kspangen]

Well since I have been masking in the layer panel as a stop-gap solution, I haven't had enough time or emotional bandwidth to troubleshoot this further, since I have too many projects with short turnarounds and my patience is pretty thin at the moment.  I'll try to see if I can nail it down and post any findings for later reference.  Thanks for the help, I've never really done the rotobezier, I have to read up to even know what it does!

[Kevin-Monahan]

Kspangen,

I just tested and had no issues with masking still image files with the pen tool. Are you still having this issue?

Thanks,

Kevin

[kspangen]

it's been inconsistent. I'm pretty sure it's an issue on the affected projects, but I don't have time to go back and troubleshoot it anymore so I'll call it good I suppose

[rosieposy]

Hi. I know this thread is old but thought I'd put this here in case it helps anybody. My set-up is not the same as in the original post but I've been having the same issue. It has turned out to be specifically due to scaling. The layer on which I'm drawing the path is parented to a null that scales everything for an animated zoom-out. I hadn't realised that this has resulted in the path layer being scaled down to only two percent. I think this means that, although the vector points look far apart to me when I'm zoomed in, they're actually very close. The pen tool behaves normally when I first create the layer but it starts behaving oddly after a while, even after trying all the usual fixes and two different versions of AE. My workaround is just to draw the points further out and drag them into place. Not ideal but I'm getting through it and will avoid ever being in this position again!

[kaiser-wb]

Hi. 
I'm also having the same issue, and have had it intermittenly on verious projects. 

Thank Rosieposy for the tip about scaling. Mine was a little different however. I unparented the layer from the null I had it parented too but that didn't work. I also had some scale keyframes on the layer to be masked. The only way I got the pen tool to work normally is by deleting the scale keyframes... not ideal at all. 
The best work around I found was to cut the scale keyframes, then drow the mask, and then paste the scale keyframes back in. 

I wish Adobe would fix this, I've been having this issue on and off since Ae 16 and on. 

[maikm92830163]

Hello, I had this problem today, it seem I was trying to mask a layer that already has an animation (scale in my case), and the pen tool was doing the same that you describe here, so my solution was to delete de animation, then mask and then, animate again.

[JJ Umana]

Just had this bug with the latest update... and deleting the animation lets me draw the mask with no problem.

[sanidays]

I had the same issue just now on an image layer with a mask. I deleted the scale keyframes and it's working again, thanks everyone!

[charleh]

Just had this issue as well, used the same fix as others (remove animation apply mask, reapply animation). This should be fixed!

[TheRossle]

Just had this problem today for the first time in 7 yrs of AE experience.

My setup:

Comp layer with animation applied to scale property.

Mask on the layer, mask edit mode active in composition panel, selection tool in use.

Click and drag with the mouse

Expected Behavior:

Maquee selection resulting in newly selected mask vertices

Actual Behavior:

Mask edges or corners nearest my cursor are automatically selected and dragged

Very suboptimal workaround:

Delete scale keyframes and replace them AFTER adjusting my mask.

[sncfrk]

Another day, another unacceptable bug from Adobe. I just started having this issue for the first time as well, of course on an important project with a tight deadline. The crazy thing is at this point, EVERY single day that I work in After Effects I encounter a brand new bug that brings my day to a grinding hault, often times several times a day. I can't believe software this popular and crucial to so many people is so unusable at this point. And it continues to get worse and worse and worse. I must be at risk for being the community ranter at this point, but this is seriously so unacceptable and so uncessesarily stressful!!

[sncfrk]

just to add something slightly constructive, I can confirm that in my experience with this pen tool issue my large video layers are scaled down to 2% and are parented to a null for tracking purposes, so it checks out that this bug is related to using the pen tool on layers that are scaled to small sizes. But along the lines of what OP mentioned, this is pretty damn normal for professional compositing when you're working with 2k 4k 8k etc files in a 1080 comp. 

Adobe needs to start making tutorials on learning how to deal with all their bugs, because they never get fixed and at this point, knowing their software is 80% bug workarounds and 20% interface usability.