How to add more free transform points into a mask in after effects?

Free Transform is always a rectangle. You can select multiple points and end up with smaller rectangles, but you can't add points to the transform tool. There is a solution.

Use the pen tool and the Alt/Option and Cmnd/Ctrl modifier keys to switch between tools. There's a lot of good information in the help files if you just use Search Help in the top right corner of AE. Here's a short tutorial on editing and animating masks that should get you started.