Stabilze motion based on face mask

Use Mocha AE instead of the AE Feature tracker. You'll get a lot better results. If you invert the tracking data from Mocha and apply it to a null, then parent the original footage to the null, the motion will be stabilized.

Here's the complete workflow:

1. Trim the layer to include only the part of the footage that needs to be motion tracked
2. Pre-compose the trimmed layer, moving all attributes to the new comp and trimming the comp to the layer length
3. Temporarily set the Main Comp resolution to Full and add a null
4. Apply Mocha AE to the nested comp (your pre-composed footage layer)
5. Drat a spline around the part of the face you want to stabilize like the forehead or the chin
6. Set tracking to Translation, Rotation and Scale only
7. Increase the percentage of pixels used to about 90%
8. Motion track the plane of the face
9. Save Mocha and return to AE
10. Check the box that says Invert and collect the tracking data in Mocha AE
11. Set the target to Null and the output to Translation (not Corner Pin)
12. Apply the tracking data to the Null
13. Parent the footage layer to the Null and the surface you tracked is not fixed.
14. If you need to reframe, add another null, parent the first null to the second, then move the second null to frame up the video

There is also a very easy way to add that motion to another layer and return the movement to the original footage by simply moving to the first frame, releasing the parent for the footage and parenting the new replacement layer to the null.

I hope this makes sense. I have a tutorial coming soon on the process. I use it all the time to help with everything from Rotoscoping to adding 3D elements to a scene.

Not easily. You would do some fancy dance with mask-based expressions or convert the mask points to Nulls, average the position, calculate the actual rotation and sclae (where applicable), invert that and apply it. Depending on the shot this could be a lot of work, as your mention of the normal tracker not delivering good results clearly indicates that there are things going on that screw with it like high frequency jitters, zoom pumping or whatever, all of which would have to be taken care of.

Mylenium