Face Tracking extremely slow?

I've found with the regular tracker that if I pause the track when it starts getting slow and then start it back again, it's fast again. I haven't messed with the face tracker much, but try it and see if it does the same.

Since multi core processing has been developed AE has never been able to use all system resources to process a single frame of video. There used to be an option to render multiple frames simultaneously but that did not work unless your comp used a limited number of effects and features. It was very easy and common to create comps that took longer to process with that feature turned on. Most common video compressing codecs also did not support rendering multiple frames at the same time. They wanted the data one frame at a time. RMFS worked best when you rendered image sequences not video. It's the same way with network rendering. If you have that option image sequences are the way to go.

Adobe's current efforts seem to be focused on utilizing the GPU more than the CPU so I may be a while before all of the computer's processing power can be used. The problem is that a frame of video, even a 4K or 8K frame just is not very big and AE looks at and processes one frame at a time. It's pretty hard to fill up 12 cores with the data from one frame of video. That is an oversimplification of the problem but it gives you the idea. It's not an easy problem to solve and there's been some pain along the way.