After Effects painfully slow on 8700k, 64GB DDR4, GTX1080ti

Many of AE's processes are still single-threaded, and as you say, the GPU isn't utilized as much as the CPU, although each new release adds more GPU-accelerated rewrites of effects.

The RAM will be used as needed, so you shouldn't expect it to jump to 100% consumption, but I'm afraid that your 8700K's base clock speed is actually slower than the 7700K, The 8700K's base is 3.7 GHz, whereas the 7700K's is 4.2 GHz, which is due to Intel adding to more cores to the same size chip as the previous generation (also, you only have 6 cores, not 12. HyperThreading will count the other 6 as threads, but not actual cores, so if a process isn't built for HyperThreading, then only physical cores count). The Turbo Boost is faster on the Coffee Lake, and I would have expected that to compensate for the clock speed drop, but I wonder if this is part of the problem. The Turbo Boost speed is only for a single-core, so it would make sense that AE would utilize that, but AE also doesn't peg your CPU at 100% for many tasks.

Check out this Puget Systems article which compares the 8700K to the 7700K, and you'll see that it's nominally faster: https://www.pugetsystems.com/labs/articles/After-Effects-CC-2017-2-CPU-Performance-Core-i7-8700K-i5-8600K-i3-8350K-1055/

All this being said, you didn't actually mention what it is you're working on, how you're determining that the new machine is 4x slower, what format your footage is (if you're using footage), what version of AE you're using, and other details about the actual project. That info could shed some light on where the bottleneck is.