1080Ti doesn't work

With newer versions of After Effects, you should not mess with the supported GPU txt file. There is a single tick box to "Use unsupported cards" in the GPU information area. If that doesn't work, neither will editing the txt file. In fact, it's more likely to work than messing with the txt file.

Anyway, it might not work at all with that newer card since the ray-traced renderer works on the Optix library and NVIDIA's been changing it, but After Effects hasn't. Here's what you can try. Rename the OptiX.1.dll from the After Effects program directory to Optix_old.1.dll .  This is typically the C:\Program Files\Adobe\Adobe After Effects CC 2017\Support Files directory.  Then install the latest version of OptiX from the NVIDIA website (https://developer.nvidia.com/optix ).  After doing these two steps, the ray tracing functionality in After Effects should be functional.  Note: After removal of the OptiX dll from the After Effects directory the CPU ray tracing support will not work.

Now, all that aside, the ray-traced renderer is considered obsolete by Adobe. The After Effects team is no longer developing it (thus why some newer cards don't work with it). Their focus for 3d is deeper integration with Cinema 4D (thus, the new Cinema 4d renderer you can pick instead of the ray-traced renderer).

Also, the new, GPU-accelerated native effects (Lumetri, Fast Gaussian Blur, etc.) use entirely different technology (and should work on your new card, no problem).