How to reduce pixelation for best quality in After Effects

Video is pixels, your layers must be at or near 100% scale or 100% effective scale if they are 3D layers, vector artwork should be continuously rasterized, and you should use the Adobe Media Encoder and one of the presets for h.264 encoding of the final video that is to be released to the public. On occasion, you can slightly improve color compression artifacts if you choose Multi-pass rendering, but the data rates for the H.264 presets match the YouTube and Vimeo standards so you'll get the best results on streaming platforms if you use the presets and don't fiddle with them. All video for distribution to the public is going to be 8-bit and the standard is h.264. Any other formats and/or data rates are going to re-compressed by the streaming services with a sledgehammer. The closer you stick to their standards, the better the quality of the final result. Make sure you stick with standard frame sizes and frame rates. 

If you are using the Render Queue/Output Module to create digital intermediates you should stick with the default templates or create a new one using your preferred mezzanine codec. I like GoPro Cineform because it is free, universally readable, supports Alpha Channels and Trillions of Colors. 

As long as your layers are at or near but not much more than 100% scale, thin vector lines are perfectly lined up on the pixel grid, and you don't resize on render, AE will produce pixel for pixel in luminance and give you very acceptable color and temporal compression using the AME and the presets, and your DI's with a suitable mezzanine codec will be as good as they can get.