Circle Y axis 3d rotation issue

What you are seeing is parallax, not antialiasing limits. Perspective is controlled by the position of the camera. Framing is determined by focal length. Add a camera to the scene with a long lens (200mm) and the distortion of the circles is minimized.

Move the camera close to the 3 layers and adjust the focal length to frame up the same shot and you get a lot of distortion, but if you add a vertical line down the middle of each circle and you add a centered reference line you can see that the Y-axis is still lined up perfectly.

It is the change in perspective caused by the position of the camera that is making your circles look like they are tilting. Put a vertical pattern in the circle and a circle and use a normal lens (50mm) and the circles will not look like they are wobbling on the X-axis.

Just in case you are interested, here's the project file. This would be a good subject for a tutorial.

Nothing wrong here. You are simply reaching the limits of AE's antialiasing algorithms. Lines at specific angles such as on your circle when it's almost rotatted 90 degrees cause the sampling to not cover a single full pixel, hence it ends up being a mix of semi-blended pixels and wrongly clipped opaque pixels. That's why 3D programs usually have much more advanced antialiasing and image reconstruction algorithms as well as a choice of multiple methods. where AE is concerned, you should start by making your strokes thicker to minimize such issues. Furthermore you should enable motion blur, even if it's only with a very short shutter phase. A minimum of blur goes a long way. That being the case, also manually softening edges ever so slightly with a tiny bit of Gaussian Blur or similar goes a long way. That and of course re-considering your design decisions: Too skinny lines are generally not good for video work and intense color contrasts with large uniform areas will cause further artifacts when using compressed outputs, anyway. Consider using more gradients, subtle shadow and glow effects and so on. You cannot overcome the technical limitations of AE's renderer, but you can disguise them and use better design to mitigate soem of the issues.

Mylenium