Your post isn't really useful, since you haven't told us anything about how the circle is actually created. Is it a shape layer? Some effect? An imported graphic? You need to be more specific. That and of course screenshots are better than describing the smell of invisible flatulence...
I've attached a screen of the circles with smooth edges and another screen of the circles with jagged edges. They switch to jagged when the layer is switched to a 3d layer. The cirlces were imported from Illustrator and converted to outlines. I was aiming to experiment with the 3d orientation of the circles.
Some of the other layers which are hidden here have wiggle effect.
Looks like there is some conversion issue with the paths not getting enough intermediate points similar to AI's crappy DXF/ DWG export making a mess of things. If it's an option, you might consider rebuilding the shapes from scratch using parametric circles in AI. Otherwise subdividing the paths in AI should give better results.