Wave warp pixelation/stepping

Maybe not the best solution, but I duplicated the wave warp effect, halved the wave amount and offset the phase slighty. This essentially halved the amount of stepping I was experiencing. You could possibly do this a few times until the problem isn't visible?

To get smooth edges, you'll need work around the hierarchal rendering order of After Effects.  In this case, it's the Wave Warp effect rendering after the Shape Layer rather than before.  Reminder: the hierarchal rendering order is reflected in the order we see items stacked inside a layer, so Contents happens before Effects which happens before Transform.  It would be great if After Effects could speak up and say, "Hey, for what you're trying to do, you'll need to tell me to do it in a different order."

Add a Fill to your Shape Layer and hide the Stroke.

Make sure that the Comp Frame Size is large enough that the Shape Layer is not cropped by any of the edges of the Comp.  If needed, expand the Comp Frame Size such that the Shape Layer never clips.

Then, while the Shape Layer is selected, choose Layer > Auto-Trace.  The settings pictured below should get you a close match to the warped shape, but of course feel free to adjust them.

The most important setting is "Apply to new layer".

How long it takes to complete depends entierly on how powerful your computer is.

Now apply the Stroke effect (Effect > Generate > Stroke).  Be sure to set "Paint Style" to "On Transparent".

Now that the Wave Warp is effectively happening before the Stroke, you'll see a smooth, anti-aliased edge.

@adobe

We are now 3 years later and this problem still exists!!!

half the world uses your programs, and pay for them  SOLVE IT ALREADY!

please

I'm not clear what you are expecting. Your wave width is set to 2300 pixels, which is simply insane and presumably wider than your comp. Of course this will cause quantization issues. Those effects aren't meant for producing these kind of large graphical waves. If you need that, you have to use other techniques and effects. Switching the project to 16 bpc might improve things a bit, but even that may not suffice.

Mylenium