h264 has always turned out blocky red edges from AME but now it seems even worse than before. Although the blocks look smaller, the sharpening effect produces rough, dark edges which are now clearly obvious no matter how high the quality. Is this a problem across all .264 encoders? Is there a remedy? I'm sending to AME CC2017 from AE CC2017.
I have this same issue. The anti aliasing on red edges seems to fade to a dark red instead of into alpha. When this is rendered out the red gets that dark edge to it.
As you are sending from AE 2017, I wonder if there isn't an unrelated issue causing you some pain. A bug was introduced in the CC2017 version where AE comps queued to AME would export at whatever resolution the comp was set to instead of defaulting to always export at Full Resolution as it has in the past. Could you please verify that your comp's resolution setting is at Full and try the export again?
I noticed the bug the very time I used AME17. A comp in AE viewed at 1/4 resolution would render at 1/4 the size it should.
I'm my case I'm rendering at full resolution (comp view and settings in AME), I've upped the bitrate to 100mbps in AME and still getting the red anitialiasing issue, though not as bad as say 10mbps. This definitely seems to be an issue with H264 though. I'm getting similar results in squeeze when converting an uncompressed QT I have previously exported from AE.
I'm at full resolution and aware of the AE comp resolution bug Thnord, thanks for pointing out. Interesting Paul you have similar results with Squeeze too. As you say this points to an issue with the .264 codec. Perhaps there's a certain amount of sharpening applied during .264 encode which is seemingly amplified on red edges.