Weird color flicker on dark scenes rendering animation in h264

[rotoworld]

As per the image... I'm getting this weird flicker rendering some dark scenes of my animation. Seems to be sometimes be related to movement, others to some lighter thing appearing on screen (like a dialog balloon)... So weird. If I render it in AVI, I get no issue, it's just rendering in h264.... But the purpose of this was to be uploaded to youtube, so I guess I have to do it in h264 right, because it will end up being compressed that way anyway... Also avi would be just a gigantic file.
Any idea how to overcome this?