Adobe Media Encoder not rendering keyed video correctly

[A.I.1]

I've keyed a video using the Keylight keyer. In After Effects with the RAM preview the video looks acceptable (yes if you look closely you can see some noise/artefacting but it's basically acceptable). But when I say "Add to Adobe Media Encoder Queue" and render it with the H264 codec from within Adobe Media Encoder, the colours look very different and it renders much more blocky. The preview in Adobe Media Encoder also show the colours looking very different and more blockyness - see attached screenshot. Since in the Keylight effect in After Effects I have it set to "View->Final Result" surely the result should be almost the same as how it should look when rendered, even if I have some other setting it it not optimal..

Note it renders very blocky and with the wrong colours (when rendered with H264) even when I set the Target Bitrate to 20 Mbps, Max bitrate to 22 Mbps, and "Render at Maximum Depth" checked and "Use Maximum Render Quality" checked and when I select "VBR, 2 pass". This is with AE CC 2017 and AME CC 2017.

Note the main keyed area looks okay (not blocky) but it's outside the main keyed area (the vast majority of the frame) where all the blockiness appears on render using AME as well as previewing in that program.

[angie_taylor]

H264 is a codec and will compress your video. What are you using the final video for? What will it be shown on? It may be that H264 is not the best codec to use for your purpose?

Also, what file type is the source video? Is it a Quicktime Movie?

Finally, what are your Color Settings in AE? Can you take a screen shot of the Color Settings panel and load it up here please?

[A.I.1]

H264 is a codec and will compress your video. What are you using the final video for? What will it be shown on? It may be that H264 is not the best codec to use for your purpose?

Also, what file type is the source video? Is it a Quicktime Movie?

Also, though I'm compressing it to H264 even when I set the bitrate high (20 Mbps) it still renders a blocky video with different colours to the original.

I'm replacing a "green screen" in the video with a made up screen (basically a made up mobile phone app - which is a composition (pre-comp), and I've tracked the "green screen" in the source video and perspective-corner pinned my pre-composition onto the "green screen" in the video.). Note: the person who sent me the source video with the "green screen" in it hasn't specified what it will be used on, but I assume it will be shown on the web. Also, H264 is the codec I normally use for videos I do for people.

The extension is .mp4 but it has "mov" just before the extension as the file name.

This is what MediaInfo says about the source file I'm using to key:

General

Format : MPEG-4

Format profile : Base Media

Codec ID : isom (isom/iso2/avc1/mp41)

Overall bit rate mode : Variable

Overall bit rate : 4 959 kb/s

Movie name <file name here>.mov

Performer : MacX Video Converter Pro

Genre : Video

Writing application : Lavf56.40.101

So the above "movie name" ends with ".mov" but the file name on the disc ends with the extension ".mp4". Note: I'm using Windows 10 and I'm using a source video someone sent me to key/composite.

These are the colour settings this project is currently using (8 bit colour):

I don't need to alter that "Match Legacy" checkbox do I? But shouldn't Adobe Media render/preview approx the same (colour wise etc.) whatever the project settings in AE were (especially if I'm "display final output" in the KeyLight setting).

[A.I.1]

Note: I've replaced the Keylight effect with the Color Range effect and now the rendered/previewed output from Adobe Media Encoder looks okay (looks approx the same as the RAM preview from After Effects, without all the added blockiness). So that's solved it for this video but not the issue itself, ie. it should still be able to work correctly with the Keylight keyer too.

[Kevin-Monahan]

Hi A.I.1,

If you'd like to troubleshoot this issue further, I'm willing to try. Can you try exporting as an image sequence or lossless codec with the alpha channel intact? I'd be interested to see that comparison to the H.264 export.

Thanks,

Kevin

[A.I.1]

Hello. Thanks for your reply - it's made me figure out what the issue was.

I just tried setting it to a lossless codec (.PNG) in Adobe Media Encoder (CC 2017) and also keeping the alpha channel and in the preview in AME it still looked blocky.  When rendered to the PNG+alpha and then one of the frames loaded into Photoshop Elements it seems the colour difference and blockyness is because of the transparency.

In the Adobe After Effects composition settings I had set the composition background colour to a shade of yellow. So what was apparently happening was the bits of that colour were showing through (or partially) in the After Effects preview, but when I changed the composition background colour to black (RGB 0,0,0), the look of it looked the same as in Media Encoder and the blockiness was the same as in Media Encoder (ie. it looked more blocky than with the shade of yellow as the background). So basically while the yellowish composition background colour had been visibility affecting the image, it must be that because it was a more similar colour to the pixels in the video being keyed (than black) it disguised the blockiness more (less visible blockiness).

When I used the older version of After Effects I was used to rendering from within After Effects most of the time rather than using Media Encoder to render and rendering from After Effects used to render the composition background colour within the rendered video file - without you needing to add a new solid or something to make the background a different colour (so with the old way at least the rendering would have looked about the same as within AE), whereas with Media Encoder (at least the current one) it doesn't keep the composition background colour you specify in AE.

So at least now I know what it was for in future, so thanks a lot for that (for the particular project like mentioned above I had replaced it with the colour range keyer (and also used advanced spill suppressor) so the final project in the end wasn't affected by this issue I was having with the Keylight effect (and with AE previewing to a different background colour than AME).

[Roei Tzoref]

If your background is showing through your keyed footage (through the parts that are not supposed to be transparent), it means your key is not optimal. You can view your key in screen matte or status mode to see exactly what you got. Also alt+4 will show you alpha. Some also use a red background to see exactly what you got.