Green screen key has buzzing noisy edges on thin hair edges

[EditingBob]

I've done a number of green screen shoots over the years, and I always try to light the green as evenly as possibly at roughly 20-30IRE and decent talent separation from screen.  Recording at a good bit depth, eg, 10/12 bit 444.  Usually green isn't a big deal.

However, I've got a multicam Tricaster based studio green screen situation now, and while most people key acceptably, wispy hair (especially blond)  is proving to be a problem.  Maybe I've just never encountered wispy hair, but trying to key creates very noisy "buzzing" edges.  My usual trick of shrinking & blurring the matte slightly doesn't help.  The native TC2 keyers give terrible results (see sample "OG lighting TC2 key") with this kind of hair so I'm recording ISOs and keying in post.

Tech: studio is set for 1080 59.94p recorded in a Tricaster TC2 Elite using their SHQ2 codec. Panasonic ak-hc3900 and AW-UE 150 cameras.  Lighting for talent and green are 56K softboxes with 56K PAR backlights with a 1/4 magenta gel to help counter spill.  For comparison I've also recorded directly into a Canon C300 MKii 10bit 444 and a Sony FX3 at it's best color depth (forget params).  Results with those two cameras/recordings are better yet still not perfect.  

I've tried keying with both PPRo Ultra Key and Hawaiki Keyer 5, neither totally solving the problem.  Still end up with a thin edge of buzzing pixels.  In the still samples it looks like blocky pixels but in motion it's a bad buzzing.  Cam3 in sample has an angle that minimizes the wispy hair.  Cam4 shows lots of wild hair and the Ultra key version best shows the blocky pixels which buzz badly when playing.  

Been pulling my hair out (lol) over these keys - suggestions on what I'm doing wrong?  

[EditingBob]

Forgot to say I've composited these over white as a 'worst case' view. When layered on darker layers the buzzing is less objectionable, but the point is to push for as close to perfect as possible.

[EditingBob]

Here's screen recordings of analysis mode in Hawaiki Keyer showing the edge noise problem.  

[Carlos Ziade]

are you keying in premiere pro the footage that is recorded on Tricaster?

did you try to key direct footage from cameras?

Also check color spaces in Tricaster and Premiere Pro, and color management.

@R Neil Haugen  

[EditingBob]

I've recorded both in Tricaster via SDI from cam (see OP for specs) as well as direct into a Canon C300 10bit 444 (see OP).  Then keyed in PPro testing both with Ultra and Hawaiki keyers.   Color space/etc is all 1080p 59.94 422 in Tricaster chain.