When I subtract front (after converting text to outlines, expanding the outlines, working strictly with vector shapes), I have been running into a situation where Illustrator is adding in these extra colored areas, that I am at a loss as to figure out how to eliminate them.
The left image is before subtract front has been applied. Right image is after.
Try View > Preview on CPU
That appears to have eliminated the additional fill.
Preview did it, also tried the Simplify method, also worked, although it did change the art a little (as that function was designed to do).