Overlay not working with brush tool

Thanks! I got it

Do you understand how blending in overlays works.  I use overlay mode all time when I work on layers mask with a paint brush.  For that mode will protects the white and white areas.  You can only  paint and change the layer mask gray areas.  Once an area turns 100% Black or White that area can no longer be changed Painting White on Black or Black on White will do nothing.  Your Painting a blending mode how colors blend is not the Same color you painting with. The color is blended with the color on canvas and the blend mode you are painting in.

Here I painted a green + a yellow x and a red z over a rainbow layer in normal mode.  Had I painted in Overlay mode it would look like the lower image .

It depends on the underlying pixels. Overlay won't do anything if the pixels you're painting on are white or black.