How to select color groups of pixels over a certain size?

Hi there! I am new to Photoshop but have scourged the web for weeks trying to find this answer with no luck. What I need is essentially the wand tool, but with the added ability to deselect groups of pixels that are small (say <10 pixels total) of that specific color. 

For example: I have a somewhat posterized looking image and used the wand tool to select all black pixels, then cut them to a new layer. Now on this new layer I have a few small scattered groups of transparent pixels within the black I need to fill in, and a few small scattered groups of black pixels in the transparent area I need to delete.

Normally I would do this by hand, but I have up to 20 colors to repeat this with and the fact that I only want to eliminate smaller groupings makes it take a very long time by hand. 

Any tips to speed this up? Thank you SO much in advance.