Using "select same fill color" after using Recolor Artwork, doesn't select all like colors

[curtl20104762]

My original artwork has over 14,000 colors.  Many of them very close. 

I used Edit / Edit Colors / Recolor Artwork to try and refine all of my 14,000 colors in the original document down to 5. 

I was able to do this by incrementally lowering the number of colors.

I created a new color group from the 5 resulting colors.  And this group did show up in my Swatches panel.

I save out of Recolor Artwork, closing the panel.

Now, what I want to do is be able to select all of the pieces of the artwork of one color. 

I have ungrouped all the pieces in the document.

I tried to select all of the "green" pieces, for example, using Select / Same / Fill Color, expecting to have every shape in the document that had that green color selected.  It didn't work.  I only get a handful that will select.

What am I missing?

Thanks ahead of time.

[meganchi]

Can you show what your color swatch palette looks like with unused colors deleted out?