Reducing colors via indexed color and color table.

Using indexed color, and selecting colors from the image - I get mostly black and very few pixels of selected color. My goal is to reduce the number of colors in the image by selecting them.

I am on a mac

Photoshop 23.2

I am not doing something correctly, but I don't know what because sometimes this works fine.

Help ?

Thanks