Recolor Art Interactions

So I’m really just trying to see if I’m stupid or what. Every time I try and use recolor is it such a pain in the butt. I have an artwork and I would like it to be only 4 colors. I go to the recolor tab and change the colors to 4. Now all of a sudden the Artwork is a lighter color and everything is just screwed up, the previous artwork has a base of only 4 colors to begin with so it shouldn’t have a problem identifying the colors and yet it always seems to mess it up. Not even mentioning that if you don't use the advance option setting then when you try and change one color on the wheel then the whole art work change to only 1 color even though I have the color count to 4. I think that whole system is ridiculous and as a graphic designer I shouldn’t have this many problems with that. It should be simple and basic and it just a pain. Please someone explain this or at least let me know that I’m stupid. and if your having issues wit the same thing then can we please just get an update to make recolor more viable. Thank you.