Sorry to hear you are having trouble with color palette in XD. It looks like you're facing the similar issue mentioned here:https://community.adobe.com/t5/adobe-xd/panel-color-picker-picks-up-the-light-color-in-adobe-xd/m-p/... Could you please try the steps mentioned in the correct answer of the post and let us know if that helps?
I don't have SRGB in the color settings of my monitor.
Look at the image below
Also, the color picker works fine in photoshop, lightroom, illustrator, paint, and any other application.