Loss of detail after baking high poly to low poly and exporting to Unity

I started learning Blender, Substance Painter, and Unity and have a very basic low poly model that I UV unwrapped and baked a high poly to it. I then painted random stuff on it in order to test my workflow before trying real models I want. With this in mind after doing this and using Unity Universal Render Pipeline (Specular). After doing this and applying the texture to my model I noticed the different colors besides the skin are gone and I lose all of my detail and baking. I am pretty new to this process and mainly program so sorry if I am missing something obvious. I've been trying different things for 2 weeks and am still stuck. I attached the image of my model in Substance Painter and Unity for reference.