Bake and Texture Shade For Modular Character

I'm making my first modular character, who is currently nude and made up of 5 objects (Head, Torso, Hands, Legs, and Feet) and all share the same material.

When importing the low-poly mesh into substance the head was a different shade to the rest of the body, this was also the case in Maya but after some research I found the issue to be related to the vertex normals. After averaging the normals this resolved the shader issue in Maya and in substance, at least for the low poly.

I then baked the details onto the low-poly model and noticed the shader issue has appeared again, but only after baking and again only on the head.

I'm really not sure what the issue is here, I'm hazzarding a guess it has something to do with the normal of the high-poly but I'm not sure the best way to go about resolving this.

Any help and advise would be much appreciated.