Tangent Normal Map looks much worse than World Space Normal map?

Hi all, I'm trying to bake details onto my model but I'm having some trouble with generating the maps.
The World space normals generate correct details, however, the tangent normal map has odd artifacts and looks pretty bad once it's imported into other programs.
Does anyone know how to fix this problem?

the problem persists especially around the nose with a weird dip, and around the mouth area with odd corners. The tangent map is not very accurate to the high poly model, however the world space normal map is for some reason.

Any help is highly appreciated thank you!