Height/normal too strong

When using Image To Material (AI powered) I often find the surface displacement to be far too much. Adjusting the micro, medium, and large detail sliders most often yields unsatisfactory results in resolving the issue. What is the proper way to reduce the normal intensity/height displacement in Sampler when creating a new material?