Hello,
The tessellation displacement technique does two things:
- Tessellation: subdivides the geometry procedurally on the GPU, which adds new vertices between the existing vertices
- Displacement: moves the vertices in the direction of their normal by x amount where x = height scale * height map. You can use the Scalar zero value parameter to define which greyscale value of the height map should be the 0 value.
Generally, the smoothness of the results will depend on a combination of the Tessellation factor value, the normal map applied to match the displacement, and the height map.
Since the techique merely moves vertices, it means that where height transitions are harder the textures will appear noticeably stretched, and the geometry can look jagged if the height map does not allow for smooth transitions in the displacement height.
You can blur the white square out just enough so the hard transitions can appear smooth for tessellation displacement. These issues also appear in the tutorial, yet probably because the shape is displaced down instead of up combined with the current lighting, it may look somewhat better.
I hope this is helpful and informative!
Best regards.