Weird distortion appearing when painting height

So I have that problem where I thought I found an older similar post to it but it didnt really help since all the video files etc. have been deleted. 
My problem beein that I am trying to add height to my model by just painting on a paint layer with the default brush and only the height channel. Works no problem, can see it in norm+mesh+height as well looking fine but when I start bluring stuff out with the blur tool preset it still looks fine but over time while painting I end up with a weird distortion of everything that I painted which doesnt make sense to me since its essentially black and white painting. It seems like its recalculating the heights and adding a super ugly effect to it. Also I am painting with tile flow but in this case only on one uv portion.

So what I mean is all those small dots and lines that u can see, that is what appears automatically after some time, I can go ahead and repaint over it to smooth it out but after a bit of painting on a different spot everything else adds this weird effekt again. Pls help