Decontaminate colors adding weird texture

I used decontaminate colors and it is adding some weird texture in output.

And this is without selecting decontaminate colors

Though, by increasing contrast value this goes away but I don't want the result with contrast.

Why am I seeing this and how can this be fixed?