As I pretty much expected ... you're applying bigger and bigger hammers to a rather fragile nail, then shocked .... SHOCKED!!! ... when the nail bends or breaks.
And using a LUT after you've bent the crud out of the pixels of the image is gonna go nuclear, no way around it.
I work with colorists, I teach colorists ... and one of the more common colorist sayings is "LUTs are the dumbest math out there". They are simply a data table saying "take all x:y:z to p2:q:a43 ... take all ... ". Most LUTs have a point at which they break the media involved. Even manufacturer supplied LUTs. So colorists test the heck out of them with tech images and funky color bars and such to find out when they can use them and how.
Your media is not pure plastic that can be molded anywhere you want. Changing pixels is possible, but ... too much or done the wrong way, it breaks rather than bends. You get harsh shifts, induced chroma artifacting and macro-blocking. Banding.
So ... this has been a learning exercise. Great! Now learn, and see what you can do ... and what you can't ...
Neil