Best way to straighten just a part of a video

For a movie scene we used fake wallpaper in order to make the set more appropriate. Unfortunately, amidst the shooting, the wallpaper started peeling-off and the vertical line in which both walls are supposed to meet looks very crooked and wrinkled (as you can see in the reference image that I have provided).

To make matters worse, the main character crosses over that line (between the camera and the walls), as do some shadows and a prop. The camera also has a slight movement (mostly a pan, but it was handheld so there is a bit of shaking as well).

I’ve been working on this scene for soooo long (seriously, a very long time) but haven’t figured out the proper way to fix it.

The best result I’ve gotten so far is using the Mesh Warp effect to distort only the part of the image in which the vertical line looks crooked. It works when you look at individual frames of the sequence, but once you play the whole video (which is about 30 seconds long) you can see the wall warping a little bit. I’ve keyframed the effect as much as I have been able to, but I still get a bit of warping.

I am hoping maybe someone has a better idea.

I’m sorry I can’t share the whole clip nor the complete frame (it’s been cropped a bit and the face is blurred), I haven’t got permission to do so. However, I think you can understand the conundrum I am facing.

Any help will be greatly appreciated! Thanks in advance!