Remove background artifact?

I'm making "vitrual choir" videos, so each video that I receive from a choir member is shot with the camera on a stable platform, with the singer in the foreground. Now, suppose the background contains some artifact that is never occluded by the foreground action, such as a dark item on a shelf in an otherwise light-colored background. I'm fishing for ways to get rid of the background artifact.

The way I do it today, which works but is a lot of work, is to hide the artifact with a matte as follows:

1. Export a frame from the video
2. Edit the video frame in Photoshop:
   1. Draw a marquee around the objectionable object
   2. Apply "Content-aware fill" to "remove" the object from the picture
   3. Invert the selection (ctrl-shift-I on Windows)
   4. Delete everything (to transparent) that isn't the "content-aware fill"
   5. Save the resultant image file. I call it the "matte".
3. Import the image file and place the matte in the timeline above the original video. Just like magic, the object is gone (hidden by the mostly-transparent matte).
4. But wait, there more... It turns out that the brightness of the surrounding area in the picture is probably not constant. So, as the video progresses, if the surrounding area in the video gets brighter or darker (because the camera is doing some sort of auto-brightness thing) then the matte appears to be a sharply-defined darker or brighter area against the image. So (and this is the time-consuming part):
   1. Select the matte layer
   2. Apply the Lumetri color effect
   3. Go to the beginning of the timeline, and enable keyframes on color wheels/mid-tone brightness.
   4. Switch to the Color workspace, and display the color wheels section
   5. Play (or manually scrub) the video. As it progresses, you will see the luma in the scope move up or down as the camera adjusts its brightness. At the same time, you will see your matte become visible against the background. Slide the mid-tone slider up or down to bring everything back into equilibrium.
   6. Repeat that process. Again. And again. And play it back and fix the places where you didn't get it quite right...

So... This works, but the brightness thing takes a lot of work to sort out. I asked a separate question and, apparently, there is no way to automate things so that the area around the non-transparent matte remains at a constant brightness.

Another option, which wouldn't be as seamless but might be "good enough" and a lot less work, would be to define a marquee around the objectionable object in the image and make all of the colors in that object match the surrounding color. But I haven't found a way to do that well enough that the area doesn't stand out worse than the original "objectional object" in the image.

Any suggestions?