Instead of trying to achieve this with an inverted mask on a single clip, consider doing this:
Duplicate the video layer:
On the bottom layer adjust the colors to Black & White, so that it will remain constant.
On the top layer, add a mask and adjust the Lumetri Color setting to your liking.
Hope this helps.