Unnatural looking transition between subject and background.

And when talking about images it can be useful to show them, so please post screenshots with the pertinent Panels (Toolbar, Layers, Channels, Options Bar, …) visible if you want to discuss the issue further. 

Thanks to everyone for your prompt and helpful responses!

Actually Stephen makes an excellent and very important point. Selections may not be necessary at all. Often a global overall adjustment is what is really needed, and then the parts will fall into place. Look at the whole first.

The "obvious" less obvious answer is to not make a selection or mask, accepting that the entire image will be impacted with a focus on the ranges that are critical. This is obviously not for every image, however, it does avoid a transition as there is no transition! That being said, channels from RGB/CMYK/LAB modes may make a better selection/mask than a more "precise" selection that ends up looking artificial.

I'll present a few PDF files from Dan Margulis that may provide you with a different view:

http://www.moderncolorworkflow.com/wp-content/uploads/stuff/Other/MA07-The_Steeper_The_Curve.pdf

https://www.ledet.com/margulis/Makeready/MA56-First_Refuge.pdf

http://www.ledet.com/margulis/Makeready/MA21-Defanging.pdf

More here: http://www.moderncolorworkflow.com/makeready-archive and here: https://www.ledet.com/margulis/articles.html

You have described an image on which you made a Selection and modified the area, then Inversed the Selection and modified that area. You found the Selection edge is too apparent. The following is a technique you may not have tried: It softens that edge using a separate layer and a blur over which you have control of blur width and blur amount.

The additional step:

With the Selection marching ants still showing, choose Select > Modify > Border which will change the Selection’s marching ants to a double marching ants selection, an "avenue" that seems to straddle the area of the original selection. 

Cmd+J will put this area of the image on a separate layer.  Hide the marching ants (Cmd+H). Apply a Gaussian blur to soften the edge where the separately modified areas butted. Each image you work on is different and may require a different amount of softening. Via drop down menu for Border and Blur, this technique gives you full control over the width of the “avenue” and the amount of blur. One more thing: If there is a section of the contour that doesn't need the gentle blur, just remove that area from the "avenue" layer. Good luck.  

It's called experience and a good eye 😉

I don't mean to be patronizing, but that really is the answer. Keep at it, learn from your mistakes, and you'll get there... 🙂