What model did you use?
If you selected a small part of the layer then the Firefly models work best. Nano Banana tends to reposition the generated pixels so they don’t align with the original. You need to select the entire layer (Ctrl A) with Nano Banana.
Unfortunately this only generates at 1024 x 1024, so will stretch the generated area downgrading IQ. What you can do is align the new and original layers as best as possible. Temporarily setting the upper layer to Difference will aid this step — get the layer to be as black as possible, and set the mode back to Normal.
Now add a Hide All mask to the upper layer, and carefully paint in the area you want with a soft white brush. That area will have the degraded IQ, but the rest will be back to full resolution.
Alternatively, if you have a good upsize plugin like Topaz Photo or Gigapixel, reduce the image to 1024 on the long side. Use Gen Fill and then upres. Topaz Labs reduced the price of its subscriptions a couple of weeks ago, and you can now get the full Studio subscription for US$163.