exporting image at smaller scale without pattern rescaling

I'm also wondering if converting it a Smart Object first would help.

Untested, but I'm guessing that you may need to rasterize the pattern fill layer/s first before exporting (create a history snapshot first so that you can easily return to the pattern layers after exporting).

That's how it works.  If you want the pattern to stay the same size while the box changes, you need to select the box layer and use Free Transform.   I can't say exactly how this will work without seeing your layer arrangement, but give it a try and come back if still stuck.

So select thew box layer

Cmd T to select Free Transform

Opt drag a corner handle to resize, and OK it.

Do you have the pattern clipped to the box?  We can only guess without seeing your layers