How can I change a precomp size without changing the masks?

It sounds like you added the masks to the nested comp (the pre-comp). If that is the case, masks are based on layer dimensions so when you change the size of the layer, or in this case, size of the nested comp, the relative size of the masks changes. 

Here is one option that will probably work. Before you change the size of the nested comp add a new solid to the main comp, select the nested-comp (pre-comp) and press "m" to reveal the masks on that layer, then select the mask path's individually and copy and paste them to the new solid. As long as the nested comp has not been re-positioned, rotated or scaled the masks you paste to the solid should perfectly match the masks in the pre-comp.  

After you have copied all of the masks from the nested comp to the solid open up the nested comp and change the comp settings and change the size of the nested comp. 

Back in the main comp delete the distorted masks on the pre-comp and then copy and paste the masks from the solid to the nested comp. You could also just use the solid as a track matte for the nested comp.