How to stop shape mask scaling when rotating?

Right click on the keyframes and check the keyframe interpolation. Make sure the spacial interpolation is linear. Actually, Andrew's suggestion sounds more like what you're looking for, since it's a mask rotation.  😉

The issue here is that you're not technically rotating the mask, you're animating the mask shape from one shape to another.  Think of it as morphing from one shape to another.  After Effects animates each mask point between your two keyframes. 

Look at the top left corner of your mask.  As you move through the timeline, that point moves in a straight line to the end point, where it becomes the top right corner.  All four points of the mask do the same thing.  Because of simple geometry, the square becomes smaller while the points move from start to finish.

The hard way to fix this would be to create curved motion paths for each of the mask's corner points, so they move in a circular path.

The easy way is to use layer rotation instead of mask animation.  Turn off or delete your current mask.  Create a square shape layer above your jpeg layer, the same size as the current mask, and set it as a Track Matte for the jpeg layer.  Then animate a 90 degree rotation of the shape layer.