I'm guessing the border you applied was using the Borders button when you click Insert > Image. That border only appears when full size images are displayed and are not actually part of the image. When you use thumbnails, it expands the true image. If you double click the image in File Explorer and open it in whatever program your machine uses for viewing images, you will see there is no border in the underlying image.
To do what you want, apply the border with an image editing tool and don't use the Borders button.