All my frames are coming out different sizes based on photo scaling %

Hi all,

I'm designing a book in InDesign and after placing all the photos I'm going back and adding white borders to them.

What I've noticed is, a 6 point white border is coming out different thickness on my photos based on the percentage of scaling I did to place the photo. E.g., a photo that I scaled down to 35% has a thinner border than a photo I scaled down to 85%. 

How do I get it where the thickness is uniform throughout the book and not tied to how I scaled the photo? 

Thanks!