I'm new to FM. how do you format page numbers in a book?
You have to go the Master Pages and apply the desired formatting there (either Paragraph or Character Style depending upon how it's set up on those pages). If you have templates, then modify those and re-import them into the appropriate files of the book (File > Import > Formats).
If completely new to FM, I'd recommend picking up a copy of Matt R. Sullivan's FM bible starting with Publishing Fundamentals and working your way up to his Creating & Publishing Content one.