The # indicates the end of a story. It will appear in the upper left of every text frame until you add text, at which point it will appear at the end of any text, includeing text threaded to another frame. Most common reason for seeing it in an unexpected place is ID defaults to converting any empty shape or frame to a text frame if you click in it with the text tool. You can select the frame and reset the content to unassigned. It would also show an any blank page with a master text frame.
Your other blue mark is the Tab. I think there's probably a chart of non-printing characters in the help, but you should also find one over at InDesignSecrets.com.
Overset indicators appear in the last frame in a thread when there is more text. Open the story editor and see what's overset, and what is in front of it that might be causing empty intervening pages, though I suspect there really aren't and you've just shuffled pages so that the thread is jumping two pages. Turn on text thread visibility to see what is linked to what.
And buy the book.