No, it really isn't OK! 
There are many versions of Arial with different glyph complements and encodings. Substitute a version of Arial that doesn't match what you composed with and you may end up with gibberish. And no, there are mobile devices that don't have Arial - they substitute something else that may or may not match what you have.
Acrobat doesn't let you unembed if based on the encoding of the embedded font and/or characters used, there is a concern that you will end up with a useless PDF file.
Quite frankly, if we had to do it over again here at Adobe, we likely would never have allowed fonts only by reference (i.e., unembedded fonts). You have no idea the problems that are encountered with such situations that are reported by our users.
- Dov
PS: 97K for a PDF file isn't that big and if you are in fact using 83K of that for fonts, it is probably indicative of use of a very wide variety of glyphs, some symbolic, and/or styles such as italic, bold, and bold italic. No one will reject your “CV” / résumé because it is 97K byes in length!