Greetings, I've spend waaaaayy to much time trying to open two old .FM documents. I don't know if they are FileMaker or FrameMaker. I've downloaded free trials but still stumped. If you can open it and provide PDFs I will pay $25 via Paypal. I prefer to work through private message or email not post it on here.
If you can't open them with the current version of FM (which the trial version will be), then they are very likely not Framemaker documents. Any later version of FM can open any earlier .fm or .book, back to FM1 as far as can be told. Some dialogs would be expected at <FM6 and <FM8, but they will open.
Open one of yours with Notepad and report the first few dozen bytes of text. Here's an example from a 1999 FM5.5 ".fm" binary file: <MakerFile 5.5Q> Even though .fm is a binary format (and the bulk of it would look like Unicode gibberish in current Notepad), the opening string is ASCII, likely due to FM's Unix origins, and needing to have a "magic number" (format ID) in the first few bytes, often chosen to be human-readable.