Is the Responsive HTML output unstable for anyone else?

Along with jaggedpeak's excellent suggestions, you might also try renaming the preferences folder in your Roaming directory.

Here's my dir for reference
C:\Users\Matt\AppData\Roaming\14

Close Fm

Change 14 to 14-renamed (or similar)

Restart Fm

The 14 (for 2017 release) or appropriate dir for your version will be rewritten with default values, and may resolve your issue.

BTW, I find that the last two releases (2015 and 2017) are significantly more reliable than previous versions for HTML5 and other online format publishing.

Not sure if you have figured out the cause, but I just saw this post and wanted to add that I had this same problem.  For me, the issue was that I was trying to modify the mapping for my cross-reference formats, and one of my cross-reference format names has a backslash in it, like this:

heading \and page

Once I remove that backslash, no more crashes.

Unfortunately, what you've described so far is not enough (I think) to get a handle on your issue(s).

For myself, I've encountered issues (both with responsive HTML publication and other features) when my docs have included robust usage of conditional text, importation (by reference) of various file image types, Visio illustrations, and illustrator files, tables with conditional rows, etc.

You could also simply have a buggy installation (are you up to date with patches?)

You could have one or more things causing your problem(s). My suggestion is to work your way 'down' to the source problem -- something like this:

1. Identify a book in which the problem occurs 'reliably' -- that is, it crashes each time you try to publish.
2. Start FrameMaker again.
3. Open the book and all documents in it.
4. Publish *one* document at a time until you crash again.
   If you *don't* crash, try publishing for the entire book again. If you crash there, then the problem is publication at the book level.
5. If you *do* crash when publishing a specific document, start FrameMaker again, and open *only* the document that failed. Try to publish it again.
   If you crash again, then the problem is definitely in the specific document.
   If you *don't* crash, then the problem appears to be publication when several docs and/or the book are open.
6. Assuming you do crash when publishing the document in the previous step, take a look at the content of the document. Look for anything other than straight text (e.g. are you using conditional text, are you importing images or other files, etc.) Any of these 'unusual' items could be the problem.
7. My suggestion -- save the document under a test name, then either (a) work your way down to the problem (by removing specific content, saving, then publishing) until you stop crashing, at which point you know the offending content is the last portion that you removed, or (b) work your way up to the problem by removing all the content and then pasting content from the original, saving, then publishing, until you crash again, at which point you know the offending content is the last portion that you added.
   If you do either of these, I'd suggest you remove or add in logical 'chunks' (e.g. blocks of text, or an imported image, or a table, etc.) to make it easier to identify the specific offending content.