Not sure why the TOC is not populating in your case. You can try this:
1. Go to File > Save Ditamap As and then in Save as Type dropdown, you select Book 201X with fm Components.
2. Click Save. A new fm book is generated with Cover page, TOC, index, list of figures etc (You can remove the cover, index, LOF etc from the book file if you dont need it. It won't affect your HTML output even if you keep it, except the cover page).
3. Now publish the book file (not the Ditamap). This should work, I guess.
When I was exploring things in FM 2017, I used to directly publish the ditamap. Even though the TOC was reflecting in navigation bar of the responsive HTML5 output, the higherarchy (indents like Heading 1, Heading 1.1 and so on) was not reflecting as per the ditamap higherarchy. Not sure why!! May be I am missing something very fundamental here. If someone else has any idea, please let me know.