Can Captivate HTML5 demonstrations be launched from a CHM file?
For CHM you would likely want to investigate publishing the Captivate as a SWF. Then adding the SWF to a topic or perhaps making it a topic of its own by importing the Captivate created HTML page. That would be the simplest approach.
For a CHM, embedding HTML5 won't work. CHM uses an old IE rendering engine that doesn't support the HTML5. So you'll have to go with SWF as Rick suggests. You may be able to open a page in the browser from the CHM. Then the regular browser is used and that one *may* support HTML5.