Video Demo and System Simulation?

Video Demo can best be compared with what Camtasia is doing: result is a pure video, no interactivity. You can insert PIP, static objects, trim, zoom and pan. The result is a MP4 video file after publishing. The raw file has the extension cpvc, and there is a dedicated video editor. You can insert a cpvc file on a slide in a cptx project, the normal Captivate projects. Advantage: better quality for pure video demos, nice editing possibilities. Disadvantage: no interactivity.

Software simulation (I suspect you mean that by 'system simulation') is the classic approach with Captivate:  recording can be done in different modes (demo, training, assessment or custom) at the same time. You'll end up with a normal Captivate project (also possible for responsive projects), static slides wherever possible, with interactive objects added (depending on the mode more or less) and editing will be done on slides. There will be small video slides, when movement or mouse actions require it (dragging with mouse is an example). Editing, especially synchronizing is much easier with that slide approach. Biggest advantage is that this can lead to interactive published files, if you publish to HTML or SWF (or to a SCORM object). That is impossible with Video Demo.