I was with support for 2 hours today trying to figure out why my Macbook pro couldn't use Software Simulation. Support told me it was an Apple issue with resolution and I needed to upgrade my video card. This is a brand new computer. I've had it for 1.5 months. It's a beast.
I got it to work slightly when selecting a program to record rather than an area. It seems that this has been an issue for a lot of Windows users that was cured with Administrator rights but I don't have that issue with a Mac. I have 5 curriculums that need to go out in 2 weeks and I just don't want to drop bad .mp4 files in them when I can do the Software Simulations. Ugh.
Yes I'm running Catalina. It wasn't my choice.
If you are on MacOS Catalina, are you also running the latest version of Captivate 2019 (version 11.5)? If not, you need to upgrade to that version before anyone here will be able to offer much help.
Are you trying to capture simulations on a high resolution monitor (e.g. a retina screen)? Elearning software simulations rarely benefit from capturing at extreme resolutions. Full HD 1920x1080 would be as high as I have ever needed to go, and that was only because the client thought it necessary since all users were on HD monitors. But I had to point out that capturing at this resolution would leave no room for TOCs, skins, and normal browser toolbars etc. When we took all things into account, the actual ideal resolution to capture was much smaller than full HD.
Are you capturing on the primary monitor or a secondary monitor? If trying to do this on a secondary monitor, try disconnecting from the secondary monitor and just use the primary one.