If you are trying initiate a capture from within an existing CPTX project file then the size of the capture is restricted to the current dimensions of the CPTX.
The recommendation is that you first work out what the final finished size of your project needs to be (height and width in pixels) and then you capture for that exact size.
If the current project file is NOT the size you want, then start your capture using File > Record New > and then choose either Screen Capture or Video Demo and set the capture dimensions to what you want. That gives you the freedom to choose the capture dimensions, but then it also means your captures might not fit in your project file later if it is not exactly the same size. Lots of compromise.
Be aware that the larger your capture window, the larger the file size will be. So in general you don't want to capture full screen.
Custom size is not available because you started the recording from within an existing project, not from the welcome screen. You used the Slides button, Software Simulation, If you capture it from the welcome screen you should be able to do the recording in any size. Since any project which is non-responsive can have only one size, the software simulation slide will be the same size (resolution) as your other slides.