How to make multiple layers visible when multiple checkboxes are selected?

Layers are not mutually exclusive in PDF files. It's only this (ill-designed) dialog that forces them to work like that.
If you used a script instead you could show (or hide) the layers you wanted to change, while leaving the others as they were.

For example, the following code can be used to set a specific layer (called "Layer1", in this case) as visible when a check-box is ticked, and to hide it when it's unticked. Use it as the Mouse Up event of the check-box:

var ocgArray = this.getOCGs();
for (var i=0; i<ocgArray.length; i++)  {
	if (ocgArray[i].name=="Layer1") {
		ocgArray[i].state=(event.target.value!="Off");
		break;
	}
}

However, there is a catch. When the form is cleared, the layer will not become hidden automatically. You will have to do it manually, using a script.

Also, when the file is re-opened, the state of the layer will not be saved, and will revert to its initState setting.

This setting can be changed in Acrobat, but not in Reader. To get it to show/hide in Reader correctly you would need to embed a script in the document that executes when it's opened to restore the last view state of all layers, when it was last saved.