custom javascript if checkbox checked

For this you definately need a script.

Are these "checkboxes" mutually exclusive? i.e radio buttons.  Or are they completely different checkboxes?

So here's a script that assumes the "checkboxes" are really radio buttons named "TaxType", with export values of  "Choice1", "Choice2","Choice3". 

This script goes into the custom calculation on Tax field.

var nTotal = Number(this.getField("Total").value);
var nChoice = this.getField("TaxType").value;
// block manual entry
event.target.readonly = true;

switch(nChoice)
{
    case "Choice1":
      event.value = .05 * nTotal;
      break;
    case "Choice2":
      event.value = 0;
      break;
    case "Choice3":
      event.rc = false; // This allows for manual entry
      event.target.readonly = false;
      break;
    default:
      event.value = "";
      break;
}