I want my default swatch palette to be empty. How do I do this without making a custom swatch panel?

When I open a new Illustrator document, I want my swatch panel to be empty by default. I don't want to have to "select unused" and delete them all manually each time I open a new doc. 

I've tried deleting them and saving that workspace without success.

I know there has to be a way to set up my preferences without creating a custom empty swatch panel or saving a template, because I've had it set up this way previously. Unfortunately, I had my preferences reset to fix a performance issue, so now I'm left with no clue how the employee who used to have my computer got it that way. 

Anyone have any ideas? Thanks