You could switch between Fill and Stroke in the Swatches panel instead and the Color panel will not open.
To add onto Jeff's suggestion try the hotkey of "x".
To add to Mike's keyboard shortcut, Shift-x will swap the colors of the Fill and Stroke of a selected object.
Yo, what if I never want that color panel to open? Like ever.
If I spend a bunch of time setting up all the panels with the tools I want displayed on the screen, I really don't want anything jumping in and mucking that up.
I agree with you. Looking for an answer too
Did you ever find one?
No, and I'm still pretty irked about it. The color panel is basically useless to me. I never want it to open up. I can't find a way to stop it.
Damn. That sucks. I 100% agree with you.
Just came across this issue, here's a solution.
Right click on the palette name and uncheck Auto Show Hidden Panels. You may need to save your layout prefs so that it remembers next time.