Ability to reduce the size of Swatch Panel so that ONLY Recent Swatches are shown

Currently, the swatch panel will not reduce to only show the small strip of recent colors. I prefer to remove the swatch libraries because they add no value to my work, but that leaves a large intrusive blank space.

I would like to see see the ability of the swatch panel to be collapsed to only show the small strip of recently used colors.