Cannot change gradient color with swatches

I used to be able to choose the colors in the gradient tool by selecting one of them from the swatch menu and then switching to my background color with 'x' to change the other color. But suddenly I can't do that anymore and I can only change the colors by going into the gradient editor or choosing gradient presets, how do I change this back to normal? I'm just trying to make a linear or radial gradient with two colors.