Select colors from the library panel directly from other areas in Ps

For example, creating a new color fill layer in Ps, there appears to be no way to directly select a color from the library colors. One can select from the color swatches/palette, but you need to have manually placed a library color in the local swatches first before trying to select from other dialogs.