Why do library items get added to my swatches?

Hello.

I'm a bit new to using library items.  I have a handful of color swatches in my library.  If I create an item and add that library swatch as the fill color everythnig works great... But I've noticed that illustrator then addes it to my swatch pallete.  If I click on another item and try to add the library swatchs to it illustrator gives me a warning... "A different color with the same name already exists in your document......"

Is this supposed to happen?  Am I missing something.  Because now if I want to add a color swatch instead of going to my library I have to hunt for the color in my color swatch pallette.

Thanks!