Although there is no way to hide any of the color functionality, I'm sorry, but I'm not sure I'm fully understanding what you're referring to when you refer to the "color library."
If possible please attach a screenshot so I know I've answered your question correctly.