It is too easy to accidentally change a swatch color in CC library

Hello, I have a UX reccomendation. 

I have frequently come across the problem of accedently changing the color of my library swatch while trying to click select it. It would be nice if one had to "unlock" a swatch in order to change the color, maybe just a simple pop up permission window. 

I'm disapointed becasue I was trusting CC library to keep track of my product line color pallate and did not write it down in other places so now I have to go re-construct my entire brand pallate of nearly 30 colors.