Annoying search field in CC Library when using colors

Am I the only one to be very annoyed with the search function in the CC Library palette (in Photoshop anyway)? Every time I click a color in the library to use it, then proceed to hit a hotkey to change a tool (like V for pointer tool) the search field in the library palatte automatically populates and I sit here typing my hotkey letters into the search field, wondering why my tools aren't changing. It's a productivity pain in the butt. Anyone else? Can this be changed?