Swatches not putting correct color on brush

I've been working on this animation for almost a year now, as i've continued i've noticed more and more instability on animates part. The latest issue i'm having is a real annoying one. I have swatches saves into 50 or so folders in my swatches panel. But for about a month now, if i click on one of those swatches to sample the color, the expected behaviour is that my brush will pick up that color. What ends up happening about half the time is that i will select something like a light blue, and suddenly my brush is hot pink. It's seriously slowed down my workflow. Any ideas?

EDIT: while attempting to troubleshoot this i've noticed that what seems to be happening is that while pulling from my custom swatches, animate will apparently try and sample from the default swatches at the top of the list, as if they were on the screen and under my custom swatches. I've added a short video below showing the issue