Using custom sort within sorting by colour labels - Colors jumbled after!

Hello, I'm hoping someone can answer this easily for me...

I have always used color labels to sort my wedding galleries, as well as used 'custom sort' within the colour label groups.

Lately, after I've worked on my images by filtering each color group, I go back into the full library gallery and all of my color labels are jumbled like a checker board! I try to sort again by color label but then I lose all of the custom sorting within each color group... Am I doing something wrong in my workflow as to why this is happening...?

Hoping someone can explain this to me! Thanks!