Certain unused swatches become transparent and can't be removed. Seems like a bug?

I have a collection of swatches in a document. I tried to delete several of them. Some of them turn transparent when you delete them and will not allow you to delete them at that point. I can't for the life of me figure out why. They are not in my document (I could delete EVERYTHING in the doc and they still refuse to delete). I have never encountered this before.

Has anyone experienced this and found a fix that doesn't require me to start a new doc and copy everything over? I have a lot of artboards with multiple layers and do not want to have to re-sort art to the proper layer with a copy/paste to another document.

I'm open to solutions that allow me to copy artboards with layers staying intact to a new doc if there isn't a better way to remove these rogue color swatches.

Attaching a before and after image of the swatches I'm trying to remove.