Can't relink or remove certain components in Assets panel

I have run into an issue where I am not able to remove or relink certain components from the Assets panel. The situation is this, I will copy an artboard from one document to a new one, with it came linked assets that don't actually exist any more. So in the assets panel they have a red link icon next to them. When I right click on the component I'm only given Highlight on Canvas and Relink. When I chose the Relink option, it tells me "No matching assets are found". That is correct because I had later updated and renamed the component. But I do not have the option of removing it from the assets panel. I have even tried readding the component to the file so that the relink can find it again but it doesn't. Nothing I seem to do allows me to remove these unwanted components from my file. Any assistance on how to resolve this would be great as this is not the first time I've come across this.