This is a problem with Illustrator's actions because they only have 'silent failure' in only some cases. Most of the time they aren't silent by default and there's no way to disable this. What you can do is put a bunch more 'filler' action events such as: make a section of your action that puts in a couple default shapes and assigns them a specific note, then groups them. Somehow, make sure that whatever your items of interest are, are also selected along with this new group. This can be done with some more notes or other ways depending on your situation, and there's a chance it can't be done in some cases too.
But supposing this is possible in your case, then when you do an ungroup and this dummy group is there, the action will not fail. All you have to do thereafter is select the dummy group's items with that original note and delete.
In any case, this is not pretty because you are having to do several more action steps for no reason other than to create a 'default' flow so that your one action doesn't fail.
It is possible to use notes and various obscure techniques to create a more robust actions this way, but due to the Action panel's UI and difficulties in finding one's way around it when an action has 50 lines - 40 of which are designed to make the 10 actual ones work (plus the thing takes 10x longer to run), the simplest method at that point is to just learn and do ExtendScript.
Here is an example of an overgrown action like so, in there you can see an example of "safe delete" where an item is first added so that in case there are no deletable items, there is still going to be at least one deletable item: https://www.linkedin.com/pulse/illustrator-variable-data-exploratory-techniques-using-vasily-hall/