Why can't I delete the "rejected" images from the export?

You have to give more information on what exactly you are doing. If you select a bunch of images and try to export them, then Lightroom (which Lightroom are you using?) will do as you ask and export them. A ‘rejected’ label is irrelevant because you selected the image anyway and told Lightroom to export it. If you want to filter out ‘rejected’ images, then you should do that first and then export the remaining images.