Ok. I see your issue. It is a problem with flattening. PDF/X1-A and PDF/X3 both have to flatten the file for print. PDF/X-4 is considered live... hence you will not see the issue until you flatten (you flatten to go to print). The problem is when you choose just Black (K) you end up with not enough colors to mix down when it has to raster an area in the flattening process. This makes it look different from the original background. You need it to be CMYK. You may only be using only the K channel in the original color, but having CMYK adds CMY to the mix down.
This is a process I always do in Illustrator and InDesign if I am going to print:
(I like to know what colors I am using). I usually use "Add Used Colors" unless you only want certain colors added then select the objects and use "Add Selected Colors". You can find these choices in the Swatches panel menu. I also "Select All Unused" colors and delete them. (That way I know exactly what I am going to print with.) Thought you do not have to do that. In any case, when you add the colors the one channel color (K) will come in as CMYK and be a global color.
You do not have to do anything else, just adding the global color will convert the color back to CMYK, where you won't have the weird box around the drop shadow.
Mike is right I keep it in RGB or CMYK and not both (PDF/X-1A will convert it all the CMYK as it can hold no RGB). I would probably convert the CMYK back to RGB or all the colors to CMYK (just use a PDF/X-1A) but I do not think mixing the color modes will make any difference in this particular case. (though you may prove me wrong on a particular digital printer)

After you add the swatches your K color all come in as CMYK as shown below

Hope this helps answer your question. Sorry about the delay on getting back to you.