Multi-selecting objects but unable to deselect the images in InDesign

In InDesign, there’s this project where I have a page full of a bunch of objects and I want to move the texts and shapes around but keep the image in the same spot. What I did was select everything and then Shift + Left Click on the image which normally deselects the image for me and I am free to edit what is left selected. Recently, I’ve been unable to do so, in the contrary, I’m still able to deselect any other object type.