Perspective grid not working

What you need to do: Draw a rectangle that has the same dimensions as your image. Drag in into the perspective grid on the position you want

Make sure it is on top of the image

Select the image and the rectangle and apply the rectangle as an envelope: Object > Envelope > Make with top object. Afterwards neither of them will be part of perspective grid.