You are using the perspective grid, not the 3D effect. In the 3D effect ths would work correctly.
In perspective grid you would have to expand patterns as well as strokes in order to get a perspective foreshortening.
This is the Perspective grid (not the 3D effect).
The perspective grid cannot distort patterns, but you can fake it with an Envelope distort.
Create a rectangle and position it in the Perspective Grid.
Make sure that you position it correctly, it cannot be changed when used with the Envelope distort.
View > Perspective Grid > Hide Grid
Create a rectangle and fill it with the pattern.
Object > Arrange > Send to Back
Edit: To make this work first check: Object > Envelope Distort > Envelope Options > Distort Pattern Tiles!
Select both rectangles and choose: Object > Envelope Distort > Make with Top Object.
Nice demo, Ton. 🙃
I just wanted to write something similar but you was faster 🙂