In a very simple test with a GLB file I have handy, I'm not able to reproduce the problem as you've described in After Effects Beta 23.3 build 11. (I'll note that build 18 is being released as I write this.)
Which specific build are you uisng? Does the problem persist in build 18?
Does the problem happen in a new project? With any 3D model file or just some? Could you share a project file and 3D model that demonstrate the problem? Feel free to DM those to me.
If the problem persists, please provide your system specs so we can investigate. It sounds like the multiple views are not getting updated somewhere in the GPU blit path, so it will be useful to know the GPU and other specs of your system.