Rotation in 3D

It's responding to a theoretical gimbal lock. You can avoid the switch to -180º in X and Z by using the timeline controls. The same thing happens in a lot of 3D applications. You cannot move more than 360º using any of the controls in the Composition Panel. If you want a layer to rotate two times, you must use the timeline. It's just a limitation of the UI.

You can see the reason for the flip if you add a camera, then use the camera tool to move the camera over the top of a layer. When you pass 180º, the layer flips over. This is avoided by using a camera null or by not looking straight down on a layer.