How to change the orientation of 3D layer?

That is to be expected. You are running into gimbal lock issues, i.e. the old Euler vs. Quarternion thing. What you want to do cannot be done by just rotating the 3D layer. You have to use auxiliary Nulls to neutralize certain rotations and then have clean values. I would suggest you look up a few basic tutorials on rigging stuff for animation, including how it's done in 3D programs. these principles also apply to how you would do it in AE.

Mylenium