the mitk::Geometry3D::ExecuteOperation() in case of opROTATE cuts off any rotation with an angle not within [-360, 360]. Consider the code lines
309: angle = (angle < -360) ? -360 : angle;
310: angle = (angle > 360) ? 360 : angle;
Is this the desired behaviour?