If you call the IndexToWorld(Vector3D, Vector3D) then the transformation's offset is ignored. Otherwise if you call IndexToWorld(Point3D, Point3D) the transformation's offset is added. This is confusing especially because this fact is mentioned nowhere. Not even in the documentation.
BUT this happens ITK-sided:
In the first case m_IndexToWorldTransform->TransformVector() is called. And in the second case ...->TransformPoint() is called.