git bisect identified the following commit:

Bisecting: 0 revisions left to test after this (roughly 0 steps)
[b38c33c5a96fcba9556c575630b4eff276395845] Fix depth buffer problems on linux

we recently switched from QVTKWidget to QVTKWidget2 and now by default anti-aliasing is disabled.

We managed to track the issue down with this VTK example:
http://www.vtk.org/Wiki/VTK/Examples/Cxx/Qt/RenderWindowUISingleInheritance

Just render the sphere in wireframe mode. If you run the example with a QVTKWidget, lines are smooth and with QVTKWidget2 they are stepped. SetMultiSamples(8) should do the job, but this does not have any effect (and is the default value).

I am waiting on the VTK developer list to reply on this.

I found a fix:

with the usage of QVTKWidget2, multisampling has to be enabled in QGLWidget as well. I believe that QGLWidget overrides the call of
glEnable( GL_MULTISAMPLE )

So in the constructor of QmitkRenderWindow this has to be added:

QGLFormat newform = this->format();
newform.setSampleBuffers(true);
newform.setSamples(8);
this->setFormat(newform);

User kolbch has pushed new remote branch:

bug-18573-EnableMultisamplingInQVTKWidget2

[4abcd7]: Merge branch 'bug-18573-EnableMultisamplingInQVTKWidget2'

Merged commits:

2015-01-14 16:17:44 Christoph Kolb [68e168]
enable multisampling in QVTKWidget2 as a workaround