In QmlMitkRenderWindowItem the mouse position has a flipped y coordinate. If you want to pick objects you can figure out that the y coordinate is flipped. I do not know why but I think you do the same in the normal render window. After adding the following lines the picking works as expected:

m_Renderer->GetDisplayGeometry()->ULDisplayToDisplay(point, point);