Page MenuHomePhabricator

[mxn multi widget] Allow mxn windows to show 3D view
Open, NormalPublic

Description

Currently, render windows in the mxn multiwidget can only select from the 2D slice views to show. It would be nice to enable a 3D view like in the std multiwidget.

Event Timeline

s434n triaged this task as Normal priority.Nov 22 2022, 12:39 PM
s434n created this task.
s434n moved this task from Backlog to In candidates on the MITK (v2023.04) board.

We discovered that it is currently not possible to use RenderWindowViewDirectionController::SetViewDirectionOfRenderer to change the view direction / rendering of a render window to 3D.
@s434n found out that in order to show the 3D rendering other render windows need to provide the three 2D views for the object in order to have data available to render.

I'm currently looking into this issue as well and found some unexpected behavior inside PlaneGeometryDataVtkMapper3D. I need to look closer if we can do something about it so that we can render the 3D view for the mxn multi widget.

I found out that the 3D render window of the StdMultiWidget does not render the data if the widget-plane helper-nodes are removed from the data storage.
If the widget-plane helper-nodes are available but hidden only from the 3D render window (set visibility for that render window to false using properties), the 3D render window does not render the data.
If e.g. one widget-plane helper-node is visible that 2D plane allows to render the data inside the 3D render window, but only the specific 2D plane is visible and not all three view directions.

As a conclusion I expect the mxnMultiWidget 3D render window to work, if at least a single plane node is visible inside the 3D render window and a plane geometry is correctly initialized.
As far as I know that should be the case in the current develop so I'll debug here more.

So it seems as if the PlaneGeometryDataVtkMapper3D is newly created (automatically) when a render window is changed to a 3D render window (mapper changed to Standard3D).
In that case, the m_DataStorage is empty and the following functions crash (e.g. mitk::DataStorage::SetOfObjects::ConstPointer all = m_DataStorage.Lock()->GetSubset(predicateAllImages);) inside PlaneGeometryDataVtkMapper3D::GenerateDataForRenderer.

A similar problem was experienced in T29394: [mxn multi widget] Image cropper shows weird behavior.

So I fixed part of it by setting a PlaneGeometryDataVtkMapper3D explicitly as done in {D759} for the 2D mapper. Then I was able to immediately set a valid data storage and the crash is fixed.
The three planes are rendered inside the 3D render window but only the frame is visible - no image information / pixel is shown.

I managed to fix the rendering of the 3D data even more and now the image information / pixel are shown.
But now the data seems to be inverted on some planes such that the three 2D planes do not match and form a 3D object.

It seems as if there is a general problem with the current crosshair: The crosshair manager provides three planes / plane geometries for each render window such that changing the view direction / anatomical plane will just switch to the corresponding geometry and render the information based on that geometry.
Having done some test with the visiblity of the planes, rendering inside the StdMultiWidget shows that all three planes (axial, sagittal, coronal) of a render window show the same image information / anatomical plane. Changing the view diretion inside the render window will change the anatomical plane, but then all three planes show the same image information of the newly selected anatomical plane.

This is the error I can see in he 3D render window where the three 2D planes do not match: it is not about inverted planes but about all three planes showing the same plane geometry, so all three planes from different directions showing only the image information from the same direction.

I may have found the problem but unfortunately for me that looks like it cannot be easily fixed:
During PlaneGeometryDataVtkMapper3D::ProcessNode, when the plane node to be rendered is processed (called from PlaneGeometryDataVtkMapper3D::GenerateDataForRenderer), imageMapper->Update(planeRenderer) is called. Here the imageMapper is the 2D mapper from the plane node.
Inside ImageVtkMapper2D::Update the mapper is updated by calling ImageVtkMapper2D::GenerateDataForRenderer. This in turn leads to the following line: const PlaneGeometry *worldGeometry = renderer->GetCurrentWorldPlaneGeometry(). I think this is problematic since here the current world plane geometry is used, which is the sliced geometry from the base renderer. However, for the 3D render window the sliced geometry is still either the axial, sagittal or coronal plane. It doesn't matter if we created three different planes from the world time geometry for the base renderer, the world plane geomertry will always be a specific sliced geometry.
Since these steps are performed for all three plane nodes to be rendered, all three plane nodes render the same world plane geometry.

I need to verify my findings but I think that might be the root cause.

kalali moved this task from Cycle to mxn MultiWidget on the MITK (v2023.04) board.