Page MenuHomePhabricator

Segmentation plugin does not work correctly with the new mxn multi widget
Open, NormalPublic

Description

Due to missing defined axial, sagittal, coronal and 3D render windows

  • the slice interpolator can not be correctly initialized and thus the segmentation plugin does not work correctly.
  • the geometry-comparison for enabling the segmentation tool fails (missing "3d" render window)

A related problem was already mentioned in T25595.

Event Timeline

kalali triaged this task as High priority.Jun 3 2019, 1:55 PM
kalali created this task.
kalali lowered the priority of this task from High to Normal.Jun 3 2019, 5:06 PM

This was fixed for T25767. Need to look closer and test if everything works as expected.
The idea is not to ask for specific render windows but to retrieve all available render windows of a multi widget and use them.

Also, an active render window should always be available so this render window can be used to retrieve basic image information. However, the active render window will currently not be reset by the user so this might lead to undesired behavior: An active render window might not show the segmentation and therefore the segmentation plugin shows a status message that the segmentation is not visible e.g., although the segmentation is visible in an non-active render window.

An active render window will now be set by using the RenderWindowManager.

Reinit did not work correctly, as the RenderingManagerViewsInitializedEvent was not send for a single view initialization.
In case of a specified base render (using the RenderWindowManager context menu), reinit will only initialize the specified render window. This will now also send the mentioned event so that the segmentation plugin can react to this event as it would if all render windows would have been initialized.

The plugin always shows the message "Please perform a reinit on the segmentation image!", when the active render window is not an axial or 3D render window, though. In that case, both world geometries (working node and current geometry of active render window) are not equal.

It seems as if "3d" in line
https://phabricator.mitk.org/source/mitk/browse/develop/Plugins/org.mitk.gui.qt.segmentation/src/internal/QmitkSegmentationView.cpp$704
can be replaced by any render window to determine the equality of the bounding boxes of the world geometry and the segmentation node geometry. This allows to use the active render window to check for equal geometry although this render window might not be a "3d" rendering window.

I will do some more tests and see how this helps with the MxNMultiWidget. I would like to include this in the next release - i.e. adding these changes together with the changes of D533.

kalali moved this task from mxn MultiWidget to Cycle on the MITK (v2022.10) board.

I worked on this again and found the following:

  • with the different data that I used for testing the boundingboxes of the node geometry and the world geometry were always and only equal, if the render window used for the world geometry was either set to "3d" or "axial"
  • therefore, using the active render window (instead of the hard-coded "3d"-render window) only works if the active window has the view direction "axial" or "original"
  • this is true for the mxnMultiWidget and the StdMultiWidget (as the StdMultiWidget also sets an "active render window").

The issue is the following:

We can simply remove the geometry-comparison so that the segmentation tools are enabled, regardless of the geometry-alignment. What happens is that a drawn contour/ segmentation mask is written into the working image. However, since the world geometry of the render window does not show the geometry of the working image in this case the mask is probably "transformed" such that it fits the working image (projected onto a working image slice). This is unexpected and undesired as it does not reflect the true "mouse-drawn" contour.

Several solutions to overcome this problem:

  • reinit the render windows / the active render window, such that the image is aligned with the plane geometry
    • with our current methods this will reset the camera / view and lead to the problem discussed in T28490 (mentioned above)
    • can we align the geometries without resetting the camera / view? (maybe T27613: Improve reinit behavior)
  • do not allow multiple images with different geometries in a single render window (at least for the mxnMultiWidget)
    • this way each render window is always aligned with the geometry of the shown image
    • the problem with the geometry-comparison for non "axial" and non "3d" render windows remains, so this comparison has to be removed
  • do not project a drawn contour onto the image but use the world coordinates to draw the segmentation mask
    • I'm not sure if this is even feasible but I highly doubt that this will reduce confusion - the user gets what he sees but the contour might still be "transformed" if the render window is initialized to the image geometry (the contour have a different geometry than the image)

We can simply remove the geometry-comparison so that the segmentation tools are enabled, regardless of the geometry-alignment.

Removing makes no sense (see below), but may be relaxing. For what the Segmentation View wants you need no equal geometries you just have to ensure that the geometries are "grid aligned". So what we realy need is the IsSubGeometry() (may be even without checking the size of the geometry (so without 1016-1067)). Have you checked it with such a check?

I think that is not possible. In the end that is even what we want, because we want to ensure WYSIWYG for the segmentation and this means that all windows where you "paint into" are alligned with the pixel geometry of the segmentation image you want to paint. So if you are of the "grid", ultimatily one has to change the view direction.

  • do not allow multiple images with different geometries in a single render window (at least for the mxnMultiWidget)

I think that is no option. We have multpli Use Cases that would need the support of displaying images with different geometries as overlays while segmenting.
I also think that it is not the problem if images with multiple geometries are visible. It is the problem if the camera is not alligned with the geometrie you want to paint the segmentation into.

  • do not project a drawn contour onto the image but use the world coordinates to draw the segmentation mask

As you already concluded your self, that is not a good solution. I think it is important that we ensure WYSIWYG to not confuse the user.

If a relaxed check (see above) is not sufficient, we should discuss it in the MITK meeting. But I think a more sound/clean approach is, that we have a way who interaction can assess/express if a geometry is appropriated for them and render windows react appropriatly (e.g. showing a overlay message if you enter the render window "Not able to interact due to wrong...")).

If a relaxed check (see above) is not sufficient, we should discuss it in the MITK meeting. But I think a more sound/clean approach is, that we have a way who interaction can assess/express if a geometry is appropriated for them and render windows react appropriatly (e.g. showing a overlay message if you enter the render window "Not able to interact due to wrong...")).

Thank you for your input. @s434n and I already discussed some aspects and would like to present some solutions, maybe next week. The aspect of overlaying warning messages was also discussed and seems to be useful for the mxnMultiWidget, because here you could have RenderWindow A be initialized to the geometry of the working node (segmenting), but RenderWindow B shows another geometry (e.g. for manual, visual inspection) so the geometry of RenderWindow B would not allow segmenting here. This could be prohibited by an overlay message or something similar.

I think that is not possible. In the end that is even what we want, because we want to ensure WYSIWYG for the segmentation and this means that all windows where you "paint into" are alligned with the pixel geometry of the segmentation image you want to paint. So if you are of the "grid", ultimatily one has to change the view direction.

We already did something like this in T28490: Segmentation plugin: "new" segmentation should not run reinit!: Reinitializing the Render Windows to the working node geometry without changing the camera view / selected slices. This of course only works to a specific extent, since the selected camera position could be outside of the bounding box of the working image and a camera position transformed into another geometry can lead to slightly different positions (position interpolation) due to slice thickness, rotation matrix multiplication etc.
I already implemented a new reinit-function and will provide the code as a diff here soon.