Page MenuHomePhabricator

Slice interpolation 'confirm for all slices' not working for large images
Open, HighPublic

Description

In the segmentation plugin, the 2D slice interpolation if one of my favorite tools. It does exactly what I want it to do and is super useful!

However, it falls flat when working with large images, for example this dummy segmentation:

When looking through the segmentation with slice interpolation enabled (2d) , you can see that the yellow preview contour works nicely. However, when I want to 'confirm for all slices' (I selected coronal of course!) MITK will compute for a couple of minutes and then at some point just stop without throwing an error. No interpolated segmentation is generated.

I can work around that by:

  • loading the segmentation in python
  • cropping to the area where the slices are (with some padding so that I dont run into border problems. I don't know whether 2D slice interpolation can move outside the bounding box)
  • saving the cropped segmentation
  • loading the cropped segmentation in MITK
  • 2d slice interpolation 'confirm all slices' now works with the smaller image. This computes interestingly MUCH faster than whatever MITK did before
  • save result
  • load result in python
  • place result back into original segmentation
  • save again
  • load in mitk

As you might expect, this is not very user friendly ;-) I don't know what caused it to crash for larger images, but you might want to look into that.
Note: I had plenty of RAM available, so that's not the problem.

image.png (24×442 px, 6 KB)

Ubuntu 18.04

Best,
Fabian

Event Timeline

isensee added a subscriber: kislinsk.

@kislinsk I know you have plenty of things to do, but this bug (along with T28516) is a blocker at the moment. I would like to recommend MITK as an annotation tool for our collaborators but cannot do that as along as the slice interpolation does not work (fast enough, T28516) for large images :-)

As it is an inexpensive operation my first guess would be an issue related to the amount of used memory. For example, the image above eats 3,5 GB, Activating the interpolation already triples it. Confirming adds at least another 3,5 GB. That could also explain why it is so slow (swapping). But at this point it is just a first assumption. On a 16 GB Windows machine I was able to complete the interpolation without crash but it took a while.

edit: But maybe it is also not as inexpensive as we though as we rarely have 2048x2048 image stacks.

And we seem to have a huge leak here. Even after deleting all the nodes again, half of my RAM is still in use.

kislinsk triaged this task as High priority.
kislinsk edited projects, added MITK (v2021.10); removed MITK.
kislinsk moved this task from Backlog to Cycle on the MITK (v2021.10) board.

A few things that impact the handling of huge images:

  • Loading an image or at least a segmentation currently required twice as much RAM for a short amount of time. I guess there's a block copy happening.
  • Interpolation is really heavily computing on a single core and adds another chunk of memory (twice as large) during the operation.
  • While one part of that memory is released after the operation is complete, the other half is still not freed until the Segmentation view is closed. Removing the image or closing the project is not enough.

As it is an inexpensive operation my first guess would be an issue related to the amount of used memory. For example, the image above eats 3,5 GB, Activating the interpolation already triples it. Confirming adds at least another 3,5 GB. That could also explain why it is so slow (swapping). But at this point it is just a first assumption. On a 16 GB Windows machine I was able to complete the interpolation without crash but it took a while.

It is not a RAM issue. I checked that :-) No swapping on my 32GB machine. I would have noticed and reported that - I am not new to these things you know :-P

Most computers now have 8 cores or more. This operation takes forever to complete, so it would be nice to have it distributed across the cores

Without looking into parallelizing the interpolation yet, I was already able to algorithmically speed up the interpolation by 7-8 times.

1024³ image block 2d-interpolation
==================================
Without optimizations      392.78s
Distance image cache        53.17s
-- Speedup factor ----------- 7.4x

Interpolation is not done in place to build a Do and Undo operation for the Undo/Redo-Stack. In the test case above this is another 30 seconds (more than half of the time with the first optimizations). While undo/redo is important, we should consider coming up with an heuristic, or option, or something similar to disable it for the interpolation of large images.

Combined with a multithreaded iteration through the interpolation slices, we should be able to reduce the duration to a few seconds.

1024³ image block 2d-interpolation
==================================
Without optimizations      392.78s
Distance image cache        53.17s
-- Speedup factor ----------- 7.4x
Distance image cache +
Extract slice cache         47.22s
-- Speedup factor ----------- 8.3x

Sounds like it goes in the write direction. :) thanks.

undo/redo: Why is there a dedicated Filter executed anyways for the undo/redo step? Couldn't we just store/clone the current state?

On my 4 cores / 8 threads workstation, I get a speedup factor of nearly 10x (or 35x, if neglecting the whole do/undo stuff during measurement). This is for a 1024³ block, where only the first and last slices contain segmentations, so the whole image block need to get interpolated.

Since we want to do a snapshot release at the end of the week, I will already request a code review. If accepted in time, 10x is better than nothing I guess. :-)

1024³ image block 2d-interpolation
==================================
Without optimizations      392.78s
Distance image cache        53.17s
-- Speedup factor ----------- 7.4x
Distance image cache +
Extract slice cache         47.22s
-- Speedup factor ----------- 8.3x
Distance image cache +
Extract slice cache +
Multithreading              40.44s
-- Speedup factor ----------- 9.7x



1024³ image block 2d-interpolation
without do/undo operations
(- fixed 30s)
==================================
Without optimizations      362.78s
Distance image cache        23.17s
-- Speedup factor ---------- 15.6x
Distance image cache +
Extract slice cache         17.22s
-- Speedup factor ---------- 21.1x
Distance image cache +
Extract slice cache +
Multithreading              10.44s
-- Speedup factor ---------- 34.8x

Deleted branch from rMITK MITK: bugfix/T28491-SpeedUp2DInterpolationsOfLargeSegmentations.

I will create another task for an optimization of the diff image operations in the do/undo stack.

This is still not fixed, can we please re-open it?

(Ubuntu 18.04, MITK 2021-07-02 shapshot)

@kalali @a178n

I looked into this using the latest Develop-branch on Windows. Using the Debug build I was able to see the stack trace after a crash (using the small, cropped image):

Exception Unhandled:
Unhandled exception at 0x00007FF96A63AFAC (ucrtbased.dll) in MitkWorkbench.exe: An invalid parameter was passed to a function that considers invalid parameters fatal.
_STL_VERIFY(this->_Getcont() == _Right._Getcont(), "vector iterators incompatible");

inside vector -> _Compat, coming from

while ( ci != m_PointsContainer->End() )

inside bool BoundingBox< TPointIdentifier, VPointDimension, TCoordRep, TPointsContainer >::ComputeBoundingBox(void) const (in itkBoundingBox.hxx)
with

ci = itk::VectorContainer<unsigned long,itk::Point<double,3> >::ConstIterator

and

m_PointsContainer = const itk::VectorContainer<unsigned long,itk::Point<double,3> > *

or

m_Iter	{...}	std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<itk::Point<double,3> > > >

and

r.m_Iter	{...}	const std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<itk::Point<double,3> > > >

respectively.

Using the large, original image I get a crash inside delete_scalar.cpp in void __CRTDECL operator delete(void* const block) noexcept, coming from void LightObject ::UnRegister(), used by m_BoundingBox->SetPoints(points); inside void mitk::TimeGeometry::UpdateBoundingBox().

I did some testing (with Amir) on ubuntu 20.04 (32GB RAM). The issue could be replicated with a questionable increase in the utilization of RAM.
Fabian provided us a couple of files for testing, viz. tmp.nii.gz and tmp_cropped.nii.gz (same content, just removed some of the zeros around the ROI)

1- Tests using tmp.nii.gz
We tested multiple times the following workflow:

  1. open MITK
  2. load tmp.nii.gz
  3. open segmentation plugin (not multilabel segmentation!)
  4. No need to select an image, but make sure the segmentation is selected
  5. at the bottom, select Interpolation-> 2-Dimensional
  6. at the bottom, click confirm for all slices -> coronal

There are two possible outcomes:

  • Aborted (core dumped) crash
  • Computation runs without a crash. Preview of interpolated results shown (yellow masks, after Step 5) but Not confirmed (after Step 6). Also, high RAM utilization is shown in the bottom right corner of the workbench.
    • Now Re-run step 6. Outcome: Core dumped OR memory utilization 99%. Same behaviour as above.

2- Tests using tmp_cropped.nii.gz

We test multiple times the workflow as before.

There are two possible outcomes:

  • Aborted (core dumped) crash.
  • Computation runs without a crash and the result is confirmed/rendered as it should with No questionable rise in RAM usage.