Page MenuHomePhabricator

Check the current behavior of the segmentation tools according static and dynamic 4D segmentation
Open, Unbreak Now!Public

Description

Some of the tools do not behave properly right now according the postconditions defined in T27506: Extend checklists for segmentation view.

E.g. Threshold tool cannot be confirmed for static segs on 4D images in the multi lable seg view; threshold tool in the segmentation view changes a 4D segmentation into a 3D segmentation...

All tools for multi lable segmentation and old segmentation should be checked according to the post conditions definiened in T27506. If errors can be found the should be listed as tasks for 2020.

Related Objects

Event Timeline

floca triaged this task as High priority.Jun 27 2020, 1:30 PM
floca created this task.
floca moved this task from Backlog to Segmentation on the MITK (2020) board.
Plugin namedynamic timestep =0dynamic timestep != 0static t=0static t =2
Thresholdperforms segmentation on all timestepsperforms segmentation on all timestepsperforms segmentation on timestep 0performs segmentation on timestep 0
UL thresholdperforms segmentation on all timestepsperforms segmentation on all timestepsperforms segmentation on timestep 0performs segmentation on timestep 0
Otsuperforms static 3D seg on the selected timestep + it only matters which time point is selected not on which it was createdsee dynamic t = 0performs segmentation on the timestep selected right before executionperforms segmentation on the timestep selected right before execution
Fast Marchingit doesn't matter on which timesteps the seeds are selected, they all account for initial image / performs segmentation only on the timestep selected, others are blanksee dynamic t=0
Watershedsegmentation cannot be produced with US4DCyl / LinearModel / performs static 3D seg on the selected timestepsee dynamic t=0performs segmentation on timestep 0performs segmentation on timestep 2 / it only matters which time point is selected not on which it was created
PickingCreates a 3D segmentation with picked region and deletes all other segmentations (in all timesteps)

wip

Most likely errors:

  • segmentation is always performed on timestep 0 (Threshold/ULThreshold)
  • dynamic segmentation is converted into a 3D segmentation (Otsu, Fast Marching, Picking)

Discussion so far:

  • perform on all timesteps (threshold / ULthreshold) might cause a huge overhead and should not be to default
  • dynamic segmentations should not lead to 3D segmentation result
  • not sure whether static execution should happen on the created or executed timestep

Discussion about the default and preferred behavior: T27509: [Segmentation] How to handle 3D and 4D Segmentations on 4D images.

I still back my own default proposal statet in T27509. I think this is the best way today.

My default proposal (see T27509) would lead to this status result:

Plugin namedynamic timestep =0dynamic timestep != 0static t=0static t =2
Threshold11 2
UL threshold11 2
Otsu66
Fast Marching??? 3??? 3??????
Watershed??? 4??? 4
Picking??? 5

1: I would make it option to perform on all timesteps (but only a nice to have for this run)
2: should be performed using the current selected timestep of image
3: @thomass I am not sure I understood the problem correctly? What do you mean with initial image when you write the currently selected image will be segmented? The static table cells are empty. Does it mean they where not checked, or that everything is fine?
4: @thomass I am not sure I understood the problem correctly? Does the tool work, but on some images (as US4DCyl) not? First half sounds like a bug, second half sound like a correct behavior.
5: I am not sure if picking makes realy sense on 4D at all. I tend to say you can pick on 4D in the current time step, but it will always produce a static mask with your picked segmentation.
6: Generates 3D segs instead of 4D segs
What do you think?

In addition to the existing results:

Plugin namedynamic timestep = 0dynamic timestep != 0static t=0static t =2
Region Growingperforms segmentation on the selected timestep (not necessarily t = 0), mask is 4Dperforms segmentation on the selected timestep (not necessarily t = 2), mask is 4Dperforms segmentation on timestep 0performs segmentation on timestep 2

Concerning static with t != 0: The problem I encountered is, that the segmentation was done on the structure of timestep 2, but the underlying image itself is the image from timestep 0:

Reinint on US4D image (t = 0), timesteps available, static mask created on timestep 0

Reinint on US4D image (t = 2), timesteps available, static mask created on timestep 2

Reinint on static mask (t = 0), timesteps not available (3D mask), static mask created on timestep 0

Reinint on static mask (t = 2), timesteps not available (3D mask), static mask created on timestep 2

@thomass Just to make sure there is no confussion: In your table, where static t = 2 and the segmentation was created on t = 0: Was it actually computed on the structure of t = 0 or was it computed on t = 2 but the underlying image after reinit showed t = 0, as in this case described?

So I think it is a good thing to create a segmentation based on the structure of the image at timestep 2 and have this as a memory-saving 3D segmentation mask. But either our reinit-algorithm is misleading or the static mask should really only cover this individual timestep, instead of the whole time span. Or am I missing something?

But either our reinit-algorithm is misleading or the static mask should really only cover this individual timestep, instead of the whole time span. Or am I missing something?

I think you are missing, that one should never reinit on the segmentation itself, but always on the reference image. Then there is always a time slider apropriate to the reference image where you can select the current time point and the reference image content associated with this time point should be used to generate the (static) segmentation.
Could you please recheck your evaluation? I think we have a fundamental difference in the assumption how the segmentation tools should behave an therefore what the correct outcomes of the tools should be. Thank you.

So I am not quite sure that about the re-init problem but to further clarify my list:

  1. What I meant here was that a static segmentation is returned by the tool although a dynamic was created by the plugin -->therefore I would call it a bug. The same holds for Otsu.
  1. I couldn't perform the operation on those images / the result was always the same for all steps (I had trouble finding right parameters that change something), but anyways: it created a 3D segmentation in the end which should not be the case if we created a dynamic segmentation before, right?
  1. Maybe we should really clarify what the Picking tool should do.

General: I agree that we should have a general Segmentation discussion this week! :)

@kalali / @thomass Can you spawn the identified open tool problems as sub tasks till tomorrow noon or at least indicated here the final results after our todays discussion/clarification how to proceed? That would be very helpfull as they have also to be tackeled. thanks. Ralf

So Region Growing 3D seems to work and could probably be used for reference for other algorithms.

@thomass I checked Fast Marching 3D and re-opened T18436: [Segmentation] Fast Marching 3D Tool only supported for first time step in 4D images for the static mask cases.
Could you re-check the dynamic case? I tested it and with t = 0 I got a dynamic mask (4D) with a segmentation only on the first (0) timestep but with t != 0 I have the same problem as described in the mentioned task (green contour nowhere near the seed point).

I can reproduce the behavior for the US4DCyl but not for the heart 3D+t. But it looks exactly like the error for static segmentation and t=2 (that I showed during the telco).

All issues have been adressed in respective subtasks.

All issues have been adressed in respective subtasks.

@thomass I cannot find a sub task for the watershed tool. Looking at the table/your explination it also is not correct, becuase it always produces 3D masks. Or is this status not correct anymore?

You totally right and the reason why I oversaw this is when I double checked:

If you perform a dynamic segmentation using watershed, it creates a new segmentation no matter if you created that dynamic segmentation before.
The dynamic segmentation stays 4D (which is correct) but the new '_Watershed' segmentation is 3D. The result is stored in that new segmentation not the dynamic one.

To not blow up the datamanager with that, I think it is a bug and the _Watershed should not be created but instead the dynamic segmentation should be used.

floca edited projects, added Missing Info; removed Cleared.

@thomass / @kalali : what i just realized. Did you also checked the automated 2D tools (like fast marching 2D)? Or only the 3D tools?

2D Segmentation tools:

  • Add
  • Substract
  • Correction
  • Paint
  • Wipe
  • Region Growing
  • Fill
  • Erase
  • Live Wire
  • 2D Fast Marching

--> First test Looks suprisingly good, even 2D Fast Marching tool works as expected on US4DCyl.nrrd for dynamic and static Segmentation
--> Region Growing works correctly in terms of timesteps but shows odd behavior (the mouse cursor is released but still more foreground pixels are added --> this often causes an oversegmentation on US4DCyl @kislinsk is this something potentially caused by your recent change?

wip

2D Segmentation tools:

  • Add
  • Substract
  • Correction
  • Paint
  • Wipe
  • Region Growing
  • Fill
  • Erase
  • Live Wire
  • 2D Fast Marching

--> First test Looks suprisingly good, even 2D Fast Marching tool works as expected on US4DCyl.nrrd for dynamic and static Segmentation
--> Region Growing works correctly in terms of timesteps but shows odd behavior (the mouse cursor is released but still more foreground pixels are added --> this often causes an oversegmentation on US4DCyl @kislinsk is this something potentially caused by your recent change?

wip

@thomass Could you please spawn a sub task for your reagion growing finding?

Region growing task can be found here T27684

Closed as all sub tasks relevent for MITK2020 are checked and corrected.

floca raised the priority of this task from High to Unbreak Now!.

It seems (see @kalali latest findings T27721 T27800) that there are more problems in the 2D tools then thought.

@kalali / @thomass : Is it possible that you assess the state till next MITK meeting this week?

It really suprises me that FastMarching2D crashes now since I explictly played with timesteps in T26975. But maybe I truly overlooked something, then I am sorry.

Static Segmentation:
-ADD: A segmentation is drawn on timestep 0, if I change the timestep, the segmentation remains and I can add a new segmentation on timestep !=0, both segmentations are visible since I only have one timestep --> As axpected
-SUBSTRACT: The drawn segmentation can be removed independent from the timestep 0 and timestep !=0 --> As expected
-CORRECTION: The drawn segmentation is removed independent from the timestep 0 and timestep !=0 --> As expected
-PAINT: Painted segmentation occurs on timestep 0. When changing the timestep, the painted segmentation is added to the static segmentation --> As expected
-WIPE: Segmentation is wiped on timestep 0 and wiped on timestep !=0 --> As expected
-RegionGrowing2D: Region Growing is applied on timestep 0 (fits to the image). When changing the timestep, the region growing result fits to the timestep. No crash. Quite slow updates. --> As expected
-FILL: Different timesteps can be filled. When changing the timestep, the segmentation remains --> As expected
-ERASE: Different timesteps can be erased. --> As expected
-LiveWire: T27800
-2D FastMarching: T27721

@thomass Thanks.
Now I might be nasty. Have you also checked everything wenn loading to 4D images with different time geometry? E.G. So that the selected time step in the navigator must not correspond to the same time step of the image you segment on?

It really suprises me that FastMarching2D crashes now since I explictly played with timesteps in T26975. But maybe I truly overlooked something, then I am sorry.

I just tested it with both mentioned datasets MITK-Data\UltrasoundImages\4D_TEE_Data_MV.dcm and MITK-Data\3D+t-Heart and it crashed with both. Maybe - because it was not explicitly stated - the newly created segmentation was a static segmentation, not a dynamic segmentation; as it seems that this crash only occurs with a static segmentation (which makes sense looking at the proposed solution in the corresponding differentials).

@thomass Thanks.
Now I might be nasty. Have you also checked everything wenn loading to 4D images with different time geometry? E.G. So that the selected time step in the navigator must not correspond to the same time step of the image you segment on?

With two 4D images 3D+t-Heart and US4Dcyl it crashes indeed when

  • using the Add tool on the static 3D+t Heart on timestep 0
  • changing the timestep and then use the Add tool again

I couldn't reproduce the crash when combining 4D_TEE_Data_MV and US4Dcyl (yet)

@thomass Thanks.
Now I might be nasty. Have you also checked everything wenn loading to 4D images with different time geometry? E.G. So that the selected time step in the navigator must not correspond to the same time step of the image you segment on?

With two 4D images 3D+t-Heart and US4Dcyl it crashes indeed when

  • using the Add tool on the static 3D+t Heart on timestep 0
  • changing the timestep and then use the Add tool again

I couldn't reproduce the crash when combining 4D_TEE_Data_MV and US4Dcyl (yet)

Could you please check (if not done already), if this scenario also crashes if you do it with a static segmentation?
This info is important for the decission, if we have to fix it directly or just partialy deactivate the dynamic option for this cycle. (Curently static segs on 4D images are more relevant then dynamic segs).

@thomass Thanks.
Now I might be nasty. Have you also checked everything wenn loading to 4D images with different time geometry? E.G. So that the selected time step in the navigator must not correspond to the same time step of the image you segment on?

With two 4D images 3D+t-Heart and US4Dcyl it crashes indeed when

  • using the Add tool on the static 3D+t Heart on timestep 0
  • changing the timestep and then use the Add tool again

I couldn't reproduce the crash when combining 4D_TEE_Data_MV and US4Dcyl (yet)

Could you please check (if not done already), if this scenario also crashes if you do it with a static segmentation?
This info is important for the decission, if we have to fix it directly or just partialy deactivate the dynamic option for this cycle. (Curently static segs on 4D images are more relevant then dynamic segs).

Since we were having problems with static segmentations, I tested (as shown in the table above) all tools with static segmentations. Therefore, the test with two Images was also carried out using static not dynamic. And (I did also not state that clearly): I checked with the Add tool and it crashed. It didn't have the time to redo all the test for two Images yet. If necessary, I might need a helping hand because static/dynamic*single/double*2DTools/3DTools = many test cases

@thomass OK. Thats enough info, thanks. No further testing for now. We do it in context of a release branch or/and explizit fixes.

We had a discussion about refactoring the segmentation tools (similar to {D380}). The idea is to outsource redundant code into base classes (e.g. time-management, slice extraction, input data checks etc.) to have a single point of truth for the starting point of the different segmentation-tool-algorithms. This would make the tools less error prone and can simplify testing (even allow automatic testing for the input management steps) (see T27800#211467).

The consequences are

  • we will not continue testing here for now, as @floca already mentioned
  • we will instead focus on the checklists (T27506: Extend checklists for segmentation view) to provide a fundamental test base to test the tools, once they are refactored (hopefully until end of November)
  • we (@floca?) will refactor the tools with

because static/dynamic*single/double*2DTools/3DTools = many test cases

in mind.

So Region Growing 3D seems to work and could probably be used for reference for other algorithms.

Since some tools have been reworked with an additional Create as new segmentation-checkbox this should be consistent. Region Growing 3D was not refactored in that regard --> T27872