Bounding boxes / ROIs in MITK
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	kislinsk
	Jul 7 2023, 12:48 PM

Description

Both in the contexts of segmentation and detection we have multiple feature requests within our division for a bounding box / ROI that goes beyond the Image Cropper bounding shape and is better integrated with the segmentation plugin.

The details of integration with the segmentation plugin are not yet finalized nor really discussed in depth. After a call with @m113r, I start already with a first draft of a JSON-based file format that suits all of our needs and is just flexible enough for future additions.

Requirements:

Axis-aligned bounding boxes
Multiple boxes per file
Optional reference to image (in particular useful when index coordinates are used)
Versioned to allow for future format-breaking changes
Extent specified as origin + size
Type of extent coordinates can be "Index", "ContIndex", or "World"
Boxes can have attributes like name, class, confidence, comment, or expected label value for segmentations and so on...
Support time-resolves images with adaptive boxes over time

Single box for a 3-d image

{
  "FileFormat": "MITK Axis-Aligned Bounding Boxes",
  "Version": 1,
  "Geometry": {
    "Type": "Path",
    "Path": "../images/foo.nii.gz"
  },
  "Properties": {
    "Comment": "Hello, world!"
  },
  "Frames": [
    {
      "ID": 0,
      "Type": "Box",
      "Extent": {
        "Type": "Index",
        "Origin": [ 10, 10, 10 ],
        "Size": [ 40, 20, 10 ]
      },
      "Properties": {
        "Name": "Tumor",
        "Class": "Malignant",
        "Confidence": 0.95,
        "Comment": "Please segment the tumor found at this location.",
        "LabelValue": 1
      }
    }
  ]
}

Single box partially changing over time for a 4-d image

{
  "FileFormat": "MITK Axis-Aligned Bounding Boxes",
  "Version": 1,
  "Geometry": {
    "Type": "Embedded",
    "Origin": [ 0, 0, 0 ],
    "Spacing": [ 1, 1, 3 ]
   },
  "Properties": {
    "Comment": "Hello, world!"
  },
  "Frames": [
    {
      "ID": 0,
      "Type": "Box",
      "Properties": {
        "Name": "Tumor",
        "Class": "Malignant",
        "Comment": "Please segment the tumor found at this location."
      }
    }
  ],
  "TimeSteps": [
    {
      "TimeStep": 0,
      "Frames": [
        {
          "ID": 0,
          "Extent": {
            "Type": "Index",
            "Origin": [ 10, 10, 10 ],
            "Size": [ 40, 20, 10 ]
          },
          "Properties": {
            "Confidence": 0.95,
          }
        }
      ]
    },
    {
      "TimeStep": 1,
      "Frames": [
        {
          "ID": 0,
          "Extent": {
            "Type": "Index",
            "Origin": [ 15, 15, 15 ],
            "Size": [ 40, 20, 10 ]
          },
          "Properties": {
            "Confidence": 0.92,
          }
        }
      ]
    }
  ]
}

The basic idea is to have a list of JSON objects for boxes and in case of a 4-d image, additional JSON objects for boxes for certain time steps. Static values of a box can be specified in the former JSON objects, dynamic values can be specified in the latter JSON objects for each time step. If both a static and dynamic value/attribute is specified, the dynamic value overrides the static default value/attribute. As the definition of a box can be split in this approach, each box must have an ID which is the same for a certain box in all time steps.

Revisions and Commits

rMITK MITK
	Restricted Differential Revision	rMITK82e5adaab84f Add boilerplate code to new ROI and ROIIO modules
	Restricted Differential Revision	rMITK26f565fbefb5 Add ROI icon
	Restricted Differential Revision	rMITK38dfa4cc1dd6 Start IO implementation for ROIs
	Restricted Differential Revision	rMITKac98a8eaba1b Start implementation of property serialization in JSON
	Restricted Differential Revision	rMITK52f2b127c954 Add missing includes
	Restricted Differential Revision	rMITKf777dc1da48e Complete core property JSON serialization
	Restricted Differential Revision	rMITK2b0e0e3ecc29 Remove empty files
	Restricted Differential Revision	rMITKd039aaa6d65b Remove undefined method declaration
	Restricted Differential Revision	rMITKdc78bede6425 Fix serialization of wrongly used generic properties
	Restricted Differential Revision	rMITK993e4d08d5f3 Implement property list JSON serialization
	Restricted Differential Revision	rMITK2765ccf892a9 Improve syntax
	Restricted Differential Revision	rMITK7a8b1d30dbd0 Fix deserialization of LookupTableProperty
	Restricted Differential Revision	rMITK0d9f69800c54 Improve property registration for deserialization
	Restricted Differential Revision	rMITK100ce51ec1d2 Add bounds and properties to ROI data
	Restricted Differential Revision	rMITKf99f7f7b5bf3 Simplify reader
	Restricted Differential Revision	rMITK2ceefbaab308 Read bounds and properties of ROIs
	Restricted Differential Revision	rMITK40081cb32704 Implement 3-d mapper for ROIs
	Restricted Differential Revision	rMITKb22dbf4db75f Adjust bounds for ROI rendering to completely cover border pixels
	Restricted Differential Revision	rMITKdf878377fe33 Improve render style of ROIs in 3-d mapper
	Restricted Differential Revision	rMITK02a00c4ce1a6 Implement 2-d mapper for ROIs
	Restricted Differential Revision	rMITKa1ca280d3eed Disable VTK info logs by default
	Restricted Differential Revision	rMITK6a2157f2d4b1 Support line width for ROIs
	Restricted Differential Revision	rMITKba321adff886 Add base data properties to ROI file format
	Restricted Differential Revision	rMITKb08e74d8089c Use a more efficient plane cutter.
	Restricted Differential Revision	rMITK47c09dde6360 Add default font properties
	Restricted Differential Revision	rMITKf1a97bd53347 Implement captions/names for ROIs
	Restricted Differential Revision	rMITK619aba23ea36 Require IDs for ROIs
	Restricted Differential Revision	rMITK62b66c55c4c0 Implement ROI caption parsing
	Restricted Differential Revision	rMITK4cd372476c14 Make ROI elements time-resolved
	Restricted Differential Revision	rMITKc2c5571974e7 Support time steps in MITK ROI file format.
	Restricted Differential Revision	rMITKfba71b9a6c54 Refactor common local storage parts of ROI mappers into base class
	Restricted Differential Revision	rMITKf2909bed0bf5 Support time steps in 2-d ROI mapper
	Restricted Differential Revision	rMITK037d9d3674e9 Adapt ROI mapper helpers to recent changes
	Restricted Differential Revision	rMITK2884a0b45c10 Support time steps in 3-d ROI mapper
	Restricted Differential Revision	rMITK5f10e62c2d7d Implement writer for MITK ROI files
	Restricted Differential Revision	rMITKcba93e162839 Add missing include
	Restricted Differential Revision	rMITK7f92ddc3b9f7 Make nlohmann_json dependency public
	Restricted Differential Revision	rMITK1a9bf2aa9ee9 Satisfy Clang compiler
	Restricted Differential Revision	rMITKf04c0d712e8f Satisfy Clang compiler (this time for real)
	Restricted Differential Revision	rMITK09a394c66855 Add missing include
	Restricted Differential Revision	rMITK8f9396c71352 Remove unused parameter
	Restricted Differential Revision	rMITK93acff4e90c8 Add missing include
	Restricted Differential Revision	rMITK82381087fa7b Remove unused function
	Restricted Differential Revision	rMITKb303d7ca9f90 Remove unused variable
	Restricted Differential Revision	rMITK4aeabc52b019 Replace deprecated method call
	Restricted Differential Revision	rMITK43a91fba572b Add missing include
	Restricted Differential Revision	rMITK16d103b465b2 Add missing include
	Restricted Differential Revision	rMITK4f28789a8c0e Implement JSON serialization for TubeGraphProperty
	Restricted Differential Revision	rMITK5fa57aac105e Implement JSON serialization for CameraIntrinsicsProperty
	Restricted Differential Revision	rMITK392eea25e27a Add missing include
	Restricted Differential Revision	rMITK745ec315b697 Reorder includes
	Restricted Differential Revision	rMITK42b3b7f82a8d Improve detection for time-resolved ROIs
	Restricted Differential Revision	rMITK5b52d58976d9 Remove outdated sentence
	Restricted Differential Revision	rMITK9b3a81340e45 Add ROI documentation
	Restricted Differential Revision	rMITK3342cc69bb7a Reorder includes
	Restricted Differential Revision	rMITK52c74468c53d Reorder includes
	Restricted Differential Revision	rMITKbb2783265f11 Fix bounding shape handle size
	Restricted Differential Revision	rMITK0ce20cf8f9f9 Make inclusion of ctkAbstractPythonManager.h compatible to nlohmann_json
	Restricted Differential Revision	rMITK7a807cbe2f44 Add icon for GeometryData
	Restricted Differential Revision	rMITKeb354f334d4c Change bounding shape handles from spheres to cubes
	Restricted Differential Revision	rMITK0c7ca73b00b3 Fix bounding shape mapper property connection
	Restricted Differential Revision	rMITK2018a9c821c9 Consistently use "bounding box" instead of "bounding shape"
	Restricted Differential Revision	rMITKf1cec65c81b1 Add const overload of GetVtkMatrix()
	Restricted Differential Revision	rMITK405c6b86c575 Add "Convert to ROI" action to GeometryData nodes
	Restricted Differential Revision	rMITK526f0f0bca85 Add conversion from GeometryData to ROI
	Restricted Differential Revision	rMITK3bbbbec20282 Only save name if set
	Restricted Differential Revision	rMITKe60d3c62a4e0 Fix missing returns
	Restricted Differential Revision	rMITK6ab482816510 Add MITK ROI file format documentation
	Restricted Differential Revision	rMITK01f785485a6b Warn against use of experimental ROI class.
	Restricted Differential Revision	rMITKb54bcd101ae7 Add TimeStepType overloads to property methods

Related Objects

Mentioned In: T30291: 2023 Week 48 (Very Late November)
rMITK7c2a1f4ff42e: Merge branch 'bugfix/T30155-FixUnitTests' into develop
rMITKdcce84a018c1: Merge branch 'develop' into feature/T30155-ROI
rMITK226bcc35c217: Merge branch 'feature/T30155-ROI' into develop

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

kislinsk triaged this task as High priority.Jul 7 2023, 12:48 PM

Thank you for the first draft. I had the following first thoughts:

Is it realy helpfull needed to support index in the file. Why not just world coordinates and it is a thing for the serializing application to convert if needed. If we allow index, the geometry should be stored directly here to make it always interpretable information.
There should be a posibility to store meta properties on top level (as we also do e.g. in the label set)
For me RefImage is already one of those meta properties.
How is RefImage defined? Just a file? Simple/easy to use but not very robust. We might keep that option, but I think it would be better to use support our PropertyReleations and e.g. use where possible DICOM Tags to encode references.
We should allign the json format used to store properties here and in our segmentations... (therefore i.a. I would not introdice a new term (attributes) when there is a sementically equal term allready use in other places (properties)

kislinsk updated the task description. (Show Details)Jul 8 2023, 8:21 AM

Thanks for the feedback. 🙏

Yes, all concrete requirements in that regard that we currently have from our stake holders are index coordinates and in case of @m113r in addition continuous index coordinates as that's the output type of the net. Hence, enforcing world coordinates is in my opinion an unnecessary burden for adoption. But, what I didn't mention so far is certain assumptions/rules that need to be fulfilled. In case of index coordinates the otherwise optional RefImage is mandatory or at least a strong warning must be issued in case of its absence. Otherwise you would be right and we would need to include geometry information which is a can of worms all of us want to avoid in the scope of this file format.
Done.
RefImage is essential in most cases (see 1) and as integral part I vote for having it separate from properties even though it is optional.
That is up for discussion and hence I intentionally wasn't specific on the RefImage value. While we still need to talk about it I now made it a little bit more specific already by splitting it up into a Type (e. g. "Path" or "Task") and the actual Ref string. Maybe that's already enough flexibility for all future cases.
Done.

@kislinsk thank you for the explanations. They do make sense.

On top of that I would propose to things for improving the current specification:

Change "RefImage" to "RefGeometry"

"RefGeometry": {
  "Type": "Task",
  "Ref": "<Task Name>"
 },

Reason: in the end it is a ball about geometries and this would give us more freedom for having different types that represent the geometry.

Introduce the type Geometry for RefGeometry

Reason: then you could directly encode in the format the reference geometry that should be used. I think that is a good compromise between offering data scientists, who have their pipeline and want just as easy as possible to create a valid file, a simple route by just reference an image that defines the geometry, while implementations that strive for robustness and have no problem with directly encode the geometry in the file (like I would recommend for our MITK implementation).

Name Boxes mor generic. E.g. Frames or so

Reasion: That would allow to introduce later also other forms like spheres for indicating the bounds without having a semantic conflict in the terminology as they are no boxes.

kislinsk updated the task description. (Show Details)Jul 10 2023, 10:43 AM

Done
If the geometry is to be defined right in the file itself, I vote against a ReferenceGeometry type of "Geometry" but rather actually a "Geometry" JSON object. So it is either having a reference to something else that defines the geometry or specifying the geometry right in the file without any redirections.
Done, also added "Type" to "Frames", which is "Box" for, well... boxes.

kislinsk updated the task description. (Show Details)Jul 10 2023, 1:04 PM

👍
I would propose to also standardizeand align some of the properties for the frames to be either in sync with MITK terminology or with DICOM.

currently drafted	DICOM	MITK	Meaning
Name or class?	SegmentLabel	Label	Human readable class name, i.a. used to denote the different classes in semantic segmentation; e.g. "metastasis" or "occlusions")
?	TrackingID	TrackingID	human readable instance name, i.a. used to denote and differentiate instances in instant segmentation; e.g. "Metastasis 1"
?	TrackingUID	TrackingUID	unique ID to identify instances in instant segmentation cross different images and points in time
LabelValue	Segment Number	LabelValue	unique value that identifies a certain instance in the data structure. Remark: do we really need that in detection as there is already the ID?
?	Recommended Display CIELab Value	Color	color that should be used for displaying the structure
?	N/a	Locked	indicating the lock state of the structure in the application
?	N/a	Visible	indicating the visibility of the structure in the application
?	N/a	Locked	indicating the lock state of the structure in the application
Comment	Segment Description	Description