Both in the contexts of segmentation and detection we have multiple feature requests within our division for a bounding box / ROI that goes beyond the Image Cropper bounding shape and is better integrated with the segmentation plugin.
The details of integration with the segmentation plugin are not yet finalized nor really discussed in depth. After a call with @m113r, I start already with a first draft of a JSON-based file format that suits all of our needs and is just flexible enough for future additions.
Requirements:
- Axis-aligned bounding boxes
- Multiple boxes per file
- Optional reference to image (in particular useful when index coordinates are used)
- Versioned to allow for future format-breaking changes
- Extent specified as origin + size
- Type of extent coordinates can be "Index", "ContIndex", or "World"
- Boxes can have attributes like name, class, confidence, comment, or expected label value for segmentations and so on...
- Support time-resolves images with adaptive boxes over time
```lang=json, name="Single box for a 3-d image
{
"FileFormat": "MITK Axis-Aligned Bounding Boxes",
"Version": 1,
"ReferenceGeometry": {
"Type": "Path",
"Reference": "../images/foo.nii.gz"
},
"Properties": {
"Comment": "Hello, world!"
},
"Frames": [
{
"ID": 0,
"Type": "Box",
"Extent": {
"Type": "Index",
"Origin": [ 10, 10, 10 ],
"Size": [ 40, 20, 10 ]
},
"Properties": {
"Name": "Tumor",
"Class": "Malignant",
"Confidence": 0.95,
"Comment": "Please segment the tumor found at this location.",
"LabelValue": 1
}
}
]
}
```
```lang=json, name="Single box partially changing over time for a 4-d image"
{
"FileFormat": "MITK Axis-Aligned Bounding Boxes",
"Version": 1,
"ReferenceGeometry": {
"Type": "Task",
"Reference": "<Task Name>"
},
"Properties": {
"Comment": "Hello, world!"
},
"Frames": [
{
"ID": 0,
"Type": "Box",
"Properties": {
"Name": "Tumor",
"Class": "Malignant",
"Comment": "Please segment the tumor found at this location."
}
}
],
"TimeSteps": [
{
"TimeStep": 0,
"Frames": [
{
"ID": 0,
"Extent": {
"Type": "Index",
"Origin": [ 10, 10, 10 ],
"Size": [ 40, 20, 10 ]
},
"Properties": {
"Confidence": 0.95,
}
}
]
},
{
"TimeStep": 1,
"Frames": [
{
"ID": 0,
"Extent": {
"Type": "Index",
"Origin": [ 15, 15, 15 ],
"Size": [ 40, 20, 10 ]
},
"Properties": {
"Confidence": 0.92,
}
}
]
}
]
}
```
The basic idea is to have a list of JSON objects for boxes and in case of a 4-d image, additional JSON objects for boxes for certain time steps. Static values of a box can be specified in the former JSON objects, dynamic values can be specified in the latter JSON objects for each time step. If both a static and dynamic value/attribute is specified, the dynamic value overrides the static default value/attribute. As the definition of a box can be split in this approach, each box must have an ID which is the same for a certain box in all time steps.