diff --git a/README.md b/README.md
index 2b49617..5f40083 100644
--- a/README.md
+++ b/README.md
@@ -1,166 +1,173 @@
Copyright © German Cancer Research Center (DKFZ), Division of Medical Image Computing (MIC).
Please make sure that your usage of this code is in compliance with the code license.
## Release Notes
0.0.2 - Now with updated torch (>=1.3) and torchvision (>=0.4) dependencies and on python-3.7.5 basis.
## Introduction
This repository holds the code framework used in the paper Reg R-CNN: Lesion Detection and Grading under Noisy Labels [1].
The framework is a fork of MIC's [medicaldetectiontoolkit](https://github.com/MIC-DKFZ/medicaldetectiontoolkit) with added regression
capabilities.
As below figure shows, the regression capability allows for the preservation of ordinal relations in the training signal as opposed to a standard categorical classification loss like the cross entropy loss (see publication for details).
Network Reg R-CNN is a version of Mask R-CNN [2] but with a regressor in place of the object-class head (see figure below). In this scenario, the first stage makes foreground (fg) vs background (bg) detections, then the regression head determines the class on an ordinal scale. Consequently, prediction confidence scores are taken from the first stage as opposed to the head in the original Mask R-CNN.
In the configs file of a data set in the framework, you may set attribute self.prediction_tasks = ["task"] to a value "task" from ["class", "regression_bin", "regression"]. "class" produces the same behavior as the original framework, i.e., standard object-detection behavior. "regression" on the other hand, swaps the class head of network Mask R-CNN [2] for a regression head. Consequently, objects are identified as fg/bg and then the class is decided by the regressor. For the sake of comparability, "regression_bin" produces a similar behavior but with a classification head. Both methods should be evaluated with the (implemented) Average Viewpoint Precision instead of only Average Precision.
Below you will found a description of the general framework operations and handling. Basic framework functionality and description are for the most part identical to the original [medicaldetectiontoolkit](https://github.com/MIC-DKFZ/medicaldetectiontoolkit).
[1] Ramien, Gregor et al., "Reg R-CNN: Lesion Detection and Grading under Noisy Labels". In: UNSURE Workshop at MICCAI, 2019.
[2] He, Kaiming, et al. "Mask R-CNN" ICCV, 2017
## Overview
This is a comprehensive framework for object detection featuring:
- 2D + 3D implementations of common object detectors: e.g., Mask R-CNN [2], Retina Net [3], Retina U-Net [4].
- Modular and light-weight structure ensuring sharing of all processing steps (incl. backbone architecture) for comparability of models.
- training with bounding box and/or pixel-wise annotations.
- dynamic patching and tiling of 2D + 3D images (for training and inference).
- weighted consolidation of box predictions across patch-overlaps, ensembles, and dimensions [4] or standard non-maximum suppression.
- monitoring + evaluation simultaneously on object and patient level.
- 2D + 3D output visualizations.
- integration of COCO mean average precision metric [5].
- integration of MIC-DKFZ batch generators for extensive data augmentation [6].
- possible evaluation of instance segmentation and/or semantic segmentation by dice scores.
[3] Lin, Tsung-Yi, et al. "Focal Loss for Dense Object Detection" TPAMI, 2018.
[4] Jaeger, Paul et al. "Retina U-Net: Embarrassingly Simple Exploitation
of Segmentation Supervision for Medical Object Detection" , 2018
[5] https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py
[6] https://github.com/MIC-DKFZ/batchgenerators
## How to cite this code
Please cite the Reg R-CNN publication [1] or the original publication [4] depending on what features you use.
## Installation
Setup package in virtual environment
```
git clone https://github.com/MIC-DKFZ/RegRCNN.git.
cd RegRCNN
virtualenv -p python3.7 regrcnn_env
source regrcnn_env/bin/activate
python setup.py install
```
This framework uses two custom mixed C++/CUDA extensions: Non-maximum suppression (NMS) and RoIAlign. Both are adapted from the original pytorch extensions (under torchvision.ops.boxes and ops.roialign).
The extensions are automatically compiled from the provided source files under RegRCNN/custom_extensions with above setup.py.
Note: If you'd like to import the raw extensions (not the wrapper modules), be sure to import torch first.
+Alternatively, you may install the framework via pip by replacing the last line above (python setup.py install) by:
+```
+pip install .
+pip install -e ./custom_extensions/nms
+pip install -e ./custom_extensions/roi_align
+```
+
## Prepare the Data
This framework is meant for you to be able to train models on your own data sets.
In order to include a data set in the framework, create a new folder in RegRCNN/datasets, for instance "example_data". Your data set needs to have a config file in the style of the provided example data sets "lidc" and "toy". It also needs a data loader meeting the same requirements as the provided examples. Likely, you will also need a preprocessing script that transforms your data (once per data set creation, i.e., not a repetitive operation) into a suitable and easily processable format.
Important requirements:
- The framework expects numpy arrays as data and segmentation ground truth input.
- Segmentations need to be suited for object detection, i.e., Regions of Interest (RoIs) need to be marked by integers (RoI-ID) in the segmentation volume (0 is background). Corresponding properties of a RoI, e.g., the "class_targets" need to be provided in a separate array or list with (RoI-ID - 1) corresponding to the index of the property in the list (-1 due to zero-indexing). Example: A data volume contains two RoIs. The second RoI is marked in the segmentation by number 2. The "class_targets" info associated with the data volume holds the list [2, 3]. Hence, RoI-ID 2 is assigned class 3.
- This framework uses a modified version of MIC's batchgenerators' segmentation-to-bounding-box conversion tool. In this version, "class_targets", i.e., object classes start at 1, 0 is reserved for background. Thus, if you use "ConvertSegToBoundingBoxCoordinates" classes in your preprocessed data need to start at 1, not 0.
Two example data loaders are provided in RegRCNN/datasets. The way I load data is to have a preprocessing script, which after preprocessing saves the data of whatever data type into numpy arrays (this is just run once). During training / testing, the data loader then loads these numpy arrays dynamically. Please note the data input side is meant to be customized by you according to your own needs and the provided data loaders are merely examples: LIDC has a powerful data loader that handles 2D/3D inputs and is optimized for patch-based training and inference. Due to the large data volumes of LIDC, this loader is slow. The provided toy data set, however, is light weight and a good starting point to get familiar with the framework. It is fully creatable from scratch within a few minutes with RegRCNN/datasets/toy/generate_toys.py.
## Execute
1. Set I/O paths, model and training specifics in the configs file: RegRCNN/datasets/_your_dataset_/configs.py
2. i) Train the model:
```
python exec.py --mode train --dataset_name your_dataset --exp_dir path/to/experiment/directory
```
This copies snapshots of configs and model to the specified exp_dir, where all outputs will be saved. By default, the data is split into 60% training and 20% validation and 20% testing data to perform a 5-fold cross validation (can be changed to hold-out test set in configs) and all folds will be trained iteratively. In order to train a single fold, specify it using the folds arg:
```
python exec.py --folds 0 1 2 .... # specify any combination of folds [0-configs.n_cv_splits]
```
ii) Alternatively, train and test consecutively:
```
python exec.py --mode train_test --dataset_name your_dataset --exp_dir path/to/experiment/directory
```
3. Run inference:
```
python exec.py --mode test --exp_dir path/to/experiment/directory
```
This runs the prediction pipeline and saves all results to exp_dir.
4. Additional settings:
- Check the args parser in exec.py to see which arguments and modes are available.
- E.g., you may pass ```-d``` or ```--dev``` to enable a short development run of the whole train_test procedure (small batch size, only one epoch, two folds, one test patient, etc.).
## Models
This framework features models explored in [4] (implemented in 2D + 3D): The proposed Retina U-Net, a simple but effective Architecture fusing state-of-the-art semantic segmentation with object detection,
also implementations of prevalent object detectors, such as Mask R-CNN, Faster R-CNN+ (Faster R-CNN w\ RoIAlign), Retina Net, Detection U-Net (a U-Net like segmentation architecture with heuristics for object detection.)
## Training annotations
This framework features training with pixelwise and/or bounding box annotations. To overcome the issue of box coordinates in
data augmentation, we feed the annotation masks through data augmentation (create a pseudo mask, if only bounding box annotations provided) and draw the boxes afterwards.
The framework further handles two types of pixel-wise annotations:
1. A label map with individual ROIs identified by increasing label values, accompanied by a vector containing in each position the class target for the lesion with the corresponding label (for this mode set get_rois_from_seg_flag = False when calling ConvertSegToBoundingBoxCoordinates in your Data Loader). This is usual use case as explained in section "Prepare the data".
2. A binary label map. There is only one foreground class and single lesions are not identified. All lesions have the same class target (foreground). In this case the data loader runs a Connected Component Labelling algorithm to create processable lesion - class target pairs on the fly (for this mode set get_rois_from_seg_flag = True when calling ConvertSegToBoundingBoxCoordinates in your data loader).
## Prediction pipeline
This framework provides an inference module, which automatically handles patching of inputs, and tiling, ensembling, and weighted consolidation of output predictions:
## Consolidation of predictions
### Weighted Box Clustering
Multiple predictions of the same image (from test time augmentations, tested epochs and overlapping patches), result in a high amount of boxes (or cubes), which need to be consolidated. In semantic segmentation, the final output would typically be obtained by averaging every pixel over all predictions. As described in [4], **weighted box clustering** (WBC) does this for box predictions:
To enable WBC, set self.clustering = "wbc" in your configs file.
### Non-Maximum Suppression
Test-time predictions can alternatively be aggregated with standard non-maximum suppression. In your configs file, simply set self.clustering = "nms" instead of "wbc".
As a further alternative you may also choose no test-time aggregation by setting self.clustering = None.
## Visualization / Monitoring
In opposition to the original framework, this fork uses tensorboard for monitoring training and validation progress. Since, for now, the framework cannot easily be updated to pytorch >= 1.x, we need third-party package [tensorboardX](https://github.com/lanpa/tensorboardX) to use tensorboard with pytorch.
You can set an applicable choice of implemented metrics like "ap" for Average Precision or "auc" for patient-level ROC-AUC in the configs under self.metrics = [...]. Metrics are then evaluated by evaluator.py and recorded in monitor_metrics. logger.metrics2tboard sends monitor_metrics to your tensorboard logfiles at the end of each epoch.
You need to separately start a virtual tensorboard server, pass it your experiment directory (or directories, but it crashes if its more than ~5 experiments) and navigate to the server address. (You can also read up on tensoardboard usage in the original documentation).
### Example:
1. Activate your virtualenv where tensorboard is installed.
2. Start tensorboard server. For instance, your experiment directory is
_yourexpdir_:
```tensorboard --port 6007 --logdir yourexpdir```
3. Navigate to ```localhost:6007``` in your browser.
### Output monitoring
For qualitative monitoring, example plots are saved to _yourexpdir_/plots for training and validation and _yourexpdir_/test/example_plots for testing. Note, that test-time example plots may contain unconsolidated predictions over test-time augmentations, thereby possibly showing many overlapping and/or noisy predictions. You may adapt/use separate file RegRCNN/inference_analysis.py to create clean and nice plots of (consolidated) test-time predictions.
## Balancing Mechanism of Example Data Loader
The data loaders of the provided example data sets employ a custom mechanism with the goal of assembling target-balanced batches or training sequences. I.e., the amount of examples shown per target class should be near balance.
The mechanism creates a sampling-likelihood distribution, as shown below, over all available patients (PIDs). At batch generation, some patients are drawn according to this distribution, others are drawn completely randomly (according to a uniform distribution across all patients). The ratio of uniformly and target-dependently drawn patients is set in your configs file by configs.batch_random_ratio. configs.balance_target determines which targets are considered for the balancing distribution.
While the balancing distribution assigns probability 0 to empty patients (contains no object of desired target kind), the random ratio allows for inclusion of those empty patients in the training exposure. Experience has shown, that showing at least one foreground example in each batch is most critical, other properties have less impact.
## Unittests
unittests.py contains some verification and testing procedures, which, however, need you to adjust paths in the TestCase classes before execution. Tests can be used, for instance, to verify if your cross-validation folds have been created correctly, or if separate experiments have the same fold splits.
# License
This framework is published under the [APACHE 2.0 License](https://github.com/MIC-DKFZ/RegRCNN/blob/master/LICENSE)
\ No newline at end of file
diff --git a/utils/exp_utils.py b/utils/exp_utils.py
index fec49db..b6e93f0 100644
--- a/utils/exp_utils.py
+++ b/utils/exp_utils.py
@@ -1,652 +1,651 @@
#!/usr/bin/env python
# Copyright 2019 Division of Medical Image Computing, German Cancer Research Center (DKFZ).
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
# import plotting as plg
import sys
import os
import subprocess
import threading
import pickle
import importlib.util
import psutil
import time
import logging
from torch.utils.tensorboard import SummaryWriter
from collections import OrderedDict
import numpy as np
import pandas as pd
import torch
def import_module(name, path):
"""
correct way of importing a module dynamically in python 3.
:param name: name given to module instance.
:param path: path to module.
:return: module: returned module instance.
"""
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def save_obj(obj, name):
"""Pickle a python object."""
with open(name + '.pkl', 'wb') as f:
pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)
def load_obj(file_path):
with open(file_path, 'rb') as handle:
return pickle.load(handle)
def IO_safe(func, *args, _tries=5, _raise=True, **kwargs):
""" Wrapper calling function func with arguments args and keyword arguments kwargs to catch input/output errors
on cluster.
:param func: function to execute (intended to be read/write operation to a problematic cluster drive, but can be
any function).
:param args: positional args of func.
:param kwargs: kw args of func.
:param _tries: how many attempts to make executing func.
"""
for _try in range(_tries):
try:
return func(*args, **kwargs)
except OSError as e: # to catch cluster issues with network drives
if _raise:
raise e
else:
print("After attempting execution {} time{}, following error occurred:\n{}".format(_try + 1,
"" if _try == 0 else "s",
e))
continue
def query_nvidia_gpu(device_id, d_keyword=None, no_units=False):
"""
:param device_id:
:param d_keyword: -d, --display argument (keyword(s) for selective display), all are selected if None
:return: dict of gpu-info items
"""
cmd = ['nvidia-smi', '-i', str(device_id), '-q']
if d_keyword is not None:
cmd += ['-d', d_keyword]
outp = subprocess.check_output(cmd).strip().decode('utf-8').split("\n")
outp = [x for x in outp if len(x) > 0]
headers = [ix for ix, item in enumerate(outp) if len(item.split(":")) == 1] + [len(outp)]
out_dict = {}
for lix, hix in enumerate(headers[:-1]):
head = outp[hix].strip().replace(" ", "_").lower()
out_dict[head] = {}
for lix2 in range(hix, headers[lix + 1]):
try:
key, val = [x.strip().lower() for x in outp[lix2].split(":")]
if no_units:
val = val.split()[0]
out_dict[head][key] = val
except:
pass
return out_dict
class CombinedPrinter(object):
"""combined print function.
prints to logger and/or file if given, to normal print if non given.
"""
def __init__(self, logger=None, file=None):
if logger is None and file is None:
self.out = [print]
elif logger is None:
self.out = [file.write]
elif file is None:
self.out = [logger.info]
else:
self.out = [logger.info, file.write]
def __call__(self, string):
for fct in self.out:
fct(string)
class Nvidia_GPU_Logger(object):
def __init__(self):
self.count = None
def get_vals(self):
cmd = ['nvidia-settings', '-t', '-q', 'GPUUtilization']
gpu_util = subprocess.check_output(cmd).strip().decode('utf-8').split(",")
gpu_util = dict([f.strip().split("=") for f in gpu_util])
cmd[-1] = 'UsedDedicatedGPUMemory'
gpu_used_mem = subprocess.check_output(cmd).strip().decode('utf-8')
current_vals = {"gpu_mem_alloc": gpu_used_mem, "gpu_graphics_util": int(gpu_util['graphics']),
"gpu_mem_util": gpu_util['memory'], "time": time.time()}
return current_vals
def loop(self, interval):
i = 0
while True:
self.get_vals()
self.log["time"].append(time.time())
self.log["gpu_util"].append(self.current_vals["gpu_graphics_util"])
if self.count is not None:
i += 1
if i == self.count:
exit(0)
time.sleep(self.interval)
def start(self, interval=1.):
self.interval = interval
self.start_time = time.time()
self.log = {"time": [], "gpu_util": []}
if self.interval is not None:
thread = threading.Thread(target=self.loop)
thread.daemon = True
thread.start()
class CombinedLogger(object):
"""Combine console and tensorboard logger and record system metrics.
"""
def __init__(self, name, log_dir, server_env=True, fold="", sysmetrics_interval=2):
self.pylogger = logging.getLogger(name)
self.tboard = SummaryWriter(log_dir=log_dir)
self.times = {}
self.fold = fold
self.pylogger.setLevel(logging.DEBUG)
self.log_file = os.path.join(log_dir, 'exec.log')
self.pylogger.addHandler(logging.FileHandler(self.log_file))
if not server_env:
self.pylogger.addHandler(ColorHandler())
else:
self.pylogger.addHandler(logging.StreamHandler())
self.pylogger.propagate = False
# monitor system metrics (cpu, mem, ...)
if not server_env and sysmetrics_interval > 0:
self.sysmetrics = pd.DataFrame(
columns=["global_step", "rel_time", r"CPU (%)", "mem_used (GB)", r"mem_used (%)",
r"swap_used (GB)", r"gpu_utilization (%)"], dtype="float16")
for device in range(torch.cuda.device_count()):
self.sysmetrics[
"mem_allocd (GB) by torch on {:10s}".format(torch.cuda.get_device_name(device))] = np.nan
self.sysmetrics[
"mem_cached (GB) by torch on {:10s}".format(torch.cuda.get_device_name(device))] = np.nan
self.sysmetrics_start(sysmetrics_interval)
pass
else:
print("NOT logging sysmetrics")
def __getattr__(self, attr):
"""delegate all undefined method requests to objects of
this class in order pylogger, tboard (first find first serve).
E.g., combinedlogger.add_scalars(...) should trigger self.tboard.add_scalars(...)
"""
for obj in [self.pylogger, self.tboard]:
if attr in dir(obj):
return getattr(obj, attr)
print("logger attr not found")
#raise AttributeError("CombinedLogger has no attribute {}".format(attr))
def time(self, name, toggle=None):
"""record time-spans as with a stopwatch.
:param name:
:param toggle: True^=On: start time recording, False^=Off: halt rec. if None determine from current status.
:return: either start-time or last recorded interval
"""
if toggle is None:
if name in self.times.keys():
toggle = not self.times[name]["toggle"]
else:
toggle = True
if toggle:
if not name in self.times.keys():
self.times[name] = {"total": 0, "last": 0}
elif self.times[name]["toggle"] == toggle:
self.info("restarting running stopwatch")
self.times[name]["last"] = time.time()
self.times[name]["toggle"] = toggle
return time.time()
else:
if toggle == self.times[name]["toggle"]:
self.info("WARNING: tried to stop stopped stop watch: {}.".format(name))
self.times[name]["last"] = time.time() - self.times[name]["last"]
self.times[name]["total"] += self.times[name]["last"]
self.times[name]["toggle"] = toggle
return self.times[name]["last"]
def get_time(self, name=None, kind="total", format=None, reset=False):
"""
:param name:
:param kind: 'total' or 'last'
:param format: None for float, "hms"/"ms" for (hours), mins, secs as string
:param reset: reset time after retrieving
:return:
"""
if name is None:
times = self.times
if reset:
self.reset_time()
return times
else:
if self.times[name]["toggle"]:
self.time(name, toggle=False)
time = self.times[name][kind]
if format == "hms":
m, s = divmod(time, 60)
h, m = divmod(m, 60)
time = "{:d}h:{:02d}m:{:02d}s".format(int(h), int(m), int(s))
elif format == "ms":
m, s = divmod(time, 60)
time = "{:02d}m:{:02d}s".format(int(m), int(s))
if reset:
self.reset_time(name)
return time
def reset_time(self, name=None):
if name is None:
self.times = {}
else:
del self.times[name]
def sysmetrics_update(self, global_step=None):
if global_step is None:
global_step = time.strftime("%x_%X")
mem = psutil.virtual_memory()
mem_used = (mem.total - mem.available)
gpu_vals = self.gpu_logger.get_vals()
rel_time = time.time() - self.sysmetrics_start_time
self.sysmetrics.loc[len(self.sysmetrics)] = [global_step, rel_time,
psutil.cpu_percent(), mem_used / 1024 ** 3,
mem_used / mem.total * 100,
psutil.swap_memory().used / 1024 ** 3,
int(gpu_vals['gpu_graphics_util']),
*[torch.cuda.memory_allocated(d) / 1024 ** 3 for d in
range(torch.cuda.device_count())],
*[torch.cuda.memory_cached(d) / 1024 ** 3 for d in
range(torch.cuda.device_count())]
]
return self.sysmetrics.loc[len(self.sysmetrics) - 1].to_dict()
def sysmetrics2tboard(self, metrics=None, global_step=None, suptitle=None):
tag = "per_time"
if metrics is None:
metrics = self.sysmetrics_update(global_step=global_step)
tag = "per_epoch"
if suptitle is not None:
suptitle = str(suptitle)
elif self.fold != "":
suptitle = "Fold_" + str(self.fold)
if suptitle is not None:
self.tboard.add_scalars(suptitle + "/System_Metrics/" + tag,
{k: v for (k, v) in metrics.items() if (k != "global_step"
and k != "rel_time")}, global_step)
def sysmetrics_loop(self):
try:
os.nice(-19)
self.info("Logging system metrics with superior process priority.")
except:
self.info("Logging system metrics without superior process priority.")
while True:
metrics = self.sysmetrics_update()
self.sysmetrics2tboard(metrics, global_step=metrics["rel_time"])
# print("thread alive", self.thread.is_alive())
time.sleep(self.sysmetrics_interval)
def sysmetrics_start(self, interval):
if interval is not None and interval > 0:
self.sysmetrics_interval = interval
self.gpu_logger = Nvidia_GPU_Logger()
self.sysmetrics_start_time = time.time()
self.thread = threading.Thread(target=self.sysmetrics_loop)
self.thread.daemon = True
self.thread.start()
def sysmetrics_save(self, out_file):
self.sysmetrics.to_pickle(out_file)
def metrics2tboard(self, metrics, global_step=None, suptitle=None):
"""
:param metrics: {'train': dataframe, 'val':df}, df as produced in
evaluator.py.evaluate_predictions
"""
# print("metrics", metrics)
if global_step is None:
global_step = len(metrics['train'][list(metrics['train'].keys())[0]]) - 1
if suptitle is not None:
suptitle = str(suptitle)
else:
suptitle = "Fold_" + str(self.fold)
for key in ['train', 'val']:
# series = {k:np.array(v[-1]) for (k,v) in metrics[key].items() if not np.isnan(v[-1]) and not 'Bin_Stats' in k}
loss_series = {}
unc_series = {}
bin_stat_series = {}
mon_met_series = {}
for tag, val in metrics[key].items():
val = val[-1] # maybe remove list wrapping, recording in evaluator?
if 'bin_stats' in tag.lower() and not np.isnan(val):
bin_stat_series["{}".format(tag.split("/")[-1])] = val
elif 'uncertainty' in tag.lower() and not np.isnan(val):
unc_series["{}".format(tag)] = val
elif 'loss' in tag.lower() and not np.isnan(val):
loss_series["{}".format(tag)] = val
elif not np.isnan(val):
mon_met_series["{}".format(tag)] = val
self.tboard.add_scalars(suptitle + "/Binary_Statistics/{}".format(key), bin_stat_series, global_step)
self.tboard.add_scalars(suptitle + "/Uncertainties/{}".format(key), unc_series, global_step)
self.tboard.add_scalars(suptitle + "/Losses/{}".format(key), loss_series, global_step)
self.tboard.add_scalars(suptitle + "/Monitor_Metrics/{}".format(key), mon_met_series, global_step)
self.tboard.add_scalars(suptitle + "/Learning_Rate", metrics["lr"], global_step)
return
def batchImgs2tboard(self, batch, results_dict, cmap, boxtype2color, img_bg=False, global_step=None):
raise NotImplementedError("not up-to-date, problem with importing plotting-file, torchvision dependency.")
if len(batch["seg"].shape) == 5: # 3D imgs
slice_ix = np.random.randint(batch["seg"].shape[-1])
seg_gt = plg.to_rgb(batch['seg'][:, 0, :, :, slice_ix], cmap)
seg_pred = plg.to_rgb(results_dict['seg_preds'][:, 0, :, :, slice_ix], cmap)
mod_img = plg.mod_to_rgb(batch["data"][:, 0, :, :, slice_ix]) if img_bg else None
elif len(batch["seg"].shape) == 4:
seg_gt = plg.to_rgb(batch['seg'][:, 0, :, :], cmap)
seg_pred = plg.to_rgb(results_dict['seg_preds'][:, 0, :, :], cmap)
mod_img = plg.mod_to_rgb(batch["data"][:, 0]) if img_bg else None
else:
raise Exception("batch content has wrong format: {}".format(batch["seg"].shape))
# from here on only works in 2D
seg_gt = np.transpose(seg_gt, axes=(0, 3, 1, 2)) # previous shp: b,x,y,c
seg_pred = np.transpose(seg_pred, axes=(0, 3, 1, 2))
seg = np.concatenate((seg_gt, seg_pred), axis=0)
# todo replace torchvision (tv) dependency
seg = tv.utils.make_grid(torch.from_numpy(seg), nrow=2)
self.tboard.add_image("Batch seg, 1st col: gt, 2nd: pred.", seg, global_step=global_step)
if img_bg:
bg_img = np.transpose(mod_img, axes=(0, 3, 1, 2))
else:
bg_img = seg_gt
box_imgs = plg.draw_boxes_into_batch(bg_img, results_dict["boxes"], boxtype2color)
box_imgs = tv.utils.make_grid(torch.from_numpy(box_imgs), nrow=4)
self.tboard.add_image("Batch bboxes", box_imgs, global_step=global_step)
return
def __del__(self): # otherwise might produce multiple prints e.g. in ipython console
for hdlr in self.pylogger.handlers:
hdlr.close()
self.pylogger.handlers = []
del self.pylogger
self.tboard.close()
def get_logger(exp_dir, server_env=False, sysmetrics_interval=2):
log_dir = os.path.join(exp_dir, "logs")
logger = CombinedLogger('Reg R-CNN', os.path.join(log_dir, "tboard"), server_env=server_env,
sysmetrics_interval=sysmetrics_interval)
print("logging to {}".format(logger.log_file))
return logger
def prep_exp(dataset_path, exp_path, server_env, use_stored_settings=True, is_training=True):
"""
I/O handling, creating of experiment folder structure. Also creates a snapshot of configs/model scripts and copies them to the exp_dir.
This way the exp_dir contains all info needed to conduct an experiment, independent to changes in actual source code. Thus, training/inference of this experiment can be started at anytime.
Therefore, the model script is copied back to the source code dir as tmp_model (tmp_backbone).
Provides robust structure for cloud deployment.
:param dataset_path: path to source code for specific data set. (e.g. medicaldetectiontoolkit/lidc_exp)
:param exp_path: path to experiment directory.
:param server_env: boolean flag. pass to configs script for cloud deployment.
:param use_stored_settings: boolean flag. When starting training: If True, starts training from snapshot in existing
experiment directory, else creates experiment directory on the fly using configs/model scripts from source code.
:param is_training: boolean flag. distinguishes train vs. inference mode.
:return: configs object.
"""
if is_training:
if use_stored_settings:
cf_file = import_module('cf', os.path.join(exp_path, 'configs.py'))
cf = cf_file.Configs(server_env)
# in this mode, previously saved model and backbone need to be found in exp dir.
if not os.path.isfile(os.path.join(exp_path, 'model.py')) or \
not os.path.isfile(os.path.join(exp_path, 'backbone.py')):
raise Exception(
"Selected use_stored_settings option but no model and/or backbone source files exist in exp dir.")
cf.model_path = os.path.join(exp_path, 'model.py')
cf.backbone_path = os.path.join(exp_path, 'backbone.py')
else: # this case overwrites settings files in exp dir, i.e., default_configs, configs, backbone, model
- if not os.path.exists(exp_path):
- os.mkdir(exp_path)
+ os.makedirs(exp_path, exist_ok=True)
# run training with source code info and copy snapshot of model to exp_dir for later testing (overwrite scripts if exp_dir already exists.)
subprocess.call('cp {} {}'.format('default_configs.py', os.path.join(exp_path, 'default_configs.py')),
shell=True)
subprocess.call(
'cp {} {}'.format(os.path.join(dataset_path, 'configs.py'), os.path.join(exp_path, 'configs.py')),
shell=True)
cf_file = import_module('cf_file', os.path.join(dataset_path, 'configs.py'))
cf = cf_file.Configs(server_env)
subprocess.call('cp {} {}'.format(cf.model_path, os.path.join(exp_path, 'model.py')), shell=True)
subprocess.call('cp {} {}'.format(cf.backbone_path, os.path.join(exp_path, 'backbone.py')), shell=True)
if os.path.isfile(os.path.join(exp_path, "fold_ids.pickle")):
subprocess.call('rm {}'.format(os.path.join(exp_path, "fold_ids.pickle")), shell=True)
else: # testing, use model and backbone stored in exp dir.
cf_file = import_module('cf', os.path.join(exp_path, 'configs.py'))
cf = cf_file.Configs(server_env)
cf.model_path = os.path.join(exp_path, 'model.py')
cf.backbone_path = os.path.join(exp_path, 'backbone.py')
cf.exp_dir = exp_path
cf.test_dir = os.path.join(cf.exp_dir, 'test')
cf.plot_dir = os.path.join(cf.exp_dir, 'plots')
if not os.path.exists(cf.test_dir):
os.mkdir(cf.test_dir)
if not os.path.exists(cf.plot_dir):
os.mkdir(cf.plot_dir)
cf.experiment_name = exp_path.split("/")[-1]
cf.dataset_name = dataset_path
cf.server_env = server_env
cf.created_fold_id_pickle = False
return cf
class ModelSelector:
'''
saves a checkpoint after each epoch as 'last_state' (can be loaded to continue interrupted training).
saves the top-k (k=cf.save_n_models) ranked epochs. In inference, predictions of multiple epochs can be ensembled
to improve performance.
'''
def __init__(self, cf, logger):
self.cf = cf
self.saved_epochs = [-1] * cf.save_n_models
self.logger = logger
def run_model_selection(self, net, optimizer, monitor_metrics, epoch):
"""rank epoch via weighted mean from self.cf.model_selection_criteria: {criterion : weight}
:param net:
:param optimizer:
:param monitor_metrics:
:param epoch:
:return:
"""
crita = self.cf.model_selection_criteria # shorter alias
non_nan_scores = {}
for criterion in crita.keys():
# exclude first entry bc its dummy None entry
non_nan_scores[criterion] = [0 if (ii is None or np.isnan(ii)) else ii for ii in
monitor_metrics['val'][criterion]][1:]
n_epochs = len(non_nan_scores[criterion])
epochs_scores = []
for e_ix in range(n_epochs):
epochs_scores.append(np.sum([weight * non_nan_scores[criterion][e_ix] for
criterion, weight in crita.items()]) / len(crita.keys()))
# ranking of epochs according to model_selection_criterion
epoch_ranking = np.argsort(epochs_scores)[::-1] + 1 # epochs start at 1
# if set in configs, epochs < min_save_thresh are discarded from saving process.
epoch_ranking = epoch_ranking[epoch_ranking >= self.cf.min_save_thresh]
# check if current epoch is among the top-k epchs.
if epoch in epoch_ranking[:self.cf.save_n_models]:
if self.cf.server_env:
IO_safe(torch.save, net.state_dict(),
os.path.join(self.cf.fold_dir, '{}_best_params.pth'.format(epoch)))
# save epoch_ranking to keep info for inference.
IO_safe(np.save, os.path.join(self.cf.fold_dir, 'epoch_ranking'), epoch_ranking[:self.cf.save_n_models])
else:
torch.save(net.state_dict(), os.path.join(self.cf.fold_dir, '{}_best_params.pth'.format(epoch)))
np.save(os.path.join(self.cf.fold_dir, 'epoch_ranking'), epoch_ranking[:self.cf.save_n_models])
self.logger.info(
"saving current epoch {} at rank {}".format(epoch, np.argwhere(epoch_ranking == epoch)))
# delete params of the epoch that just fell out of the top-k epochs.
for se in [int(ii.split('_')[0]) for ii in os.listdir(self.cf.fold_dir) if 'best_params' in ii]:
if se in epoch_ranking[self.cf.save_n_models:]:
subprocess.call('rm {}'.format(os.path.join(self.cf.fold_dir, '{}_best_params.pth'.format(se))),
shell=True)
self.logger.info('deleting epoch {} at rank {}'.format(se, np.argwhere(epoch_ranking == se)))
state = {
'epoch': epoch,
'state_dict': net.state_dict(),
'optimizer': optimizer.state_dict(),
}
if self.cf.server_env:
IO_safe(torch.save, state, os.path.join(self.cf.fold_dir, 'last_state.pth'))
else:
torch.save(state, os.path.join(self.cf.fold_dir, 'last_state.pth'))
def load_checkpoint(checkpoint_path, net, optimizer):
checkpoint = torch.load(checkpoint_path)
net.load_state_dict(checkpoint['state_dict'])
optimizer.load_state_dict(checkpoint['optimizer'])
return checkpoint['epoch']
def prepare_monitoring(cf):
"""
creates dictionaries, where train/val metrics are stored.
"""
metrics = {}
# first entry for loss dict accounts for epoch starting at 1.
metrics['train'] = OrderedDict() # [(l_name, [np.nan]) for l_name in cf.losses_to_monitor] )
metrics['val'] = OrderedDict() # [(l_name, [np.nan]) for l_name in cf.losses_to_monitor] )
metric_classes = []
if 'rois' in cf.report_score_level:
metric_classes.extend([v for k, v in cf.class_dict.items()])
if hasattr(cf, "eval_bins_separately") and cf.eval_bins_separately:
metric_classes.extend([v for k, v in cf.bin_dict.items()])
if 'patient' in cf.report_score_level:
metric_classes.extend(['patient_' + cf.class_dict[cf.patient_class_of_interest]])
if hasattr(cf, "eval_bins_separately") and cf.eval_bins_separately:
metric_classes.extend(['patient_' + cf.bin_dict[cf.patient_bin_of_interest]])
for cl in metric_classes:
for m in cf.metrics:
metrics['train'][cl + '_' + m] = [np.nan]
metrics['val'][cl + '_' + m] = [np.nan]
return metrics
class _AnsiColorizer(object):
"""
A colorizer is an object that loosely wraps around a stream, allowing
callers to write text to the stream in a particular color.
Colorizer classes must implement C{supported()} and C{write(text, color)}.
"""
_colors = dict(black=30, red=31, green=32, yellow=33,
blue=34, magenta=35, cyan=36, white=37, default=39)
def __init__(self, stream):
self.stream = stream
@classmethod
def supported(cls, stream=sys.stdout):
"""
A class method that returns True if the current platform supports
coloring terminal output using this method. Returns False otherwise.
"""
if not stream.isatty():
return False # auto color only on TTYs
try:
import curses
except ImportError:
return False
else:
try:
try:
return curses.tigetnum("colors") > 2
except curses.error:
curses.setupterm()
return curses.tigetnum("colors") > 2
except:
raise
# guess false in case of error
return False
def write(self, text, color):
"""
Write the given text to the stream in the given color.
@param text: Text to be written to the stream.
@param color: A string label for a color. e.g. 'red', 'white'.
"""
color = self._colors[color]
self.stream.write('\x1b[%sm%s\x1b[0m' % (color, text))
class ColorHandler(logging.StreamHandler):
def __init__(self, stream=sys.stdout):
super(ColorHandler, self).__init__(_AnsiColorizer(stream))
def emit(self, record):
msg_colors = {
logging.DEBUG: "green",
logging.INFO: "default",
logging.WARNING: "red",
logging.ERROR: "red"
}
color = msg_colors.get(record.levelno, "blue")
self.stream.write(record.msg + "\n", color)