diff --git a/DESCRIPTION b/DESCRIPTION
index c785775..a83af02 100644
--- a/DESCRIPTION
+++ b/DESCRIPTION
@@ -1,34 +1,34 @@
 Package: challengeR
 Type: Package
 Title: Analyzing assessment data of biomedical image analysis competitions and visualization of results
-Version: 1.0.0
-Date: 2020-12-20
+Version: 1.0.1
+Date: 2021-01-27
 Author: Manuel Wiesenfarth, Matthias Eisenmann, Laura Aguilera Saiz, Annette Kopp-Schneider
 Maintainer: Manuel Wiesenfarth <m.wiesenfarth@dkfz.de>
 Description: Analyzing assessment data of biomedical image analysis competitions and visualization of results.
 License: GPL (>= 2)
 Depends:
   R (>= 3.5.2),
   ggplot2 (>= 3.3.0),
   purrr (>= 0.3.3)
 Imports:
   dplyr (>= 0.8.5),
   graph (>= 1.64.0),
   knitr (>= 1.28),
   methods (>= 3.6.0),
   plyr (>= 1.8.6),
   relations (>= 0.6-9),
   reshape2 (>= 1.4.3),
   rlang (>= 0.4.5),
   rmarkdown (>= 2.1),
   tidyr (>= 1.0.2),
   viridisLite (>= 0.3.0)
 Suggests:
   doParallel (>= 1.0.15),
   foreach (>= 1.4.8),
   ggpubr (>= 0.2.5),
   Rgraphviz (>= 2.30.0),
   testthat (>= 2.1.0)
 VignetteBuilder: knitr
 Roxygen: list(markdown = TRUE)
 RoxygenNote: 7.1.0
diff --git a/R/firstlib.R b/R/firstlib.R
index 157a3e6..d2b1a80 100644
--- a/R/firstlib.R
+++ b/R/firstlib.R
@@ -1,35 +1,35 @@
 # Copyright (c) German Cancer Research Center (DKFZ)
 # All rights reserved.
 #
 # This file is part of challengeR.
 #
 # challengeR is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 2 of the License, or
 # (at your option) any later version.
 #
 # challengeR is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with challengeR. If not, see <https://www.gnu.org/licenses/>.
 
 .onAttach <- function (lib, pkg) {
   ver <- read.dcf(file.path(lib,pkg,"DESCRIPTION"),"Version")
   ver <- as.character(ver)
   packageStartupMessage("\nchallengeR ",
                         ver,
                         " loaded. \n",
                         #                       "Note: Layouting in case of many algorithms or tasks is not yet optimized. Please be patient, we are steadily working on improving the package",
-                        "\nPlease cite as:\n   Wiesenfarth, M., Reinke, A., Landman, B.A., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. (2019). Methods and open-source toolkit for analyzing and visualizing challenge results. arXiv preprint arXiv:1910.05121\n",
+                        "\nPlease cite as:\n   Wiesenfarth, M., Reinke, A., Landman, B.A., Eisenmann, M., Aguilera Saiz, L., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. Methods and open-source toolkit for analyzing and visualizing challenge results. Sci Rep 11, 2369 (2021). https://doi.org/10.1038/s41598-021-82017-6\n",
                         domain = NULL,
                         appendLF = TRUE)
 }
 
 .onLoad <- function(...) {
 }
 
 .onUnload <- function (libpath) {
 }
diff --git a/README.md b/README.md
index 40d7428..182a618 100644
--- a/README.md
+++ b/README.md
@@ -1,615 +1,630 @@
 Methods and open-source toolkit for analyzing and visualizing challenge
 results
 ================
 
 -   [Introduction](#introduction)
 -   [Installation](#installation)
 -   [Terms of use](#terms-of-use)
 -   [Usage](#usage)
 -   [Troubleshooting](#troubleshooting)
 -   [Changes](#changes)
 -   [Team](#team)
 -   [Reference](#reference)
 
 # Introduction
 
 The current framework is a tool for analyzing and visualizing challenge
 results in the field of biomedical image analysis and beyond.
 
 Biomedical challenges have become the de facto standard for benchmarking
 biomedical image analysis algorithms. While the number of challenges is
 steadily increasing, surprisingly little effort has been invested in
 ensuring high quality design, execution and reporting for these
 international competitions. Specifically, results analysis and
 visualization in the event of uncertainties have been given almost no
 attention in the literature.
 
 Given these shortcomings, the current framework aims to enable fast and
 wide adoption of comprehensively analyzing and visualizing the results
 of single-task and multi-task challenges. This approach offers an
 intuitive way to gain important insights into the relative and absolute
 performance of algorithms, which cannot be revealed by commonly applied
 visualization techniques.
 
 # Installation
 
 Requires R version &gt;= 3.5.2 (<https://www.r-project.org>).
 
 Further, a recent version of Pandoc (&gt;= 1.12.3) is required. RStudio
 (<https://rstudio.com>) automatically includes this so you do not need
 to download Pandoc if you plan to use rmarkdown from the RStudio IDE,
 otherwise you’ll need to install Pandoc for your platform
 (<https://pandoc.org/installing.html>). Finally, if you want to generate
 a PDF report you will need to have LaTeX installed (e.g. MiKTeX, MacTeX
 or TinyTeX).
 
 To get the latest released version (master branch) of the R package from
 GitHub:
 
 ``` r
 if (!requireNamespace("devtools", quietly = TRUE)) install.packages("devtools")
 if (!requireNamespace("BiocManager", quietly = TRUE)) install.packages("BiocManager")
 BiocManager::install("Rgraphviz", dependencies = TRUE)
 devtools::install_github("wiesenfa/challengeR", dependencies = TRUE)
 ```
 
 If you are asked whether you want to update installed packages and you
 type “a” for all, you might need administrator permissions to update R
 core packages. You can also try to type “n” for updating no packages. If
 you are asked “Do you want to install from sources the packages which
 need compilation? (Yes/no/cancel)”, you can safely type “no”.
 
 If you get *warning* messages (in contrast to *error* messages), these
 might not be problematic and you can try to proceed. If you encounter
 errors during the setup, looking into the “Troubleshooting” section
 might be worth it.
 
 For Linux users: Some system libraries might be missing. Check the
 output in the R console for further hints carefully during the
 installation of packages.
 
 # Terms of use
 
 Copyright (c) German Cancer Research Center (DKFZ). All rights reserved.
 
 challengeR is available under license GPLv2 or any later version.
 
 If you use this software for a publication, please cite:
 
-Wiesenfarth, M., Reinke, A., Landman, B.A., Cardoso, M.J., Maier-Hein,
-L. and Kopp-Schneider, A. (2019). Methods and open-source toolkit for
-analyzing and visualizing challenge results. *arXiv preprint
-arXiv:1910.05121*
+Wiesenfarth, M., Reinke, A., Landman, B.A., Eisenmann, M., Aguilera
+Saiz, L., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. Methods
+and open-source toolkit for analyzing and visualizing challenge results.
+*Sci Rep* **11**, 2369 (2021).
+<https://doi.org/10.1038/s41598-021-82017-6>
 
 # Usage
 
 Each of the following steps has to be run to generate the report: (1)
 Load package, (2) load data, (3) perform ranking, (4) perform
 bootstrapping and (5) generation of the report
 
 You can find R scripts for quickstart in the directory “vignettes”. An
 overview of all available plots is provided in the “Visualizations”
 vignette demonstrating the use of their corresponding plot functions as
 well.
 
 Here, we provide a step-by-step guide that leads you to your final
 report.
 
 ## 1. Load package
 
 Load package
 
 ``` r
 library(challengeR)
 ```
 
 ## 2. Load data
 
 ### Data requirements
 
 Data requires the following *columns*:
 
 -   *task identifier* in case of multi-task challenges (string or
     numeric)
 -   *test case identifier* (string or numeric)
 -   *algorithm identifier* (string or numeric)
 -   *metric value* (numeric)
 
 In case of missing metric values, a missing observation has to be
 provided (either as blank field or “NA”).
 
 For example, in a challenge with 2 tasks, 2 test cases and 2 algorithms,
 where in task “T2”, test case “case2”, algorithm “A2” didn’t give a
 prediction (and thus NA or a blank field for missing value is inserted),
 the data set might look like this:
 
 | Task | TestCase | Algorithm | MetricValue |
 |:-----|:---------|:----------|------------:|
 | T1   | case1    | A1        |       0.266 |
 | T1   | case1    | A2        |       0.202 |
 | T1   | case2    | A1        |       0.573 |
 | T1   | case2    | A2        |       0.945 |
 | T2   | case1    | A1        |       0.372 |
 | T2   | case1    | A2        |       0.898 |
 | T2   | case2    | A1        |       0.908 |
 | T2   | case2    | A2        |          NA |
 
 ### 2.1 Load data from file
 
 If you have assessment data at hand stored in a csv file (if you want to
 use simulated data, skip the following code line) use
 
 ``` r
 data_matrix <- read.csv(file.choose()) # type ?read.csv for help
 ```
 
 This allows to choose a file interactively, otherwise replace
 *file.choose()* by the file path (in style “/path/to/dataset.csv”) in
 quotation marks.
 
 ### 2.2 Simulate data
 
 In the following, simulated data is generated *instead* for illustration
 purposes (skip the following code chunk if you have already loaded
 data). The data is also stored as “inst/extdata/data\_matrix.csv” in the
 repository.
 
 ``` r
 if (!requireNamespace("permute", quietly = TRUE)) install.packages("permute")
 
 n <- 50
 
 set.seed(4)
 strip <- runif(n,.9,1)
 c_ideal <- cbind(task="c_ideal",
             rbind(
               data.frame(alg_name="A1",value=runif(n,.9,1),case=1:n),
               data.frame(alg_name="A2",value=runif(n,.8,.89),case=1:n),
               data.frame(alg_name="A3",value=runif(n,.7,.79),case=1:n),
               data.frame(alg_name="A4",value=runif(n,.6,.69),case=1:n),
               data.frame(alg_name="A5",value=runif(n,.5,.59),case=1:n)
             ))
 
 set.seed(1)
 c_random <- data.frame(task="c_random",
                        alg_name=factor(paste0("A",rep(1:5,each=n))),
                        value=plogis(rnorm(5*n,1.5,1)),case=rep(1:n,times=5)
                        )
 
 strip2 <- seq(.8,1,length.out=5)
 a <- permute::allPerms(1:5)
 c_worstcase <- data.frame(task="c_worstcase",
                      alg_name=c(t(a)),
                      value=rep(strip2,nrow(a)),
                      case=rep(1:nrow(a),each=5)
                      )
 c_worstcase <- rbind(c_worstcase,
                 data.frame(task="c_worstcase",alg_name=1:5,value=strip2,case=max(c_worstcase$case)+1)
           )
 c_worstcase$alg_name <- factor(c_worstcase$alg_name,labels=paste0("A",1:5))
 
 data_matrix <- rbind(c_ideal, c_random, c_worstcase)
 ```
 
 ## 3. Perform ranking
 
 ### 3.1 Define challenge object
 
 Code differs slightly for single- and multi-task challenges.
 
 In case of a single-task challenge use
 
 ``` r
 # Use only task "c_random" in object data_matrix
 dataSubset <- subset(data_matrix, task=="c_random")
 
 challenge <- as.challenge(dataSubset,
                           # Specify which column contains the algorithms, 
                           # which column contains a test case identifier 
                           # and which contains the metric value:
                           algorithm = "alg_name", case = "case", value = "value", 
                           # Specify if small metric values are better
                           smallBetter = FALSE)
 ```
 
 *Instead*, for a multi-task challenge use
 
 ``` r
 # Same as above but with 'by="task"' where variable "task" contains the task identifier
 challenge <- as.challenge(data_matrix, 
                           by = "task", 
                           algorithm = "alg_name", case = "case", value = "value", 
                           smallBetter = FALSE)
 ```
 
 ### 3.2 Configure ranking
 
 Different ranking methods are available, choose one of them:
 
 -   for “aggregate-then-rank” use (here: take mean for aggregation)
 
 ``` r
 ranking <- challenge%>%aggregateThenRank(FUN = mean, # aggregation function, 
                                                      # e.g. mean, median, min, max, 
                                                      # or e.g. function(x) quantile(x, probs=0.05)
                                          na.treat = 0, # either "na.rm" to remove missing data, 
                                                        # set missings to numeric value (e.g. 0) 
                                                        # or specify a function, 
                                                        # e.g. function(x) min(x)
                                          ties.method = "min" # a character string specifying 
                                                              # how ties are treated, see ?base::rank
                                         )  
 ```
 
 -   *alternatively*, for “rank-then-aggregate” with arguments as above
     (here: take mean for aggregation)
 
 ``` r
 ranking <- challenge%>%rankThenAggregate(FUN = mean,
                                          ties.method = "min"
                                         )
 ```
 
 -   *alternatively*, for test-then-rank based on Wilcoxon signed rank
     test
 
 ``` r
 ranking <- challenge%>%testThenRank(alpha = 0.05, # significance level
                                     p.adjust.method = "none", # method for adjustment for
                                                               # multiple testing, see ?p.adjust
                                     na.treat = 0, # either "na.rm" to remove missing data,
                                                   # set missings to numeric value (e.g. 0)
                                                   # or specify a function, e.g. function(x) min(x)
                                     ties.method = "min" # a character string specifying
                                                         # how ties are treated, see ?base::rank
                                    )
 ```
 
 ## 4. Perform bootstrapping
 
 Perform bootstrapping with 1000 bootstrap samples using one CPU
 
 ``` r
 set.seed(1)
 ranking_bootstrapped <- ranking%>%bootstrap(nboot = 1000)
 ```
 
 If you want to use multiple CPUs (here: 8 CPUs), use
 
 ``` r
 library(doParallel)
 registerDoParallel(cores = 8)  
 set.seed(1)
 ranking_bootstrapped <- ranking%>%bootstrap(nboot = 1000, parallel = TRUE, progress = "none")
 stopImplicitCluster()
 ```
 
 ## 5. Generate the report
 
 Generate report in PDF, HTML or DOCX format. Code differs slightly for
 single- and multi-task challenges.
 
 ### 5.1 For single-task challenges
 
 ``` r
 ranking_bootstrapped %>% 
   report(title = "singleTaskChallengeExample", # used for the title of the report
          file = "filename", 
          format = "PDF", # format can be "PDF", "HTML" or "Word"
          latex_engine = "pdflatex", #LaTeX engine for producing PDF output. Options are "pdflatex", "lualatex", and "xelatex"
          clean = TRUE #optional. Using TRUE will clean intermediate files that are created during rendering.
         ) 
 ```
 
 Argument *file* allows for specifying the output file path as well,
 otherwise the working directory is used. If file is specified but does
 not have a file extension, an extension will be automatically added
 according to the output format given in *format*. Using argument
 *clean=FALSE* allows to retain intermediate files, such as separate
 files for each figure.
 
 If argument “file” is omitted, the report is created in a temporary
 folder with file name “report”.
 
 ### 5.2 For multi-task challenges
 
 Same as for single-task challenges, but additionally consensus ranking
 (rank aggregation across tasks) has to be given.
 
 Compute ranking consensus across tasks (here: consensus ranking
 according to mean ranks across tasks)
 
 ``` r
 # See ?relation_consensus for different methods to derive consensus ranking
 meanRanks <- ranking%>%consensus(method = "euclidean") 
 meanRanks # note that there may be ties (i.e. some algorithms have identical mean rank)
 ```
 
 Generate report as above, but with additional specification of consensus
 ranking
 
 ``` r
 ranking_bootstrapped %>% 
   report(consensus = meanRanks,
          title = "multiTaskChallengeExample",
          file = "filename", 
          format = "PDF", # format can be "PDF", "HTML" or "Word"
          latex_engine = "pdflatex"#LaTeX engine for producing PDF output. Options are "pdflatex", "lualatex", and "xelatex"
         )
 ```
 
 The consensus ranking is given according to mean ranks across tasks if
 method=“euclidean” where in case of ties (equal ranks for multiple
 algorithms) the average rank is used, i.e. ties.method=“average”.
 
 # Troubleshooting
 
 In this section we provide an overview of issues that the users reported
 and how they were solved.
 
 ## Issues related to RStudio
 
 ### Issue: Rtools is missing
 
 While trying to install the current version of the repository:
 
 ``` r
 devtools::install_github("wiesenfa/challengeR", dependencies = TRUE)
 ```
 
 The following warning showed up in the output:
 
 ``` r
 WARNING: Rtools is required to build R packages, but is not currently installed.
 ```
 
 Therefore, Rtools was installed via a separate executable:
 <https://cran.r-project.org/bin/windows/Rtools/> and the warning
 disappeared.
 
 #### Solution:
 
 Actually there is no need of installing Rtools, it is not really used in
 the toolkit. Insted, choose not to install it when it is asked. See
 comment in the installation section:
 
 “If you are asked whether you want to update installed packages and you
 type “a” for all, you might need administrator rights to update R core
 packages. You can also try to type “n” for updating no packages. If you
 are asked “Do you want to install from sources the packages which need
 compilation? (Yes/no/cancel)”, you can safely type “no”.”
 
 ### Issue: Package versions are mismatching
 
 Installing the current version of the tool from GitHub failed.
 
 The error message was:
 
 ``` r
 byte-compile and prepare package for lazy loading
 Error: (converted from warning) package 'ggplot2' was built under R version 3.6.3
 Execution halted
 ERROR: lazy loading failed for package 'challengeR'
 * removing 'C:/Users/.../Documents/R/win-library/3.6/challengeR'
 * restoring previous 'C:/Users/.../Documents/R/win-library/3.6/challengeR'
 Error: Failed to install 'challengeR' from GitHub:
   (converted from warning) installation of package 'C:/Users/.../AppData/Local/Temp/Rtmp615qmV/file4fd419555eb4/challengeR_0.3.1.tar.gz' had non-zero exit status
 ```
 
 The problem was that some of the packages that were built under R3.6.1
 had been updated, but the current installed version was still R3.6.1.
 
 #### Solution:
 
 The solution was to update R3.6.1 to R3.6.3. Another way would have been
 to reset the single packages to the versions built under R3.6.1.
 
 ### Issue: Package is missing
 
 Installing the current version of the tool from GitHub failed.
 
 ``` r
  devtools::install_github("wiesenfa/challengeR", dependencies = TRUE)
 ```
 
 The error message was:
 
 ``` r
 Error: .onLoad failed in loadNamespace() for 'pkgload', details:
   call: loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]])
   error: there is no package called ‘backports’
 ```
 
 The problem was that the packages ‘backports’ had not been installed.
 
 #### Solution:
 
 The solution was to install ‘backports’ manually.
 
 ``` r
  install.packages("backports")
 ```
 
 ### Issue: Packages are not detected correctly
 
 While trying to install the package after running the following
 commands:
 
 ``` r
 if (!requireNamespace("devtools", quietly = TRUE)) install.packages("devtools")
 if (!requireNamespace("BiocManager", quietly = TRUE)) install.packages("BiocManager")
 BiocManager::install("Rgraphviz", dependencies = TRUE)
 devtools::install_github("wiesenfa/challengeR", dependencies = TRUE)
 ```
 
 The error message was:
 
 ``` r
 ERROR:
 1: In file(con, "r") :
  URL 'https://bioconductor.org/config.yaml': status was 'SSL connect error'
 2: packages ‘BiocVersion’, ‘Rgraphviz’ are not available (for R version 3.6.1)
 ```
 
 #### Solution:
 
 The solution was to restart RStudio.
 
 ## Issues related to MiKTeX
 
 ### Issue: Missing packages
 
 While generating the PDF with MiKTeX (2.9), the following error showed
 up:
 
 ``` r
 fatal pdflatex - gui framework cannot be initialized
 ```
 
 There is an issue with installing missing packages in LaTeX.
 
 ##### Solution:
 
 Open your MiKTeX Console –&gt; Settings, select “Always install missing
 packages on-the-fly”. Then generate the report. Once the report is
 generated, you can reset the settings to your preferred ones.
 
 ### Issue: Unable to generate report
 
 While generating the PDF with MiKTeX (2.9):
 
 ``` r
 ranking_bootstrapped %>% 
   report(title = "singleTaskChallengeExample", # used for the title of the report
          file = "filename", 
          format = "PDF", # format can be "PDF", "HTML" or "Word"
          latex_engine = "pdflatex", #LaTeX engine for producing PDF output. Options are "pdflatex", "lualatex", and "xelatex"
          clean = TRUE #optional. Using TRUE will clean intermediate files that are created during rendering.
         ) 
 ```
 
 The following error showed up:
 
 ``` r
 output file: filename.knit.md
 
 "C:/Program Files/RStudio/bin/pandoc/pandoc" +RTS -K512m -RTS filename.utf8.md --to latex --from markdown+autolink_bare_uris+tex_math_single_backslash --output filename.tex --self-contained --number-sections --highlight-style tango --pdf-engine pdflatex --variable graphics --lua-filter "C:/Users/adm/Documents/R/win-library/3.6/rmarkdown/rmd/lua/pagebreak.lua" --lua-filter "C:/Users/adm/Documents/R/win-library/3.6/rmarkdown/rmd/lua/latex-div.lua" --variable "geometry:margin=1in" 
 
 Error: LaTeX failed to compile filename.tex. See https://yihui.org/tinytex/r/#debugging for debugging tips.
 
   Warning message:
 In system2(..., stdout = if (use_file_stdout()) f1 else FALSE, stderr = f2) :
   '"pdflatex"' not found
 ```
 
 #### Solution:
 
 The solution was to restart RStudio.
 
 # Changes
 
+#### Version 1.0.1
+
+-   Fix error raised in case there are more tasks than algorithms
+    contained in the dataset
+    ([T28193](https://phabricator.mitk.org/T28193))
+-   Drop restriction that at least three algorithms are required for
+    bootstrapping ([T28194](https://phabricator.mitk.org/T28194))
+-   Avoid blank pages in PDF report when bootstrapping is disabled
+    ([T28201](https://phabricator.mitk.org/T28201))
+-   Handle tasks having only one case for bootstrapping
+    ([T28202](https://phabricator.mitk.org/T28202))
+-   Update citation ([T28210](https://phabricator.mitk.org/T28210))
+
 #### Version 1.0.0
 
 -   Revision of the underlying data structure
 -   Roxygen documentation for main functionality
 -   Vignettes for quickstart and overview of available plots
     demonstrating the use of their corresponding plot functions
 -   Introduction of unit tests (package coverage &gt;70%)
 -   Troubleshooting section covering potential issues during setup
 -   Finally: Extensive bug fixes and improvements (for a complete
     overview please check the [Phabricator
     tasks](https://phabricator.mitk.org/search/query/vtj0qOqH5qL6/))
 
 #### Version 0.3.3
 
 -   Force line break to avoid that authors exceed the page in generated
     PDF reports
 
 #### Version 0.3.2
 
 -   Correct names of authors
 
 #### Version 0.3.1
 
 -   Refactoring
 
 #### Version 0.3.0
 
 -   Major bug fix release
 
 #### Version 0.2.5
 
 -   Bug fixes
 
 #### Version 0.2.4
 
 -   Automatic insertion of missings
 
 #### Version 0.2.3
 
 -   Bug fixes
 -   Reports for subsets (top list) of algorithms: Use
     e.g. `subset(ranking_bootstrapped, top=3) %>% report(...)` (or
     `subset(ranking, top=3) %>% report(...)` for report without
     bootstrap results) to only show the top 3 algorithms according to
     the chosen ranking methods, where `ranking_bootstrapped` and
     `ranking` objects as defined in the example. Line plot for ranking
     robustness can be used to check whether algorithms performing well
     in other ranking methods are excluded. Bootstrapping still takes
     entire uncertainty into account. Podium plot and ranking heatmap
     neglect excluded algorithms. Only available for single-task
     challenges (for multi-task challenges not sensible because each task
     would contain a different set of algorithms).
 -   Reports for subsets of tasks: Use
     e.g. `subset(ranking_bootstrapped, tasks=c("task1", "task2","task3")) %>% report(...)`
     to restrict report to tasks “task1”, “task2”,"task3. You may want to
     recompute the consensus ranking before using
     `meanRanks=subset(ranking, tasks=c("task1", "task2", "task3"))%>%consensus(method = "euclidean")`
 
 #### Version 0.2.1
 
 -   Introduction in reports now mentions e.g. ranking method, number of
     test cases,…
 -   Function `subset()` allows selection of tasks after bootstrapping,
     e.g. `subset(ranking_bootstrapped,1:3)`
 -   `report()` functions gain argument `colors` (default:
     `default_colors`). Change e.g. to `colors=viridisLite::inferno`
     which “is designed in such a way that it will analytically be
     perfectly perceptually-uniform, both in regular form and also when
     converted to black-and-white. It is also designed to be perceived by
     readers with the most common form of color blindness.” See package
     `viridis` for further similar functions.
 
 #### Version 0.2.0
 
 -   Improved layout in case of many algorithms and tasks (while probably
     still not perfect)
 -   Consistent coloring of algorithms across figures
 -   `report()` function can be applied to ranked object before
     bootstrapping (and thus excluding figures based on bootstrapping),
     i.e. in the example `ranking %>% report(...)`
 -   bug fixes
 
 # Team
 
 The developer team includes members from both division of Computer
 Assisted Medical Interventions (CAMI) and Biostatistics at the German
 Cancer Research Center (DKFZ):
 
 -   Manuel Wiesenfarth
 -   Annette Kopp-Schneider
 -   Annika Reinke
 -   Matthias Eisenmann
 -   Laura Aguilera Saiz
 -   Elise Récéjac
 -   Lena Maier-Hein
 
 # Reference
 
-Wiesenfarth, M., Reinke, A., Landman, B.A., Cardoso, M.J., Maier-Hein,
-L. and Kopp-Schneider, A. (2019). Methods and open-source toolkit for
-analyzing and visualizing challenge results. *arXiv preprint
-arXiv:1910.05121*
+Wiesenfarth, M., Reinke, A., Landman, B.A., Eisenmann, M., Aguilera
+Saiz, L., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. Methods
+and open-source toolkit for analyzing and visualizing challenge results.
+*Sci Rep* **11**, 2369 (2021).
+<https://doi.org/10.1038/s41598-021-82017-6>
 
 <img src="DKFZ_Logo.png" width="300px" />
 <img src="HIP_Logo.png" width="340px" />
diff --git a/Readme.Rmd b/Readme.Rmd
index 7679ae5..eed2927 100644
--- a/Readme.Rmd
+++ b/Readme.Rmd
@@ -1,514 +1,521 @@
 ---
 title: "Methods and open-source toolkit for analyzing and visualizing challenge results"
 output:
   github_document:
     toc: yes
     toc_depth: '1'
   html_document:
     toc: yes
     toc_depth: '1'
   pdf_document:
     toc: yes
     toc_depth: '1'
 editor_options:
   chunk_output_type: console
 ---
 
 ```{r, echo = FALSE}
 knitr::opts_chunk$set(
  collapse = TRUE,
   comment = "#>",
  # fig.path = "README-",
     fig.width = 9,
     fig.height = 5,
     width=160
 )
 ```
 
 # Introduction
 
 The current framework is a tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.
 
 Biomedical challenges have become the de facto standard for benchmarking biomedical image analysis algorithms. While the number of challenges is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature.
 
 Given these shortcomings, the current framework aims to enable fast and wide adoption of comprehensively analyzing and visualizing the results of single-task and multi-task challenges. This approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques.
 
 # Installation
 
 Requires R version >= 3.5.2 (https://www.r-project.org).
 
 Further, a recent version of Pandoc (>= 1.12.3) is required. RStudio (https://rstudio.com) automatically includes this so you do not need to download Pandoc if you plan to use rmarkdown from the RStudio IDE, otherwise you’ll need to install Pandoc for your platform (https://pandoc.org/installing.html). Finally, if you want to generate a PDF report you will need to have LaTeX installed (e.g. MiKTeX, MacTeX or TinyTeX).
 
 
 To get the latest released version (master branch) of the R package from GitHub:
 
 ```{r, eval=F,R.options,}
 if (!requireNamespace("devtools", quietly = TRUE)) install.packages("devtools")
 if (!requireNamespace("BiocManager", quietly = TRUE)) install.packages("BiocManager")
 BiocManager::install("Rgraphviz", dependencies = TRUE)
 devtools::install_github("wiesenfa/challengeR", dependencies = TRUE)
 ```
 
 If you are asked whether you want to update installed packages and you type "a" for all, you might need administrator permissions to update R core packages. You can also try to type "n" for updating no packages. If you are asked "Do you want to install from sources the packages which need compilation? (Yes/no/cancel)", you can safely type "no".
 
 If you get *warning* messages (in contrast to *error* messages), these might not be problematic and you can try to proceed. If you encounter errors during the setup, looking into the "Troubleshooting" section might be worth it.
 
 For Linux users: Some system libraries might be missing. Check the output in the R console for further hints carefully during the installation of packages.
 
 # Terms of use
 Copyright (c) German Cancer Research Center (DKFZ). All rights reserved.
 
 challengeR is available under license GPLv2 or any later version.
 
 If you use this software for a publication, please cite:
 
-Wiesenfarth, M., Reinke, A., Landman, B.A., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. (2019). Methods and open-source toolkit for analyzing and visualizing challenge results. *arXiv preprint arXiv:1910.05121*
+Wiesenfarth, M., Reinke, A., Landman, B.A., Eisenmann, M., Aguilera Saiz, L., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. Methods and open-source toolkit for analyzing and visualizing challenge results. *Sci Rep* **11**, 2369 (2021). https://doi.org/10.1038/s41598-021-82017-6
 
 
  
 # Usage
 Each of the following steps has to be run to generate the report: (1) Load package, (2) load data, (3) perform ranking, (4) perform bootstrapping and (5) generation of the report
 
 You can find R scripts for quickstart in the directory "vignettes". An overview of all available plots is provided in the "Visualizations" vignette demonstrating the use of their corresponding plot functions as well.
 
 Here, we provide a step-by-step guide that leads you to your final report.
 
 ## 1. Load package
 Load package
 ```{r, eval=F}
 library(challengeR)
 ```
 
 
 ## 2. Load data
 
 ### Data requirements
 Data requires the following *columns*:
 
 * *task identifier* in case of multi-task challenges (string or numeric)
 * *test case identifier* (string or numeric)
 * *algorithm identifier* (string or numeric)
 * *metric value* (numeric)
 
 In case of missing metric values, a missing observation has to be provided (either as blank field or "NA").
 
 
 For example, in a challenge with 2 tasks, 2 test cases and 2 algorithms, where in task "T2", test case "case2", algorithm "A2" didn't give a prediction (and thus NA or a blank field for missing value is inserted), the data set might look like this: 
 
 ```{r, eval=T, echo=F,results='asis'}
 set.seed(1)
 a <- cbind(expand.grid(Task=paste0("T",1:2),TestCase=paste0("case",1:2),Algorithm=paste0("A",1:2)),MetricValue=round(c(runif(7,0,1),NA),3))
 print(knitr::kable(a[order(a$Task,a$TestCase,a$Algorithm),],row.names=F))
 ```
 
 
 
 ### 2.1 Load data from file
 If you have assessment data at hand stored in a csv file (if you want to use simulated data, skip the following code line) use
 ```{r, eval=F, echo=T}
 data_matrix <- read.csv(file.choose()) # type ?read.csv for help
 
 ```
 
 This allows to choose a file interactively, otherwise replace *file.choose()* by the file path (in style "/path/to/dataset.csv") in quotation marks.
 
 
 
 ### 2.2 Simulate data
 
 In the following, simulated data is generated *instead* for illustration purposes (skip the following code chunk if you have already loaded data). The data is also stored as "inst/extdata/data_matrix.csv" in the repository.
 ```{r, eval=F, echo=T}
 if (!requireNamespace("permute", quietly = TRUE)) install.packages("permute")
 
 n <- 50
 
 set.seed(4)
 strip <- runif(n,.9,1)
 c_ideal <- cbind(task="c_ideal",
             rbind(
               data.frame(alg_name="A1",value=runif(n,.9,1),case=1:n),
               data.frame(alg_name="A2",value=runif(n,.8,.89),case=1:n),
               data.frame(alg_name="A3",value=runif(n,.7,.79),case=1:n),
               data.frame(alg_name="A4",value=runif(n,.6,.69),case=1:n),
               data.frame(alg_name="A5",value=runif(n,.5,.59),case=1:n)
             ))
 
 set.seed(1)
 c_random <- data.frame(task="c_random",
                        alg_name=factor(paste0("A",rep(1:5,each=n))),
                        value=plogis(rnorm(5*n,1.5,1)),case=rep(1:n,times=5)
                        )
 
 strip2 <- seq(.8,1,length.out=5)
 a <- permute::allPerms(1:5)
 c_worstcase <- data.frame(task="c_worstcase",
                      alg_name=c(t(a)),
                      value=rep(strip2,nrow(a)),
                      case=rep(1:nrow(a),each=5)
                      )
 c_worstcase <- rbind(c_worstcase,
                 data.frame(task="c_worstcase",alg_name=1:5,value=strip2,case=max(c_worstcase$case)+1)
           )
 c_worstcase$alg_name <- factor(c_worstcase$alg_name,labels=paste0("A",1:5))
 
 data_matrix <- rbind(c_ideal, c_random, c_worstcase)
 
 ```
 
 
 ## 3. Perform ranking
 
 ### 3.1 Define challenge object
 Code differs slightly for single- and multi-task challenges.
 
 In case of a single-task challenge use
 
 ```{r, eval=F, echo=T}
 # Use only task "c_random" in object data_matrix
 dataSubset <- subset(data_matrix, task=="c_random")
 
 challenge <- as.challenge(dataSubset,
                           # Specify which column contains the algorithms, 
                           # which column contains a test case identifier 
                           # and which contains the metric value:
                           algorithm = "alg_name", case = "case", value = "value", 
                           # Specify if small metric values are better
                           smallBetter = FALSE)
 ```
 
 *Instead*, for a multi-task challenge use 
 
 ```{r, eval=F, echo=T}
 # Same as above but with 'by="task"' where variable "task" contains the task identifier
 challenge <- as.challenge(data_matrix, 
                           by = "task", 
                           algorithm = "alg_name", case = "case", value = "value", 
                           smallBetter = FALSE)
 ```
 
 
 ### 3.2 Configure ranking
 
 Different ranking methods are available, choose one of them:
 
 - for "aggregate-then-rank" use (here: take mean for aggregation)
 ```{r, eval=F, echo=T}
 ranking <- challenge%>%aggregateThenRank(FUN = mean, # aggregation function, 
                                                      # e.g. mean, median, min, max, 
                                                      # or e.g. function(x) quantile(x, probs=0.05)
                                          na.treat = 0, # either "na.rm" to remove missing data, 
                                                        # set missings to numeric value (e.g. 0) 
                                                        # or specify a function, 
                                                        # e.g. function(x) min(x)
                                          ties.method = "min" # a character string specifying 
                                                              # how ties are treated, see ?base::rank
                                         )  
 ```
 
 - *alternatively*, for "rank-then-aggregate" with arguments as above (here: take mean for aggregation)
 ```{r, eval=F, echo=T}
 ranking <- challenge%>%rankThenAggregate(FUN = mean,
                                          ties.method = "min"
                                         )
 ```
 
 - *alternatively*, for test-then-rank based on Wilcoxon signed rank test
 ```{r, eval=F, echo=T}
 ranking <- challenge%>%testThenRank(alpha = 0.05, # significance level
                                     p.adjust.method = "none", # method for adjustment for
                                                               # multiple testing, see ?p.adjust
                                     na.treat = 0, # either "na.rm" to remove missing data,
                                                   # set missings to numeric value (e.g. 0)
                                                   # or specify a function, e.g. function(x) min(x)
                                     ties.method = "min" # a character string specifying
                                                         # how ties are treated, see ?base::rank
                                    )
 
 ```
 
 ## 4. Perform bootstrapping
 
 Perform bootstrapping with 1000 bootstrap samples using one CPU
 ```{r, eval=F, echo=T}
 set.seed(1)
 ranking_bootstrapped <- ranking%>%bootstrap(nboot = 1000)
 ```
 
 If you want to use multiple CPUs (here: 8 CPUs), use
 
 ```{r, eval=F, echo=T}
 library(doParallel)
 registerDoParallel(cores = 8)  
 set.seed(1)
 ranking_bootstrapped <- ranking%>%bootstrap(nboot = 1000, parallel = TRUE, progress = "none")
 stopImplicitCluster()
 ```
 
 
 
 ## 5. Generate the report
 Generate report in PDF, HTML or DOCX format. Code differs slightly for single- and multi-task challenges.
 
 ### 5.1 For single-task challenges
 ```{r, eval=F, echo=T}
 ranking_bootstrapped %>% 
   report(title = "singleTaskChallengeExample", # used for the title of the report
          file = "filename", 
          format = "PDF", # format can be "PDF", "HTML" or "Word"
          latex_engine = "pdflatex", #LaTeX engine for producing PDF output. Options are "pdflatex", "lualatex", and "xelatex"
          clean = TRUE #optional. Using TRUE will clean intermediate files that are created during rendering.
         ) 
 
 ```
 
 Argument *file* allows for specifying the output file path as well, otherwise the working directory is used.
 If file is specified but does not have a file extension, an extension will be automatically added according to the output format given in *format*. 
 Using argument *clean=FALSE* allows to retain intermediate files, such as separate files for each figure.
 
 If argument "file" is omitted, the report is  created in a temporary folder with file name "report".
 
 
 
 
 ### 5.2 For multi-task challenges
 Same as for single-task challenges, but additionally consensus ranking (rank aggregation across tasks) has to be given.
 
 Compute ranking consensus across tasks (here: consensus ranking according to mean ranks across tasks)
  
 ```{r, eval=F, echo=T}
 # See ?relation_consensus for different methods to derive consensus ranking
 meanRanks <- ranking%>%consensus(method = "euclidean") 
 meanRanks # note that there may be ties (i.e. some algorithms have identical mean rank)
 ```
 
 Generate report as above, but with additional specification of consensus ranking
 ```{r, eval=F, echo=T}
 ranking_bootstrapped %>% 
   report(consensus = meanRanks,
          title = "multiTaskChallengeExample",
          file = "filename", 
          format = "PDF", # format can be "PDF", "HTML" or "Word"
          latex_engine = "pdflatex"#LaTeX engine for producing PDF output. Options are "pdflatex", "lualatex", and "xelatex"
         )
 ```
 The consensus ranking is given according to mean ranks across tasks if method="euclidean" where in case of ties (equal ranks for multiple algorithms) the average rank is used, i.e. ties.method="average".
 
 
 # Troubleshooting
 
 In this section we provide an overview of issues that the users reported and how they were solved.
 
 ## Issues related to RStudio
 
 ### Issue: Rtools is missing
 
 While trying to install the current version of the repository:
 
 ```{r, eval=F, echo=T}
 devtools::install_github("wiesenfa/challengeR", dependencies = TRUE)
 ```
 
 The following warning showed up in the output:
 ```{r, eval=F, echo=T}
 WARNING: Rtools is required to build R packages, but is not currently installed.
 ```
 
 Therefore, Rtools was installed via a separate executable: https://cran.r-project.org/bin/windows/Rtools/ and the warning disappeared.
 	
 #### Solution: 
 
 Actually there is no need of installing Rtools, it is not really used in the toolkit. Insted, choose not to install it when it is asked. See comment in the installation section: 
 
 “If you are asked whether you want to update installed packages and you type “a” for all, you might need administrator rights to update R core packages. You can also try to type “n” for updating no packages. If you are asked “Do you want to install from sources the packages which need compilation? (Yes/no/cancel)”, you can safely type “no”.”
 
 ### Issue: Package versions are mismatching
 
 Installing the current version of the tool from GitHub failed.
 
 The error message was:
 
 ```{r, eval=F, echo=T}
 byte-compile and prepare package for lazy loading
 Error: (converted from warning) package 'ggplot2' was built under R version 3.6.3
 Execution halted
 ERROR: lazy loading failed for package 'challengeR'
 * removing 'C:/Users/.../Documents/R/win-library/3.6/challengeR'
 * restoring previous 'C:/Users/.../Documents/R/win-library/3.6/challengeR'
 Error: Failed to install 'challengeR' from GitHub:
   (converted from warning) installation of package 'C:/Users/.../AppData/Local/Temp/Rtmp615qmV/file4fd419555eb4/challengeR_0.3.1.tar.gz' had non-zero exit status
 ```
 
 The problem was that some of the packages that were built under R3.6.1 had been updated, but the current installed version was still R3.6.1. 
 
 #### Solution:
 
 The solution was to update R3.6.1 to R3.6.3. Another way would have been to reset the single packages to the versions built under R3.6.1.
 
 
 ### Issue: Package is missing
 
 Installing the current version of the tool from GitHub failed.
 
 ```{r, eval=F, echo=T}
  devtools::install_github("wiesenfa/challengeR", dependencies = TRUE)
 ```
 
 The error message was:
 
 ```{r, eval=F, echo=T}
 Error: .onLoad failed in loadNamespace() for 'pkgload', details:
   call: loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]])
   error: there is no package called ‘backports’
 ```
 
 The problem was that the packages 'backports' had not been installed. 
 
 #### Solution:
 
 The solution was to install 'backports' manually.
 
 ```{r, eval=F, echo=T}
  install.packages("backports")
 ```
 
 
 ### Issue: Packages are not detected correctly
 
 While trying to install the package after running the following commands:
 
 ```{r, eval=F, echo=T}
 if (!requireNamespace("devtools", quietly = TRUE)) install.packages("devtools")
 if (!requireNamespace("BiocManager", quietly = TRUE)) install.packages("BiocManager")
 BiocManager::install("Rgraphviz", dependencies = TRUE)
 devtools::install_github("wiesenfa/challengeR", dependencies = TRUE)
 ```
 
 The error message was:
 
 ```{r, eval=F, echo=T}
 ERROR:
 1: In file(con, "r") :
  URL 'https://bioconductor.org/config.yaml': status was 'SSL connect error'
 2: packages ‘BiocVersion’, ‘Rgraphviz’ are not available (for R version 3.6.1)
 ```
 
 #### Solution:
 
 The solution was to restart RStudio.
 
 
 ## Issues related to MiKTeX
 
 ### Issue: Missing packages
 
 While generating the PDF with MiKTeX (2.9), the following error showed up:
 
 ```{r, eval=F, echo=T}
 fatal pdflatex - gui framework cannot be initialized
 ```
 
 There is an issue with installing missing packages in LaTeX.
 
 ##### Solution:
 
 Open your MiKTeX Console --> Settings, select "Always install missing packages on-the-fly". Then generate the report. Once the report is generated, you can reset the settings to your preferred ones.
 
 
 ### Issue: Unable to generate report
 
 While generating the PDF with MiKTeX (2.9):
 
 ```{r, eval=F, echo=T}
 ranking_bootstrapped %>% 
   report(title = "singleTaskChallengeExample", # used for the title of the report
          file = "filename", 
          format = "PDF", # format can be "PDF", "HTML" or "Word"
          latex_engine = "pdflatex", #LaTeX engine for producing PDF output. Options are "pdflatex", "lualatex", and "xelatex"
          clean = TRUE #optional. Using TRUE will clean intermediate files that are created during rendering.
         ) 
 
 ```
 
 The following error showed up: 
 
 ```{r, eval=F, echo=T}
 output file: filename.knit.md
 
 "C:/Program Files/RStudio/bin/pandoc/pandoc" +RTS -K512m -RTS filename.utf8.md --to latex --from markdown+autolink_bare_uris+tex_math_single_backslash --output filename.tex --self-contained --number-sections --highlight-style tango --pdf-engine pdflatex --variable graphics --lua-filter "C:/Users/adm/Documents/R/win-library/3.6/rmarkdown/rmd/lua/pagebreak.lua" --lua-filter "C:/Users/adm/Documents/R/win-library/3.6/rmarkdown/rmd/lua/latex-div.lua" --variable "geometry:margin=1in" 
 
 Error: LaTeX failed to compile filename.tex. See https://yihui.org/tinytex/r/#debugging for debugging tips.
 
   Warning message:
 In system2(..., stdout = if (use_file_stdout()) f1 else FALSE, stderr = f2) :
   '"pdflatex"' not found
 ```
 
 #### Solution:
 
 The solution was to restart RStudio.
 
 
 # Changes
 
+#### Version 1.0.1
+- Fix error raised in case there are more tasks than algorithms contained in the dataset ([T28193](https://phabricator.mitk.org/T28193))
+- Drop restriction that at least three algorithms are required for bootstrapping ([T28194](https://phabricator.mitk.org/T28194))
+- Avoid blank pages in PDF report when bootstrapping is disabled ([T28201](https://phabricator.mitk.org/T28201))
+- Handle tasks having only one case for bootstrapping ([T28202](https://phabricator.mitk.org/T28202))
+- Update citation ([T28210](https://phabricator.mitk.org/T28210))
+
 #### Version 1.0.0
 - Revision of the underlying data structure
 - Roxygen documentation for main functionality
 - Vignettes for quickstart and overview of available plots demonstrating the use of their corresponding plot functions
 - Introduction of unit tests (package coverage >70%)
 - Troubleshooting section covering potential issues during setup
 - Finally: Extensive bug fixes and improvements (for a complete overview please check the [Phabricator tasks](https://phabricator.mitk.org/search/query/vtj0qOqH5qL6/))
 
 
 #### Version 0.3.3
 - Force line break to avoid that authors exceed the page in generated PDF reports
 
 #### Version 0.3.2
 - Correct names of authors
 
 #### Version 0.3.1
 - Refactoring
 
 #### Version 0.3.0
 - Major bug fix release
 
 #### Version 0.2.5
 - Bug fixes
 
 
 #### Version 0.2.4
 - Automatic insertion of missings
 
 #### Version 0.2.3
 - Bug fixes
 - Reports for subsets (top list) of algorithms: Use e.g. `subset(ranking_bootstrapped, top=3) %>% report(...)` (or `subset(ranking, top=3) %>% report(...)` for report without bootstrap results) to only show the top 3 algorithms according to the chosen ranking methods, where  `ranking_bootstrapped` and `ranking` objects as defined in the example. Line plot for ranking robustness can be used to check whether algorithms performing well in other ranking methods are excluded. Bootstrapping still takes entire uncertainty into account. Podium plot and ranking heatmap neglect excluded algorithms. Only available for single-task challenges (for multi-task challenges not sensible because each task would contain a different set of algorithms). 
 - Reports for subsets of tasks: Use e.g. `subset(ranking_bootstrapped, tasks=c("task1", "task2","task3")) %>% report(...)` to restrict report to tasks "task1", "task2","task3. You may want to recompute the consensus ranking before using `meanRanks=subset(ranking, tasks=c("task1", "task2", "task3"))%>%consensus(method = "euclidean")`
 
 #### Version 0.2.1
 - Introduction in reports now mentions e.g. ranking method, number of test cases,...
 - Function `subset()` allows selection of tasks after bootstrapping, e.g. `subset(ranking_bootstrapped,1:3)`
 - `report()` functions gain argument `colors` (default: `default_colors`). Change e.g. to `colors=viridisLite::inferno` which "is designed in such a way that it will analytically be perfectly perceptually-uniform, both in regular form and also when converted to black-and-white. It is also designed to be perceived by readers with the most common form of color blindness." See package `viridis` for further similar functions.
 
 #### Version 0.2.0
 - Improved layout in case of many algorithms and tasks (while probably still not perfect)
 - Consistent coloring of algorithms across figures
 - `report()` function can be applied to ranked object before bootstrapping (and thus excluding figures based on bootstrapping), i.e. in the example `ranking %>% report(...)`
 - bug fixes
   
 # Team
 
 The developer team includes members from both division of Computer Assisted Medical Interventions (CAMI) and Biostatistics at the German Cancer Research Center (DKFZ): 
 
 - Manuel Wiesenfarth 
 - Annette Kopp-Schneider
 - Annika Reinke
 - Matthias Eisenmann
 - Laura Aguilera Saiz
 - Elise Récéjac
 - Lena Maier-Hein
   
 # Reference
 
-Wiesenfarth, M., Reinke, A., Landman, B.A., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. (2019). Methods and open-source toolkit for analyzing and visualizing challenge results. *arXiv preprint arXiv:1910.05121*
+Wiesenfarth, M., Reinke, A., Landman, B.A., Eisenmann, M., Aguilera Saiz, L., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. Methods and open-source toolkit for analyzing and visualizing challenge results. *Sci Rep* **11**, 2369 (2021). https://doi.org/10.1038/s41598-021-82017-6
 
 <img src="DKFZ_Logo.png" width="300px" />
 <img src="HIP_Logo.png" width="340px" />
diff --git a/inst/CITATION b/inst/CITATION
index 0bd03ba..3593e03 100644
--- a/inst/CITATION
+++ b/inst/CITATION
@@ -1,8 +1,8 @@
 bibentry(
   bibtype="Article",
   title={"Methods and open-source toolkit for analyzing and visualizing challenge results"},
-  journal={"arXiv preprint arXiv:1910.05121"},
-  year="2019",
-  author=personList(as.person("Manuel Wiesenfarth"), as.person("Annika Reinke"), as.person("Bennett A. Landman"), as.person("Manuel Jorge Cardoso"), as.person("Lena Maier-Hein"), as.person("Annette Kopp-Schneider")),
-  textVersion="Wiesenfarth, M., Reinke, A., Landman, B.A., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. (2019). Methods and open-source toolkit for analyzing and visualizing challenge results. arXiv preprint arXiv:1910.05121"
+  journal={"Scientific Reports"},
+  year="2021",
+  author=personList(as.person("Manuel Wiesenfarth"), as.person("Annika Reinke"), as.person("Bennett A. Landman"), as.person("Matthias Eisenmann"), as.person("Laura Aguilera Saiz"), as.person("Manuel Jorge Cardoso"), as.person("Lena Maier-Hein"), as.person("Annette Kopp-Schneider")),
+  textVersion="Wiesenfarth, M., Reinke, A., Landman, B.A., Eisenmann, M., Aguilera Saiz, L., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. Methods and open-source toolkit for analyzing and visualizing challenge results. Sci Rep 11, 2369 (2021). https://doi.org/10.1038/s41598-021-82017-6"
 )
diff --git a/inst/appdir/report.Rmd b/inst/appdir/report.Rmd
index 74cb6e2..b704672 100644
--- a/inst/appdir/report.Rmd
+++ b/inst/appdir/report.Rmd
@@ -1,440 +1,440 @@
 ---
 params:
   object: NA
   colors: NA
   name: NULL
   consensus: NA
   isMultiTask: NA
   bootstrappingEnabled: NA
   fig.format: NULL
   dpi: NULL
 title: "Benchmarking report for `r params$name` "
 author: "created by challengeR v`r packageVersion('challengeR')`"
 date: "`r Sys.setlocale('LC_TIME', 'English'); format(Sys.time(), '%d %B, %Y')`"
 editor_options: 
   chunk_output_type: console
 ---
 
 <!-- This text is outcommented -->
 <!-- R code chunks start with "```{r }" and end with "```" -->
 <!-- Please do not change anything inside of code chunks, otherwise any latex code is allowed -->
 
 <!-- inline code with `r 0` -->
 
 
 ```{r setup, include=FALSE}
 options(width=80)
 #out.format <- knitr::opts_knit$get("out.format")
 out.format <- knitr::opts_knit$get("rmarkdown.pandoc.to")
 
 img_template <- switch( out.format,
                      docx = list("img-params"=list(dpi=150,
                                                fig.width=6,
                                                fig.height=6,
                                                out.width="504px",
                                                out.height="504px")),
                      {
                        # default
                        list("img-params"=list( fig.width=7,
                                                fig.height = 3,
                                                dpi=300))
                      } )
 
 knitr::opts_template$set( img_template )
 
 knitr::opts_chunk$set(echo = F) # ,#fig.width=7,fig.height = 3,dpi=300,
 
 if (out.format != "docx") knitr::opts_chunk$set(fig.align = "center")
 if (!is.null(params$fig.format)) knitr::opts_chunk$set(dev = params$fig.format) # can be vector, e.g. fig.format=c('jpeg','png', 'pdf')
 if (!is.null(params$dpi)) knitr::opts_chunk$set(dpi = params$dpi) 
 
 theme_set(theme_light())
 
 isMultiTask = params$isMultiTask
 bootstrappingEnabled = params$bootstrappingEnabled
 
 ```
 
 
 ```{r }
 object = params$object
 
 if (isMultiTask) {
       if (is.numeric(params$consensus) & !is.null(names(params$consensus)) ){
       ordering_consensus <- names(params$consensus)[order(params$consensus)]
     } else if (is.character(ordering)){
       ordering_consensus=names(params$consensus)
     } else stop("Argument ordering has to be a named vector of ranks or a vector of algorithm names in the ranking order.")
 } else
 {
   ordering_consensus=names(sort(t(object$matlist[[1]][,"rank",drop=F])["rank",]))
 }
 color.fun=params$colors
 
 ```
 
 ```{r }
 
 challenge_multiple=object$data
 ranking.fun=object$FUN
 
 cols_numbered=cols=color.fun(length(ordering_consensus))
 names(cols)=ordering_consensus
 names(cols_numbered)= paste(1:length(cols),names(cols))
 
 if (bootstrappingEnabled) {
   boot_object = params$object
   challenge_multiple=boot_object$data
 
   ranking.fun=boot_object$FUN
   object=challenge_multiple%>%ranking.fun
   object$FUN.list = boot_object$FUN.list
 
   object$fulldata=boot_object$fulldata  # only not NULL if subset of algorithms used
 }
 
 n.tasks <- length(object$matlist)
 n.algorithms <- nrow((object$matlist[[1]]))
 ```
 
 This document presents a systematic report on the benchmark study "`r params$name`". Input data comprises raw metric values for all algorithms and cases. Generated plots are:
 
 ```{r, child=if (!isMultiTask && !bootstrappingEnabled) system.file("appdir", "overviewSingleTaskNoBootstrapping.Rmd", package="challengeR")}
 
 ```
 
 ```{r, child=if (!isMultiTask && bootstrappingEnabled) system.file("appdir", "overviewSingleTaskBootstrapping.Rmd", package="challengeR")}
 
 ```
 
 ```{r, child=if (isMultiTask && !bootstrappingEnabled) system.file("appdir", "overviewMultiTaskNoBootstrapping.Rmd", package="challengeR")}
 
 ```
 
 ```{r, child=if (isMultiTask && bootstrappingEnabled) system.file("appdir", "overviewMultiTaskBootstrapping.Rmd", package="challengeR")}
 
 ```
 
-Details can be found in Wiesenfarth et al. (2019).
+Details can be found in Wiesenfarth et al. (2021).
 
 ```{r,results='asis'}
 
 if (isMultiTask) {
   cat("# Rankings\n")
 } else {
   cat("# Ranking")
 }
 
 ```
 
 
 Algorithms within a task are ranked according to the following ranking scheme:
 
 ```{r,results='asis'}
 a=(  lapply(object$FUN.list[1:2],function(x) {
                 if (class(x)== "standardGeneric") return(paste0("aggregate using function ",
                                                                 x@generic
                                                                 )) 
                 else if (!is.character(x)) return(paste0("aggregate using function ",
                                                      paste(gsub("UseMethod","",
                                                                 deparse(functionBody(x))),
                                                            collapse=" ")
                                                      ))
                  else if (x=="rank") return(x)
                  else return(paste0("aggregate using function ",x))
   }))
 cat("&nbsp; &nbsp; *",paste0(a,collapse=" then "),"*",sep="")
 
 if (is.character(object$FUN.list[[1]]) && object$FUN.list[[1]]=="significance") cat("\n\n Column 'prop_significance' is equal to the number of pairwise significant test results for a given algorithm divided by the number of algorithms.")
 ```
 
 ```{r,results='asis'}
 
 if (isMultiTask) {
   cat("Ranking for each task:\n")
   
   for (t in 1:n.tasks){
     cat("\n",names(object$matlist)[t],": ")
     n.cases=nrow(challenge_multiple[[t]])/length(unique(challenge_multiple[[t]][[attr(challenge_multiple,"algorithm")]]))
 
     numberOfAlgorithms <- length(levels(challenge_multiple[[t]][[attr(challenge_multiple, "algorithm")]]))
     
     cat("\nThe analysis is based on",
         numberOfAlgorithms,
         "algorithms and",
         n.cases,
         "cases.",
         attr(object$data,"n.missing")[[t]], "missing cases have been found in the data set. ")
     
     if (nrow(attr(object$data,"missingData")[[t]])>0)  {
       if(attr(object$data,"n.missing")[[t]]==0 ) cat("However, ")
       else if(attr(object$data,"n.missing")[[t]]>0 ) cat("Additionally, ")
       cat("performance of not all algorithms has been observed for all cases in task '",
           names(object$matlist)[t],
           "'. Therefore, missings have been inserted in the following cases:")
       print(knitr::kable(as.data.frame(attr(object$data,"missingData")[[t]])))
     }
     
     if (nrow(attr(object$data,"missingData")[[t]])>0 | attr(object$data,"n.missing")[[t]]>0)  {
       if (is.numeric(attr(object$data,"na.treat"))) cat("All missings have been replaced by values of", attr(object$data,"na.treat"),".\n")
       else if (is.character(attr(object$data,"na.treat")) && attr(object$data,"na.treat")=="na.rm") cat("All missings have been removed.")
       else if (is.function(attr(object$data,"na.treat"))) {
         cat("Missings have been replaced using function ")
         print(attr(object$data,"na.treat"))
       }
       else if (is.character(object$FUN.list[[1]]) && object$FUN.list[[1]]=="rank") cat("Missings lead to the algorithm ranked last for the missing case.")
     }
     
     x=object$matlist[[t]]
     print(knitr::kable(x[order(x$rank),]))
   }
 } else {
 
   n.cases=nrow(challenge_multiple[[1]])/length(unique(challenge_multiple[[1]][[attr(challenge_multiple,"algorithm")]]))
   
   # Is subset of algorithms used?
   if (!is.null(object$fulldata[[1]])) {
     cat("The top ",
         length(levels(challenge_multiple[[1]][[attr(challenge_multiple, "algorithm")]])),
         " out of ",
         length(levels(object$fulldata[[1]][[attr(challenge_multiple, "algorithm")]])),
         " algorithms are considered.\n")
     cat("\nThe analysis is based on",
         n.cases,
         "cases. ")
   } else {
     cat("\nThe analysis is based on",
         length(levels(challenge_multiple[[1]][[attr(challenge_multiple, "algorithm")]])),
         "algorithms and",
         n.cases,
         "cases. ")
   }
 
   cat(attr(object$data,"n.missing")[[1]], "missing cases have been found in the data set. ")
 
   if (nrow(attr(object$data,"missingData")[[1]])>0)  {
     
     if(attr(object$data,"n.missing")[[1]]==0 ) cat("However, ")
     else if(attr(object$data,"n.missing")[[1]]>0 ) cat("Additionally, ")
     cat("performance of not all algorithms has been observed for all cases. Therefore, missings have been inserted in the following cases:")
     print(knitr::kable(as.data.frame(attr(object$data,"missingData")[[1]])))
    }
   
   if (nrow(attr(object$data,"missingData")[[1]])>0 | attr(object$data,"n.missing")[[1]]>0)  {
     if (is.numeric(attr(object$data,"na.treat"))) cat("All missings have been replaced by values of", attr(object$data,"na.treat"),".\n")
     else if (is.character(attr(object$data,"na.treat")) && attr(object$data,"na.treat")=="na.rm") cat("All missings have been removed.")
     else if (is.function(attr(object$data,"na.treat"))) {
       cat("Missings have been replaced using function ")
       print(attr(object$data,"na.treat"))
     }
       else if (is.character(object$FUN.list[[1]]) && object$FUN.list[[1]]=="rank") cat("Missings lead to the algorithm ranked last for the missing case.")
   }
   cat("\n\nRanking:")
   
   x=object$matlist[[1]]
   print(knitr::kable(x[order(x$rank),]))
 }
 
 ```
 
 \bigskip
 
 
 ```{r, child=if (isMultiTask) system.file("appdir", "consensusRanking.Rmd", package="challengeR")}
 
 ```
 
 \newpage
 
 # Visualization of raw assessment data
 
 ```{r,results='asis'}
 
 if (isMultiTask) {
   cat("The algorithms are ordered according to the computed ranks for each task.")
 }
 
 ```
 
 ## Dot- and boxplot
 
 *Dot- and boxplots* for visualizing raw assessment data separately for each algorithm. Boxplots representing descriptive statistics over all cases (median, quartiles and outliers) are combined with horizontally jittered dots representing individual cases.
 
 \bigskip
 
 ```{r boxplots}
 boxplot(object, size=.8) %++% scale_color_manual(values=cols)
 ```
 
 \newpage
 
 ## Podium plot
 *Podium plots* (see also Eugster et al., 2008) for visualizing raw assessment data. Upper part (spaghetti plot): Participating algorithms are color-coded, and each colored dot in the plot represents a metric value achieved with the respective algorithm. The actual metric value is encoded by the y-axis. Each podium (here: $p$=`r length(ordering_consensus)`) represents one possible rank, ordered from best (1) to last (here: `r length(ordering_consensus)`). The assignment of metric values (i.e. colored dots) to one of the podiums is based on the rank that the respective algorithm achieved on the corresponding case. Note that the plot part above each podium place is further subdivided into $p$ "columns", where each column represents one participating algorithm (here: $p=$ `r length(ordering_consensus)`).  Dots corresponding to identical cases are connected by a line, leading to the shown spaghetti structure. Lower part: Bar charts represent the relative frequency for each algorithm to achieve the rank encoded by the podium place. 
 
 ```{r, include=FALSE, fig.keep="none",dev=NULL}
 cex.legend = 1.2
 plot.new()
 
 # bottom legend  
   # get number of columns
     legend.width=1e5 #arbitrarily large
     ncol.legend <- n.algorithms + 1
     while(legend.width>12){
       ncol.legend <- ncol.legend-1
       algs=ordering_consensus
       l=legend("bottom", 
                paste0(1:length(algs),": ",algs), 
                 xpd=NA, 
                 lwd = 1, col =  cols, 
                 bg = NA,
                 ncol=ncol.legend,
                 bty="n",
                 cex=cex.legend, seg.len=1.1,
                 title="Rank: Alg.",
                 title.adj = 0,
                plot=F) 
       legend.width=grconvertX(l$rect$w,"user","inches")
     }
   w=0 
   h<-hh<- grconvertY(l$rect$h, to='ndc') - grconvertY(0, to='ndc')
   nrow.legend=length(unique(l$text$y))
 
 legend.height = nrow.legend* max( cex.legend * par()$cin[2],max(strheight(ordering_consensus, units = "inches", cex = cex.legend)))   
 addy= 6+ legend.height
 
 ```
 
 
 ```{r podium,eval=T,fig.width=12, fig.height=addy}
 #c(bottom, left, top, right
 op <- par(no.readonly = TRUE)
 
 hh<- grconvertY(legend.height, from="inches",to='ndc')
 par(pin=c(par()$pin[1],6),
         omd=c(0, 1, hh, 1),
         mar=c(par('mar')[1:3], 0)+c(-.5,0.5,-.5,0),
         cex.axis=1.5,
         cex.lab=1.5,
         cex.main=1.7)
 
 par(omi=c(legend.height+grconvertY(3, from="lines",to='inches'),0,0,0))
 
 l1=par('usr')[1] 
 l2=par('usr')[3] - (par()$mai[1]/par()$pin[2]) 
  
   
 set.seed(38)
 podium(object,
        col=cols,
        lines.show = T, lines.alpha = .4,
        dots.cex=.9,
        ylab="Metric value",
        layout.heights=c(1,.35),
        legendfn = function(algs, cols) {
          legend(l1,l2, 
                 paste0(1:length(algs),": ",algs), 
                 xpd=NA, 
                 lwd = 1, col =  cols, 
                 bg = NA,
                 ncol=ncol.legend,
                 bty="n",
                 cex=cex.legend, seg.len=1.1,
                 title="Rank: Alg.",
                 title.adj = 0
          ) 
        }
 )
 par(op)
   
 ```
 
 \newpage
 
 ## Ranking heatmap
 *Ranking heatmaps* for visualizing raw assessment data. Each cell $\left( i, A_j \right)$ shows the absolute frequency of cases in which algorithm $A_j$ achieved rank $i$.
 
 \bigskip
 
 ```{r rankingHeatmap,fig.width=9, fig.height=9,out.width='70%'}
 rankingHeatmap(object)
 ```
 
 \newpage
 
 # Visualization of ranking stability
 
 ```{r, child=if (bootstrappingEnabled) system.file("appdir", "visualizationBlobPlots.Rmd", package="challengeR")}
 
 ```
 
 ```{r, child=if (bootstrappingEnabled) system.file("appdir", "visualizationViolinPlots.Rmd", package="challengeR")}
 
 ```
 
 ## *Significance maps* for visualizing ranking stability based on statistical significance
 
 *Significance maps* depict incidence matrices of
 pairwise significant test results for the one-sided Wilcoxon signed rank test at a 5\% significance level with adjustment for multiple testing according to Holm. Yellow shading indicates that metric values from the algorithm on the x-axis were significantly superior to those from the algorithm on the y-axis, blue color indicates no significant difference.
 
 
 \bigskip
 
 ```{r significancemap,fig.width=6, fig.height=6,out.width='200%'}
 significanceMap(object,alpha=0.05,p.adjust.method="holm")
 ```
 
 \newpage
 
 ## Ranking robustness to ranking methods
 *Line plots* for visualizing ranking robustness across different ranking methods. Each algorithm is represented by one colored line. For each ranking method encoded on the x-axis, the height of the line represents the corresponding rank. Horizontal lines indicate identical ranks for all methods.
 
 \bigskip
 
 ```{r lineplot,fig.width=8, fig.height=6,out.width='95%'}
 
 if (n.tasks<=6 &n.algorithms<=10 ){
   methodsplot(challenge_multiple,
               ordering = ordering_consensus,
               na.treat=object$call[[1]][[1]]$na.treat) + 
     scale_color_manual(values=cols) 
 } else {
   for (subt in names(challenge_multiple)){
     dd=as.challenge(challenge_multiple[[subt]],
                     value=attr(challenge_multiple,"value"), 
                     algorithm=attr(challenge_multiple,"algorithm") ,
                     case=attr(challenge_multiple,"case"),
                     annotator = attr(challenge_multiple,"annotator"), 
                     by=attr(challenge_multiple,"by"),
                     smallBetter = attr(challenge_multiple,"smallBetter"),
                     na.treat=object$call[[1]][[1]]$na.treat
     )
     
     print(methodsplot(dd,
                       ordering = ordering_consensus) + 
             ggtitle(subt) +
             scale_color_manual(values=cols) +
             theme(legend.position = ifelse(n.algorithms>20,
                                            yes = "bottom",
                                            no = "right"))
           
     )
   }
 }
 ```
 
 
 
 ```{r, child=if (isMultiTask) system.file("appdir", "visualizationAcrossTasks.Rmd", package="challengeR")}
 
 ```
 
 \newpage
 
 # References
 
-Wiesenfarth, M., Reinke, A., Landman, B.A., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. (2019). Methods and open-source toolkit for analyzing and visualizing challenge results. *arXiv preprint arXiv:1910.05121*
+Wiesenfarth, M., Reinke, A., Landman, B.A., Eisenmann, M., Aguilera Saiz, L., Cardoso, M.J., Maier-Hein, L. and Kopp-Schneider, A. Methods and open-source toolkit for analyzing and visualizing challenge results. *Sci Rep* **11**, 2369 (2021). https://doi.org/10.1038/s41598-021-82017-6
 
 M. J. A. Eugster, T. Hothorn, and F. Leisch, “Exploratory
 and inferential analysis of benchmark experiments,”
 Institut fuer Statistik, Ludwig-Maximilians-Universitaet Muenchen, Germany, Technical Report 30,
 2008. [Online]. Available: http://epub.ub.uni-muenchen.de/4134/.
diff --git a/tests/testthat/test-blobPlotStabilityByAlgorithm.R b/tests/testthat/test-blobPlotStabilityByAlgorithm.R
index 66d0ff6..392fe06 100644
--- a/tests/testthat/test-blobPlotStabilityByAlgorithm.R
+++ b/tests/testthat/test-blobPlotStabilityByAlgorithm.R
@@ -1,140 +1,207 @@
 # Copyright (c) German Cancer Research Center (DKFZ)
 # All rights reserved.
 #
 # This file is part of challengeR.
 #
 # challengeR is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 2 of the License, or
 # (at your option) any later version.
 #
 # challengeR is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with challengeR. If not, see <https://www.gnu.org/licenses/>.
 
 test_that("blob plot for visualizing ranking stability by algorithm raises error for single-task data set", {
   data <- rbind(
     data.frame(algo="A1", value=0.8, case="C1"),
     data.frame(algo="A2", value=0.6, case="C1"),
     data.frame(algo="A3", value=0.4, case="C1"),
     data.frame(algo="A1", value=0.2, case="C2"),
     data.frame(algo="A2", value=0.1, case="C2"),
     data.frame(algo="A3", value=0.0, case="C2"))
 
   challenge <- as.challenge(data, taskName="T1", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   expect_error(stabilityByAlgorithm(rankingBootstrapped),
                "The stability of rankings by algorithm cannot be computed for less than two tasks.", fixed=TRUE)
 })
 
 test_that("blob plot for visualizing ranking stability by algorithm returns one plot for multi-task data set", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                         data.frame(algo="A1", value=0.8, case="C1"),
                         data.frame(algo="A2", value=0.6, case="C1"),
                         data.frame(algo="A3", value=0.4, case="C1"),
                         data.frame(algo="A1", value=0.2, case="C2"),
                         data.frame(algo="A2", value=0.1, case="C2"),
                         data.frame(algo="A3", value=0.0, case="C2")
                       ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.7, case="C2"),
                        data.frame(algo="A2", value=0.8, case="C2"),
                        data.frame(algo="A3", value=0.9, case="C2")
                      ))
 
   data <- rbind(dataTask1, dataTask2)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   actualPlot <- stabilityByAlgorithm(rankingBootstrapped)
   expect_is(actualPlot, "ggplot")
 })
 
 test_that("blob plot for visualizing ranking stability by algorithm returns a plot for each algorithm", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                        data.frame(algo="A1", value=0.8, case="C1"),
                        data.frame(algo="A2", value=0.6, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.2, case="C2"),
                        data.frame(algo="A2", value=0.1, case="C2"),
                        data.frame(algo="A3", value=0.0, case="C2")
                      ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.7, case="C2"),
                        data.frame(algo="A2", value=0.8, case="C2"),
                        data.frame(algo="A3", value=0.9, case="C2")
                      ))
 
   data <- rbind(dataTask1, dataTask2)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   meanRanks <- ranking%>%consensus(method = "euclidean")
 
   actualPlot <- stabilityByAlgorithm(rankingBootstrapped, ordering = names(meanRanks), single = TRUE)
   expect_equal(length(actualPlot), 3)
   expect_is(actualPlot[[1]], "ggplot")
   expect_is(actualPlot[[2]], "ggplot")
   expect_is(actualPlot[[3]], "ggplot")
 })
 
-
-
 test_that("Multi task bootstrapping, only one task with >1 test case stability plot works", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                        data.frame(algo="A1", value=0.8, case="C1"),
                        data.frame(algo="A2", value=0.6, case="C1")
                      ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1"),
                        data.frame(algo="A1", value=0.2, case="C2"),
                        data.frame(algo="A2", value=0.3, case="C2")
                      ))
   dataTask3 <- cbind(task="T3",
                      rbind(
                        data.frame(algo="A1", value=0.1, case="C1"),
                        data.frame(algo="A2", value=0.8, case="C1")
                      ))
-  
+
   data <- rbind(dataTask1, dataTask2, dataTask3)
-  
+
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
-  
+
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
   rankingBootstrapped <- ranking%>%bootstrap(nboot=3)
   meanRanks <- ranking%>%consensus(method = "euclidean")
   actualPlot <- stabilityByAlgorithm(rankingBootstrapped, ordering = names(meanRanks), single = FALSE)
   expect_is(actualPlot, "ggplot")
 })
+
+test_that("blob plot for visualizing ranking stability by algorithm returns one plot if #algorithms equals #tasks", {
+  dataTask1 <- cbind(task="T1",
+                     rbind(
+                       data.frame(algo="A1", value=0.8, case="C1"),
+                       data.frame(algo="A2", value=0.6, case="C1"),
+                       data.frame(algo="A1", value=0.2, case="C2"),
+                       data.frame(algo="A2", value=0.1, case="C2")
+                     ))
+  dataTask2 <- cbind(task="T2",
+                     rbind(
+                       data.frame(algo="A1", value=0.2, case="C1"),
+                       data.frame(algo="A2", value=0.3, case="C1"),
+                       data.frame(algo="A1", value=0.7, case="C2"),
+                       data.frame(algo="A2", value=0.8, case="C2")
+                     ))
+
+  data <- rbind(dataTask1, dataTask2)
+
+  challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
+
+  ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
+
+  set.seed(1)
+  rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
+
+  meanRanks <- ranking%>%consensus(method = "euclidean")
+
+  actualPlot <- stabilityByAlgorithm(rankingBootstrapped, ordering = names(meanRanks), single = FALSE)
+  expect_is(actualPlot, "ggplot")
+})
+
+test_that("blob plot for visualizing ranking stability by algorithm returns one plot if #algorithms < #tasks", {
+  dataTask1 <- cbind(task="T1",
+                     rbind(
+                       data.frame(algo="A1", value=0.8, case="C1"),
+                       data.frame(algo="A2", value=0.6, case="C1"),
+                       data.frame(algo="A1", value=0.8, case="C2"),
+                       data.frame(algo="A2", value=0.6, case="C2")
+                     ))
+  dataTask2 <- cbind(task="T2",
+                     rbind(
+                       data.frame(algo="A1", value=0.2, case="C1"),
+                       data.frame(algo="A2", value=0.3, case="C1"),
+                       data.frame(algo="A1", value=0.2, case="C2"),
+                       data.frame(algo="A2", value=0.3, case="C2")
+                     ))
+  dataTask3 <- cbind(task="T3",
+                     rbind(
+                       data.frame(algo="A1", value=0.1, case="C1"),
+                       data.frame(algo="A2", value=0.8, case="C1"),
+                       data.frame(algo="A1", value=0.1, case="C2"),
+                       data.frame(algo="A2", value=0.8, case="C2")
+                     ))
+
+  data <- rbind(dataTask1, dataTask2, dataTask3)
+
+  challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
+
+  ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
+
+  set.seed(1)
+  rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
+
+  meanRanks <- ranking%>%consensus(method = "euclidean")
+
+  actualPlot <- stabilityByAlgorithm(rankingBootstrapped, ordering = names(meanRanks), single = FALSE)
+  expect_is(actualPlot, "ggplot")
+})
diff --git a/tests/testthat/test-report.R b/tests/testthat/test-report.R
index 54c0fbd..f911f0b 100644
--- a/tests/testthat/test-report.R
+++ b/tests/testthat/test-report.R
@@ -1,485 +1,536 @@
 # Copyright (c) German Cancer Research Center (DKFZ)
 # All rights reserved.
 #
 # This file is part of challengeR.
 #
 # challengeR is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 2 of the License, or
 # (at your option) any later version.
 #
 # challengeR is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with challengeR. If not, see <https://www.gnu.org/licenses/>.
 
 test_that("PDF report for single-task data set without bootstrapping is created", {
   data <- rbind(
     data.frame(algo="A1", value=0.8, case="C1"),
     data.frame(algo="A2", value=0.6, case="C1"))
 
   challenge <- as.challenge(data, taskName="T1", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   ranking %>%
     report(title="Test Challenge",
            file="testthat_single_task_no_bootstrapping",
            format="PDF",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_single_task_no_bootstrapping.pdf"))
 
   # Clean up
   if (file.exists("testthat_single_task_no_bootstrapping.pdf")) {
     file.remove("testthat_single_task_no_bootstrapping.pdf")
   }
 
 })
 
 test_that("HTML report for single-task data set without bootstrapping is created", {
   data <- rbind(
     data.frame(algo="A1", value=0.8, case="C1"),
     data.frame(algo="A2", value=0.6, case="C1"))
 
   challenge <- as.challenge(data, taskName="T1", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   ranking %>%
     report(title="Test Challenge",
            file="testthat_single_task_no_bootstrapping",
            format="HTML",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_single_task_no_bootstrapping.html"))
 
   # Clean up
   if (file.exists("testthat_single_task_no_bootstrapping.html")) {
     file.remove("testthat_single_task_no_bootstrapping.html")
   }
 
 })
 
 test_that("Word report for single-task data set without bootstrapping is created", {
   data <- rbind(
     data.frame(algo="A1", value=0.8, case="C1"),
     data.frame(algo="A2", value=0.6, case="C1"))
 
   challenge <- as.challenge(data, taskName="T1", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   ranking %>%
     report(title="Test Challenge",
            file="testthat_single_task_no_bootstrapping",
            format="Word",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_single_task_no_bootstrapping.docx"))
 
   # Clean up
   if (file.exists("testthat_single_task_no_bootstrapping.docx")) {
     file.remove("testthat_single_task_no_bootstrapping.docx")
   }
 
 })
 
 test_that("PDF report for single-task data set with bootstrapping is created", {
   data <- rbind(
     data.frame(algo="A1", value=0.8, case="C1"),
     data.frame(algo="A2", value=0.6, case="C1"),
     data.frame(algo="A3", value=0.4, case="C1"),
     data.frame(algo="A1", value=0.2, case="C2"),
     data.frame(algo="A2", value=0.1, case="C2"),
     data.frame(algo="A3", value=0.0, case="C2"))
 
   challenge <- as.challenge(data, taskName="T1", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   rankingBootstrapped %>%
     report(title="Test Challenge",
            file="testthat_single_task_bootstrapping",
            format="PDF",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_single_task_bootstrapping.pdf"))
 
   # Clean up
   if (file.exists("testthat_single_task_bootstrapping.pdf")) {
     file.remove("testthat_single_task_bootstrapping.pdf")
   }
 
 })
 
 test_that("HTML report for single-task data set with bootstrapping is created", {
   data <- rbind(
     data.frame(algo="A1", value=0.8, case="C1"),
     data.frame(algo="A2", value=0.6, case="C1"),
     data.frame(algo="A3", value=0.4, case="C1"),
     data.frame(algo="A1", value=0.2, case="C2"),
     data.frame(algo="A2", value=0.1, case="C2"),
     data.frame(algo="A3", value=0.0, case="C2"))
 
   challenge <- as.challenge(data, taskName="T1", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   rankingBootstrapped %>%
     report(title="Test Challenge",
            file="testthat_single_task_bootstrapping",
            format="HTML",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_single_task_bootstrapping.html"))
 
   # Clean up
   if (file.exists("testthat_single_task_bootstrapping.html")) {
     file.remove("testthat_single_task_bootstrapping.html")
   }
 
 })
 
 test_that("Word report for single-task data set with bootstrapping is created", {
   data <- rbind(
     data.frame(algo="A1", value=0.8, case="C1"),
     data.frame(algo="A2", value=0.6, case="C1"),
     data.frame(algo="A3", value=0.4, case="C1"),
     data.frame(algo="A1", value=0.2, case="C2"),
     data.frame(algo="A2", value=0.1, case="C2"),
     data.frame(algo="A3", value=0.0, case="C2"))
 
   challenge <- as.challenge(data, taskName="T1", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   rankingBootstrapped %>%
     report(title="Test Challenge",
            file="testthat_single_task_bootstrapping",
            format="Word",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_single_task_bootstrapping.docx"))
 
   # Clean up
   if (file.exists("testthat_single_task_bootstrapping.docx")) {
     file.remove("testthat_single_task_bootstrapping.docx")
   }
 
 })
 
 test_that("PDF report for multi-task data set without bootstrapping is created", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                        data.frame(algo="A1", value=0.8, case="C1"),
                        data.frame(algo="A2", value=0.6, case="C1")
                      ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1")
                      ))
   dataTask3 <- cbind(task="T3",
                      rbind(
                        data.frame(algo="A1", value=0.1, case="C1"),
                        data.frame(algo="A2", value=0.8, case="C1")
                      ))
 
   data <- rbind(dataTask1, dataTask2, dataTask3)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   meanRanks <- ranking%>%consensus(method = "euclidean")
 
   ranking %>%
     report(consensus=meanRanks,
            title="Test Challenge",
            file="testthat_multi_task_no_bootstrapping",
            format="PDF",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_multi_task_no_bootstrapping.pdf"))
 
   # Clean up
   if (file.exists("testthat_multi_task_no_bootstrapping.pdf")) {
     file.remove("testthat_multi_task_no_bootstrapping.pdf")
   }
 
 })
 
 test_that("HTML report for multi-task data set without bootstrapping is created", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                        data.frame(algo="A1", value=0.8, case="C1"),
                        data.frame(algo="A2", value=0.6, case="C1")
                      ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1")
                      ))
   dataTask3 <- cbind(task="T3",
                      rbind(
                        data.frame(algo="A1", value=0.1, case="C1"),
                        data.frame(algo="A2", value=0.8, case="C1")
                      ))
 
   data <- rbind(dataTask1, dataTask2, dataTask3)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   meanRanks <- ranking%>%consensus(method = "euclidean")
 
   ranking %>%
     report(consensus=meanRanks,
            title="Test Challenge",
            file="testthat_multi_task_no_bootstrapping",
            format="HTML",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_multi_task_no_bootstrapping.html"))
 
   # Clean up
   if (file.exists("testthat_multi_task_no_bootstrapping.html")) {
     file.remove("testthat_multi_task_no_bootstrapping.html")
   }
 
 })
 
 test_that("Word report for multi-task data set without bootstrapping is created", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                        data.frame(algo="A1", value=0.8, case="C1"),
                        data.frame(algo="A2", value=0.6, case="C1")
                      ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1")
                      ))
   dataTask3 <- cbind(task="T3",
                      rbind(
                        data.frame(algo="A1", value=0.1, case="C1"),
                        data.frame(algo="A2", value=0.8, case="C1")
                      ))
 
   data <- rbind(dataTask1, dataTask2, dataTask3)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   meanRanks <- ranking%>%consensus(method = "euclidean")
 
   ranking %>%
     report(consensus=meanRanks,
            title="Test Challenge",
            file="testthat_multi_task_no_bootstrapping",
            format="Word",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_multi_task_no_bootstrapping.docx"))
 
   # Clean up
   if (file.exists("testthat_multi_task_no_bootstrapping.docx")) {
     file.remove("testthat_multi_task_no_bootstrapping.docx")
   }
 
 })
 
 test_that("PDF report for multi-task data set with bootstrapping is created", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                        data.frame(algo="A1", value=0.8, case="C1"),
                        data.frame(algo="A2", value=0.6, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.2, case="C2"),
                        data.frame(algo="A2", value=0.1, case="C2"),
                        data.frame(algo="A3", value=0.0, case="C2")
                      ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.7, case="C2"),
                        data.frame(algo="A2", value=0.8, case="C2"),
                        data.frame(algo="A3", value=0.9, case="C2")
                      ))
   dataTask3 <- cbind(task="T3",
                      rbind(
                        data.frame(algo="A1", value=0.1, case="C1"),
                        data.frame(algo="A2", value=0.2, case="C1"),
                        data.frame(algo="A3", value=0.3, case="C1"),
                        data.frame(algo="A1", value=0.6, case="C2"),
                        data.frame(algo="A2", value=0.7, case="C2"),
                        data.frame(algo="A3", value=0.8, case="C2")
                      ))
 
   data <- rbind(dataTask1, dataTask2, dataTask3)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   meanRanks <- ranking%>%consensus(method = "euclidean")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   rankingBootstrapped %>%
     report(consensus=meanRanks,
            title="Test Challenge",
            file="testthat_multi_task_bootstrapping",
            format="PDF",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_multi_task_bootstrapping.pdf"))
 
   # Clean up
   if (file.exists("testthat_multi_task_bootstrapping.pdf")) {
     file.remove("testthat_multi_task_bootstrapping.pdf")
   }
 
 })
 
 test_that("HTML report for multi-task data set with bootstrapping is created", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                        data.frame(algo="A1", value=0.8, case="C1"),
                        data.frame(algo="A2", value=0.6, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.2, case="C2"),
                        data.frame(algo="A2", value=0.1, case="C2"),
                        data.frame(algo="A3", value=0.0, case="C2")
                      ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.7, case="C2"),
                        data.frame(algo="A2", value=0.8, case="C2"),
                        data.frame(algo="A3", value=0.9, case="C2")
                      ))
   dataTask3 <- cbind(task="T3",
                      rbind(
                        data.frame(algo="A1", value=0.1, case="C1"),
                        data.frame(algo="A2", value=0.2, case="C1"),
                        data.frame(algo="A3", value=0.3, case="C1"),
                        data.frame(algo="A1", value=0.6, case="C2"),
                        data.frame(algo="A2", value=0.7, case="C2"),
                        data.frame(algo="A3", value=0.8, case="C2")
                      ))
 
   data <- rbind(dataTask1, dataTask2, dataTask3)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   meanRanks <- ranking%>%consensus(method = "euclidean")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   rankingBootstrapped %>%
     report(consensus=meanRanks,
            title="Test Challenge",
            file="testthat_multi_task_bootstrapping",
            format="HTML",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_multi_task_bootstrapping.html"))
 
   # Clean up
   if (file.exists("testthat_multi_task_bootstrapping.html")) {
     file.remove("testthat_multi_task_bootstrapping.html")
   }
 
 })
 
 test_that("Word report for multi-task data set with bootstrapping is created", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                        data.frame(algo="A1", value=0.8, case="C1"),
                        data.frame(algo="A2", value=0.6, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.2, case="C2"),
                        data.frame(algo="A2", value=0.1, case="C2"),
                        data.frame(algo="A3", value=0.0, case="C2")
                      ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.7, case="C2"),
                        data.frame(algo="A2", value=0.8, case="C2"),
                        data.frame(algo="A3", value=0.9, case="C2")
                      ))
   dataTask3 <- cbind(task="T3",
                      rbind(
                        data.frame(algo="A1", value=0.1, case="C1"),
                        data.frame(algo="A2", value=0.2, case="C1"),
                        data.frame(algo="A3", value=0.3, case="C1"),
                        data.frame(algo="A1", value=0.6, case="C2"),
                        data.frame(algo="A2", value=0.7, case="C2"),
                        data.frame(algo="A3", value=0.8, case="C2")
                      ))
 
   data <- rbind(dataTask1, dataTask2, dataTask3)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   meanRanks <- ranking%>%consensus(method = "euclidean")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   rankingBootstrapped %>%
     report(consensus=meanRanks,
            title="Test Challenge",
            file="testthat_multi_task_bootstrapping",
            format="Word",
            clean=TRUE,
            open=FALSE)
 
   expect_true(file.exists("testthat_multi_task_bootstrapping.docx"))
 
   # Clean up
   if (file.exists("testthat_multi_task_bootstrapping.docx")) {
     file.remove("testthat_multi_task_bootstrapping.docx")
   }
 
 })
+
+test_that("PDF report for multi-task data set with bootstrapping is created (#algorithms < #tasks)", {
+  dataTask1 <- cbind(task="T1",
+                     rbind(
+                       data.frame(algo="A1", value=0.80, case="C1"),
+                       data.frame(algo="A2", value=0.60, case="C1"),
+                       data.frame(algo="A1", value=0.85, case="C2"),
+                       data.frame(algo="A2", value=0.65, case="C2")
+                     ))
+  dataTask2 <- cbind(task="T2",
+                     rbind(
+                       data.frame(algo="A1", value=0.20, case="C1"),
+                       data.frame(algo="A2", value=0.30, case="C1"),
+                       data.frame(algo="A1", value=0.25, case="C2"),
+                       data.frame(algo="A2", value=0.35, case="C2")
+                     ))
+  dataTask3 <- cbind(task="T3",
+                     rbind(
+                       data.frame(algo="A1", value=0.10, case="C1"),
+                       data.frame(algo="A2", value=0.80, case="C1"),
+                       data.frame(algo="A1", value=0.15, case="C2"),
+                       data.frame(algo="A2", value=0.85, case="C2")
+                     ))
+
+  data <- rbind(dataTask1, dataTask2, dataTask3)
+
+  challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
+
+  ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
+
+  meanRanks <- ranking%>%consensus(method = "euclidean")
+
+  set.seed(1)
+  rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
+
+  rankingBootstrapped %>%
+    report(consensus=meanRanks,
+           title="Test Challenge",
+           file="testthat_multi_task_bootstrapping_more_tasks_than_algorithms",
+           format="PDF",
+           clean=TRUE,
+           open=FALSE)
+
+  expect_true(file.exists("testthat_multi_task_bootstrapping_more_tasks_than_algorithms.pdf"))
+
+  # Clean up
+  if (file.exists("testthat_multi_task_bootstrapping_more_tasks_than_algorithms.pdf")) {
+    file.remove("testthat_multi_task_bootstrapping_more_tasks_than_algorithms.pdf")
+  }
+
+})
diff --git a/tests/testthat/test-stackedBarPlotStabilityByAlgorithm.R b/tests/testthat/test-stackedBarPlotStabilityByAlgorithm.R
index c07078f..b633905 100644
--- a/tests/testthat/test-stackedBarPlotStabilityByAlgorithm.R
+++ b/tests/testthat/test-stackedBarPlotStabilityByAlgorithm.R
@@ -1,70 +1,139 @@
 # Copyright (c) German Cancer Research Center (DKFZ)
 # All rights reserved.
 #
 # This file is part of challengeR.
 #
 # challengeR is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 2 of the License, or
 # (at your option) any later version.
 #
 # challengeR is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with challengeR. If not, see <https://www.gnu.org/licenses/>.
 
 test_that("stacked bar plot for visualizing ranking stability by algorithm raises error for single-task data set", {
   data <- rbind(
     data.frame(algo="A1", value=0.8, case="C1"),
     data.frame(algo="A2", value=0.6, case="C1"),
     data.frame(algo="A3", value=0.4, case="C1"),
     data.frame(algo="A1", value=0.2, case="C2"),
     data.frame(algo="A2", value=0.1, case="C2"),
     data.frame(algo="A3", value=0.0, case="C2"))
 
   challenge <- as.challenge(data, taskName="T1", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
-  expect_error(stabilityByAlgorithm(rankingBootstrapped, stacked =TRUE),
+  expect_error(stabilityByAlgorithm(rankingBootstrapped, stacked = TRUE),
                "The stability of rankings by algorithm cannot be computed for less than two tasks.", fixed=TRUE)
 })
 
 test_that("stacked bar plot for visualizing ranking stability by algorithm returns one plot for multi-task data set", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                         data.frame(algo="A1", value=0.8, case="C1"),
                         data.frame(algo="A2", value=0.6, case="C1"),
                         data.frame(algo="A3", value=0.4, case="C1"),
                         data.frame(algo="A1", value=0.2, case="C2"),
                         data.frame(algo="A2", value=0.1, case="C2"),
                         data.frame(algo="A3", value=0.0, case="C2")
                       ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.7, case="C2"),
                        data.frame(algo="A2", value=0.8, case="C2"),
                        data.frame(algo="A3", value=0.9, case="C2")
                      ))
 
   data <- rbind(dataTask1, dataTask2)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
-  actualPlot <- stabilityByAlgorithm(rankingBootstrapped, stacked =TRUE)
+  actualPlot <- stabilityByAlgorithm(rankingBootstrapped, stacked = TRUE)
+  expect_is(actualPlot, "ggplot")
+})
+
+test_that("stacked bar plot for visualizing ranking stability by algorithm returns one plot if #algorithms equals #tasks", {
+  dataTask1 <- cbind(task="T1",
+                     rbind(
+                       data.frame(algo="A1", value=0.8, case="C1"),
+                       data.frame(algo="A2", value=0.6, case="C1"),
+                       data.frame(algo="A1", value=0.2, case="C2"),
+                       data.frame(algo="A2", value=0.1, case="C2")
+                     ))
+  dataTask2 <- cbind(task="T2",
+                     rbind(
+                       data.frame(algo="A1", value=0.2, case="C1"),
+                       data.frame(algo="A2", value=0.3, case="C1"),
+                       data.frame(algo="A1", value=0.7, case="C2"),
+                       data.frame(algo="A2", value=0.8, case="C2")
+                     ))
+
+  data <- rbind(dataTask1, dataTask2)
+
+  challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
+
+  ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
+
+  set.seed(1)
+  rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
+
+  meanRanks <- ranking%>%consensus(method = "euclidean")
+
+  actualPlot <- stabilityByAlgorithm(rankingBootstrapped, stacked = TRUE)
+  expect_is(actualPlot, "ggplot")
+})
+
+test_that("stacked bar plot for visualizing ranking stability by algorithm returns one plot if #algorithms < #tasks", {
+  dataTask1 <- cbind(task="T1",
+                     rbind(
+                       data.frame(algo="A1", value=0.8, case="C1"),
+                       data.frame(algo="A2", value=0.6, case="C1"),
+                       data.frame(algo="A1", value=0.8, case="C2"),
+                       data.frame(algo="A2", value=0.6, case="C2")
+                     ))
+  dataTask2 <- cbind(task="T2",
+                     rbind(
+                       data.frame(algo="A1", value=0.2, case="C1"),
+                       data.frame(algo="A2", value=0.3, case="C1"),
+                       data.frame(algo="A1", value=0.2, case="C2"),
+                       data.frame(algo="A2", value=0.3, case="C2")
+                     ))
+  dataTask3 <- cbind(task="T3",
+                     rbind(
+                       data.frame(algo="A1", value=0.1, case="C1"),
+                       data.frame(algo="A2", value=0.8, case="C1"),
+                       data.frame(algo="A1", value=0.1, case="C2"),
+                       data.frame(algo="A2", value=0.8, case="C2")
+                     ))
+
+  data <- rbind(dataTask1, dataTask2, dataTask3)
+
+  challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
+
+  ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
+
+  set.seed(1)
+  rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
+
+  meanRanks <- ranking%>%consensus(method = "euclidean")
+
+  actualPlot <- stabilityByAlgorithm(rankingBootstrapped, stacked = TRUE)
   expect_is(actualPlot, "ggplot")
 })
diff --git a/tests/testthat/test-stackedFrequencyPlotStabilityByAlgorithm.R b/tests/testthat/test-stackedFrequencyPlotStabilityByAlgorithm.R
index 859cfb4..2c3a9ac 100644
--- a/tests/testthat/test-stackedFrequencyPlotStabilityByAlgorithm.R
+++ b/tests/testthat/test-stackedFrequencyPlotStabilityByAlgorithm.R
@@ -1,108 +1,177 @@
 # Copyright (c) German Cancer Research Center (DKFZ)
 # All rights reserved.
 #
 # This file is part of challengeR.
 #
 # challengeR is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 2 of the License, or
 # (at your option) any later version.
 #
 # challengeR is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
 # along with challengeR. If not, see <https://www.gnu.org/licenses/>.
 
 test_that("stacked frequency plot for visualizing ranking stability by algorithm raises error for single-task data set", {
   data <- rbind(
     data.frame(algo="A1", value=0.8, case="C1"),
     data.frame(algo="A2", value=0.6, case="C1"),
     data.frame(algo="A3", value=0.4, case="C1"),
     data.frame(algo="A1", value=0.2, case="C2"),
     data.frame(algo="A2", value=0.1, case="C2"),
     data.frame(algo="A3", value=0.0, case="C2"))
 
   challenge <- as.challenge(data, taskName="T1", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   expect_error(stabilityByAlgorithm(rankingBootstrapped, stacked = TRUE),
                "The stability of rankings by algorithm cannot be computed for less than two tasks.", fixed=TRUE)
 })
 
 test_that("stacked frequency plot for visualizing ranking stability by algorithm returns one plot for multi-task data set", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                        data.frame(algo="A1", value=0.8, case="C1"),
                        data.frame(algo="A2", value=0.6, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.2, case="C2"),
                        data.frame(algo="A2", value=0.1, case="C2"),
                        data.frame(algo="A3", value=0.0, case="C2")
                      ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.7, case="C2"),
                        data.frame(algo="A2", value=0.8, case="C2"),
                        data.frame(algo="A3", value=0.9, case="C2")
                      ))
 
   data <- rbind(dataTask1, dataTask2)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   actualPlot <- stabilityByAlgorithm(rankingBootstrapped, stacked = TRUE)
   expect_is(actualPlot, "ggplot")
 })
 
 test_that("stacked frequency plot for visualizing ranking stability by algorithm returns a plot for each algorithm", {
   dataTask1 <- cbind(task="T1",
                      rbind(
                        data.frame(algo="A1", value=0.8, case="C1"),
                        data.frame(algo="A2", value=0.6, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.2, case="C2"),
                        data.frame(algo="A2", value=0.1, case="C2"),
                        data.frame(algo="A3", value=0.0, case="C2")
                      ))
   dataTask2 <- cbind(task="T2",
                      rbind(
                        data.frame(algo="A1", value=0.2, case="C1"),
                        data.frame(algo="A2", value=0.3, case="C1"),
                        data.frame(algo="A3", value=0.4, case="C1"),
                        data.frame(algo="A1", value=0.7, case="C2"),
                        data.frame(algo="A2", value=0.8, case="C2"),
                        data.frame(algo="A3", value=0.9, case="C2")
                      ))
 
   data <- rbind(dataTask1, dataTask2)
 
   challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
 
   ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
 
   set.seed(1)
   rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
 
   meanRanks <- ranking%>%consensus(method = "euclidean")
 
   actualPlot <- stabilityByAlgorithm(rankingBootstrapped, ordering = names(meanRanks), stacked = TRUE, single = TRUE)
   expect_equal(length(actualPlot), 3)
   expect_is(actualPlot[[1]], "ggplot")
   expect_is(actualPlot[[2]], "ggplot")
   expect_is(actualPlot[[3]], "ggplot")
 })
+
+test_that("stacked frequency plot for visualizing ranking stability by algorithm returns one plot if #algorithms equals #tasks", {
+  dataTask1 <- cbind(task="T1",
+                     rbind(
+                       data.frame(algo="A1", value=0.8, case="C1"),
+                       data.frame(algo="A2", value=0.6, case="C1"),
+                       data.frame(algo="A1", value=0.2, case="C2"),
+                       data.frame(algo="A2", value=0.1, case="C2")
+                     ))
+  dataTask2 <- cbind(task="T2",
+                     rbind(
+                       data.frame(algo="A1", value=0.2, case="C1"),
+                       data.frame(algo="A2", value=0.3, case="C1"),
+                       data.frame(algo="A1", value=0.7, case="C2"),
+                       data.frame(algo="A2", value=0.8, case="C2")
+                     ))
+
+  data <- rbind(dataTask1, dataTask2)
+
+  challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
+
+  ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
+
+  set.seed(1)
+  rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
+
+  meanRanks <- ranking%>%consensus(method = "euclidean")
+
+  actualPlot <- stabilityByAlgorithm(rankingBootstrapped, ordering = names(meanRanks), stacked = TRUE, single = FALSE)
+  expect_is(actualPlot, "ggplot")
+})
+
+test_that("stacked frequency plot for visualizing ranking stability by algorithm returns one plot if #algorithms < #tasks", {
+  dataTask1 <- cbind(task="T1",
+                     rbind(
+                       data.frame(algo="A1", value=0.8, case="C1"),
+                       data.frame(algo="A2", value=0.6, case="C1"),
+                       data.frame(algo="A1", value=0.8, case="C2"),
+                       data.frame(algo="A2", value=0.6, case="C2")
+                     ))
+  dataTask2 <- cbind(task="T2",
+                     rbind(
+                       data.frame(algo="A1", value=0.2, case="C1"),
+                       data.frame(algo="A2", value=0.3, case="C1"),
+                       data.frame(algo="A1", value=0.2, case="C2"),
+                       data.frame(algo="A2", value=0.3, case="C2")
+                     ))
+  dataTask3 <- cbind(task="T3",
+                     rbind(
+                       data.frame(algo="A1", value=0.1, case="C1"),
+                       data.frame(algo="A2", value=0.8, case="C1"),
+                       data.frame(algo="A1", value=0.1, case="C2"),
+                       data.frame(algo="A2", value=0.8, case="C2")
+                     ))
+
+  data <- rbind(dataTask1, dataTask2, dataTask3)
+
+  challenge <- as.challenge(data, by="task", algorithm="algo", case="case", value="value", smallBetter=FALSE)
+
+  ranking <- challenge%>%aggregateThenRank(FUN=median, ties.method="min")
+
+  set.seed(1)
+  rankingBootstrapped <- ranking%>%bootstrap(nboot=10)
+
+  meanRanks <- ranking%>%consensus(method = "euclidean")
+
+  actualPlot <- stabilityByAlgorithm(rankingBootstrapped, ordering = names(meanRanks), stacked = TRUE, single = FALSE)
+  expect_is(actualPlot, "ggplot")
+})
diff --git a/vignettes/visualizations.Rmd b/vignettes/visualizations.Rmd
index 74c481f..93c3796 100644
--- a/vignettes/visualizations.Rmd
+++ b/vignettes/visualizations.Rmd
@@ -1,167 +1,167 @@
 ---
 title: "Visualizations"
 output: rmarkdown::html_vignette
 vignette: >
   %\VignetteIndexEntry{Visualizations}
   %\VignetteEncoding{UTF-8}
   %\VignetteEngine{knitr::rmarkdown}
 ---
 
 ```{r, include = FALSE}
 knitr::opts_chunk$set(
   collapse = TRUE,
   comment = "#>"
 )
 ```
 
 ```{r setup}
 library(challengeR)
 ```
 
 The package offers an intuitive way to gain important insights into the relative and absolute performance of algorithms. It enables you to generate a benchmarking report that contains visualizations and respective explanations. An overview of all available visualizations is provided on this page demonstrating the use of their corresponding plot functions. This might be of interest if you want to generate the plots separately (e.g. to apply other styles).
 
 The provided plots are described in the following sections:
 
 * Visualizing assessment data
 * Visualizing ranking stability
 * Visualizing cross-task insights
 
-Details can be found in [Wiesenfarth et al. (2019)](https://arxiv.org/abs/1910.05121).
+Details can be found in [Wiesenfarth et al. (2021)](https://rdcu.be/ceiaN).
 
 # Visualizing assessment data
 
 ```{r}
 data <- read.csv(system.file("extdata", "data_matrix.csv", package = "challengeR", mustWork = TRUE))
 
 challenge <- as.challenge(data, by = "task", algorithm = "alg_name", case = "case", value = "value", smallBetter = FALSE)
 
 ranking <- challenge%>%aggregateThenRank(FUN = mean, ties.method = "min")
 ```
 
 ## Dot- and boxplots
 Dot- and boxplots visualize the assessment data separately for each algorithm. Boxplots representing descriptive statistics for all test cases
 (median, quartiles and outliers) are combined with horizontally jittered dots representing individual test cases.
 ```{r boxplot}
 boxplot(ranking)
 ```
 
 ## Podium plots
 Upper part of the podium plot: Algorithms are color-coded, and each colored dot in the plot represents a performance value achieved with the respective algorithm. The actual value is encoded by the y-axis. Each podium (here: $p = 5$) represents one possible rank, ordered from best (1) to worst (here: 5). The assignment of values (i.e. colored dots) to one of the podiums is based on the rank that the respective algorithm achieved on the corresponding test case.Note that the plot part above each podium place is further subdivided into $p$ “columns”, where each column represents one algorithm. Dots corresponding to identical test cases are connected by a line, producing the spaghetti structure shown here. Lower part: Bar charts represent the relative frequency at which each algorithm actually achieves the rank encoded by the podium place.
 ```{r podium,fig.width=5}
  podium(ranking,
         layout.heights = c(.6, 0.4))
 ```
 
 ## Ranking heatmaps
 In a ranking heatmap, each cell $\left( i, A_j \right)$ shows the absolute frequency of cases in which algorithm $A_j$ achieved rank $i$.
 ```{r rankingHeatmap}
 rankingHeatmap(ranking)
 ```
 
 # Visualizing ranking stability
 
-The ranking robustness can by analyzed with respect to the ranking method used (see [Wiesenfarth et al. (2019)](https://arxiv.org/abs/1910.05121) for different ranking methods).
+The ranking robustness can by analyzed with respect to the ranking method used (see [Wiesenfarth et al. (2021)](https://rdcu.be/ceiaN) for different ranking methods).
 
 ## Line plots
 Line plots visualize the robustness of ranking across different ranking methods. Each algorithm is represented by one colored line.
 For each ranking method encoded on the x-axis, the height of the line represents the corresponding rank. Horizontal lines indicate identical ranks for all methods.
 ```{r lineplot, fig.width = 7}
 methodsplot(challenge)
 ```
 
 For a specific ranking method, the ranking stability can be investigated via bootstrapping and the testing approach.
 
 A ranking object containing the bootstrapping samples has to be created which serves as the basis for the plots.
 
 ```{r bootstrapping, results = "hide"}
 set.seed(1)
 rankingBootstrapped <- ranking%>%bootstrap(nboot = 1000)
 ```
 
 ## Blob plots
 
 Blob plots for visualizing ranking stability are based on bootstrap sampling. Algorithms are color-coded, and the area of each blob at position $\left( A_i, \text{rank } j \right)$ is proportional to the relative frequency $A_i$ achieved rank $j$ (here across $b = 1000$ bootstrap samples). The median rank for each algorithm is indicated by a black cross. 95% bootstrap intervals across bootstrap samples (ranging from the 2.5th to
 the 97.5th percentile of the bootstrap distribution) are indicated by black lines.
 
 ```{r stabilityByTask1, fig.width = 7}
 stabilityByTask(rankingBootstrapped)
 ```
 
 ## Violin plots
 
 Violin plots provide a more condensed way to analyze bootstrap results. In these plots, the focus is on the comparison of the ranking list computed on the full assessment data and the individual bootstrap samples, respectively. Kendall’s $\tau$ is chosen for comparison as it is has an upper and lower bound (+1/-1). Kendall’s $\tau$ is computed for each pair of rankings, and a violin plot that simultaneously depicts a boxplot and a density plot is generated from the results.
 
 ```{r violin, results = "hide"}
 violin(rankingBootstrapped)
 ```
 
 ## Significance maps
 
 Significance maps visualize ranking stability based on statistical significance. They depict incidence matrices of pairwise significant test results for the one-sided Wilcoxon signed rank test at 5% significance level with adjustment for multiple testing according to Holm. Yellow shading indicates that performance values of the algorithm on the x-axis are significantly superior to those from the algorithm on the y-axis, blue color indicates no significant difference.
 
 ```{r significanceMap}
 significanceMap(ranking)
 ```
 
 # Visualizing cross-task insights
 
 For cross-task insights, a consensus ranking (rank aggregation across tasks) has to be given additionally. The consensus ranking according to mean ranks across tasks is computed here.
 
 ```{r}
 meanRanks <- ranking%>%consensus(method = "euclidean")
 ```
 The consensus ranking is given according to mean ranks across tasks if method="euclidean" where in case of ties (equal ranks for multiple algorithms) the average rank is used, i.e. ties.method="average".
 
 ## Characterization of algorithms
 
 The primary goal of most multi-task challenges is to identify methods that consistently outperform competing algorithms across all tasks. We propose the followig methods for analyzing this:
 
 ### Blob plots visualizing the ranking variability across tasks
 
 Blob plots visualize the distribution of ranks across tasks. All ranks that an algorithm achieved in any task are displayed along the y-axis, with the area of the blob being proportional to the frequency. If all tasks provided the same stable ranking, narrow intervals around the diagonal would be expected. Consensus rankings above algorithm names highlight the presence of ties.
 
 ```{r stability, fig.width = 5, fig.height = 4}
 stability(ranking, ordering = names(meanRanks))
 ```
 
 ### Blob plots visualizing the ranking variability based on bootstrapping
 
 This variant of the blob plot approach involves replacing the algorithms on the x-axis with the tasks and then generating a separate plot for each algorithm. This allows assessing the variability of rankings for each algorithm across multiple tasks and bootstrap samples. Here, color coding is used for the tasks, and separation by algorithm enables a relatively straightforward strength-weaknesses analysis for individual methods.
 
 ```{r stabilityByAlgorithm1, fig.width = 7, fig.height = 5}
 stabilityByAlgorithm(rankingBootstrapped, ordering = names(meanRanks))
 ```
 
 ### Stacked frequency plots visualizing the ranking variability based on bootstrapping
 
 An alternative representation is provided by a stacked frequency plot of the observed ranks, separated by algorithm. Observed ranks across bootstrap samples are displayed with coloring according to the task. For algorithms that achieve the same rank in different tasks for the full assessment data set, vertical lines are on top of each other. Vertical lines allow to compare the achieved rank of each algorithm over different tasks.
 
 ```{r stabilityByAlgorithm2, fig.width = 7, fig.height = 5}
 stabilityByAlgorithm(rankingBootstrapped, ordering = names(meanRanks), stacked = TRUE)
 ```
 
 ## Characterization of tasks
 
 It may also be useful to structure the analysis around the different tasks. This section proposes visualizations to analyze and compare tasks of a competition.
 
 ### Blob plots visualizing bootstrap results
 
 Bootstrap results can be shown in a blob plot showing one plot for each task. Algorithms should be ordered according to
 the consensus ranking. In this view, the spread of the blobs for each algorithm can be compared across tasks. Deviations from the diagonal indicate deviations from the consensus ranking (over tasks). Specifically, if rank distribution of an algorithm is consistently below the diagonal, the algorithm performed better in this task than on average across tasks, while if the rank distribution of an algorithm is consistently above the diagonal, the algorithm performed worse in this task than on average across tasks. At the bottom of each panel, ranks for each algorithm in the tasks are provided.
 
 ```{r stabilityByTask2, fig.width = 7, fig.height = 3.5}
 stabilityByTask(rankingBootstrapped, ordering = names(meanRanks))
 ```
 
 ### Violin plots visualizing bootstrap results
 
 To obtain a more condensed visualization, violin plots (see above) can be applied separately to all tasks. The overall stability of the rankings can then be compared by assessing the locations and lengths of the violins.
 
 ### Cluster analysis
 
 There is increasing interest in assessing the similarity of the tasks, e.g., for pre-training a machine learning algorithm. A potential approach to this could involve the comparison of the rankings for a challenge. Given the same teams participate in all tasks, it may be of interest to cluster tasks into groups where rankings of algorithms are similar and to identify tasks which lead to very dissimilar rankings of algorithms. To enable such an analysis, we propose the generation of a dendrogram from hierarchical cluster analysis. Here, it depicts clusters according to a chosen distance measure (Spearman’s footrule) as well as a chosen agglomeration method (complete agglomeration).
 
 
 ```{r dendrogram, fig.width = 7, fig.height = 3.5}
 dendrogram(ranking)
 ```