Page MenuHomePhabricator

eisenman (Matthias Eisenmann)
User

Projects (8)

User Details

User Since
Aug 1 2016, 12:10 PM (216 w, 3 d)

Recent Activity

Today

eisenman edited projects for T27326: autolayout networks in multi task report ugly, added: challengeR; removed challengeR (v1.0).

I agree!

Fri, Sep 25, 8:33 AM · challengeR

Yesterday

eisenman moved T27774: wrong text in report on algorithm subsets from In Progress to Done on the challengeR (v1.0) board.

It's important to mention that the subset of algorithms should be drawn from the final ranking to avoid wrong results. So if bootstrapping should be performed, create the subset from the bootstrapped ranking, not from the initial ranking that is passed to perform bootstrapping.

Thu, Sep 24, 5:15 PM · challengeR (v1.0)
eisenman committed rCHALLENGER595046260c52: Merge branch 'feature/T27774-FixReportingOfAlgorithmSubset' into develop (authored by eisenman).
Merge branch 'feature/T27774-FixReportingOfAlgorithmSubset' into develop
Thu, Sep 24, 5:09 PM
eisenman committed rCHALLENGER6fbbc9bb3920: Test that full data set is preserved (authored by eisenman).
Test that full data set is preserved
Thu, Sep 24, 5:08 PM
eisenman committed rCHALLENGERb55aaaf309cf: Fix reporting of algorithm subset (authored by eisenman).
Fix reporting of algorithm subset
Thu, Sep 24, 5:08 PM
eisenman triaged T27777: Failing unit tests for challenge as High priority.
Thu, Sep 24, 3:39 PM · challengeR (v1.0)
eisenman triaged T27776: Failing unit tests for subset of tasks as High priority.
Thu, Sep 24, 3:38 PM · challengeR (v1.0)
eisenman added a comment to T27774: wrong text in report on algorithm subsets.

Ok, weird. For me it says: "The top 5 out of 5 algorithms are considered."

your result is strange given the fact that not the correct things are actually counted

Thu, Sep 24, 1:16 PM · challengeR (v1.0)
eisenman claimed T27774: wrong text in report on algorithm subsets.
  1. I think it will be confusing if it writes "top 5 of 5 algorithms", I would put an if
  2. which changes? as.challenge does not have a full data attribute. The changes in subset have been done by you, see my comments in T27685.
Thu, Sep 24, 12:53 PM · challengeR (v1.0)
eisenman added a comment to T27774: wrong text in report on algorithm subsets.

Ok, weird. For me it says: "The top 5 out of 5 algorithms are considered."

Thu, Sep 24, 12:32 PM · challengeR (v1.0)
eisenman moved T27773: package dependencies from In Progress to Done on the challengeR (v1.0) board.
Thu, Sep 24, 12:23 PM · challengeR (v1.0)
eisenman committed rCHALLENGER64f2452c0f45: Merge branch 'feature/T27773-FixPackageDependencies' into develop (authored by eisenman).
Merge branch 'feature/T27773-FixPackageDependencies' into develop
Thu, Sep 24, 12:23 PM
eisenman committed rCHALLENGERce606a59b22c: Move ggplot2 and purrr to Depends (authored by eisenman).
Move ggplot2 and purrr to Depends
Thu, Sep 24, 12:20 PM
eisenman claimed T27773: package dependencies.
Thu, Sep 24, 12:12 PM · challengeR (v1.0)
eisenman added a comment to T27774: wrong text in report on algorithm subsets.

currently It says "The top 0 out of 0 algorithms are considered." in my latest example

Thu, Sep 24, 12:07 PM · challengeR (v1.0)
eisenman added a comment to T27774: wrong text in report on algorithm subsets.
  1. This is on purpose as I wanted to avoid further nested ifs, but can be discussed.
  2. This was the only variable where I found all algorithm factors. How can they be accessed now? fulldata is not working anymore after you latest changes.
Thu, Sep 24, 11:40 AM · challengeR (v1.0)
eisenman added a comment to T27773: package dependencies.

Are you using the latest develop branch? There everything works for me.

Thu, Sep 24, 11:34 AM · challengeR (v1.0)

Wed, Sep 23

eisenman moved T27769: Top x performing algorithms are not mentioned in the reports anymore from In Progress to Done on the challengeR (v1.0) board.

This issue is fixed in T27677.

Wed, Sep 23, 11:24 PM · challengeR (v1.0)
eisenman moved T27677: Update text in reports (feedback from Lena) from In Progress to Done on the challengeR (v1.0) board.
Wed, Sep 23, 11:24 PM · challengeR (v1.0)
eisenman added a comment to T27677: Update text in reports (feedback from Lena).

In section 3.2.1 "Visualizing bootstrap results", two visualization methods are mentioned, but only one is contained in the report: "To investigate which tasks separate algorithms well (i.e., lead to a stable ranking), two visualization methods
are recommended."

Do we add a second plot or adapt the text?

Wed, Sep 23, 11:24 PM · challengeR (v1.0)
eisenman created T27770: Inconsistency in report description.
Wed, Sep 23, 11:22 PM · challengeR (v1.0)
eisenman committed rCHALLENGER53cad48c02bb: Merge branch 'feature/T27677-RestructureReports' into develop (authored by eisenman).
Merge branch 'feature/T27677-RestructureReports' into develop
Wed, Sep 23, 11:20 PM
eisenman committed rCHALLENGER3487afdc60ae: Rephrase some explanations (authored by eisenman).
Rephrase some explanations
Wed, Sep 23, 11:17 PM
eisenman committed rCHALLENGERce659d3df8a2: Mention top x performing algorithms in the single-task reports (authored by eisenman).
Mention top x performing algorithms in the single-task reports
Wed, Sep 23, 11:17 PM
eisenman claimed T27769: Top x performing algorithms are not mentioned in the reports anymore.
Wed, Sep 23, 6:58 PM · challengeR (v1.0)
eisenman triaged T27769: Top x performing algorithms are not mentioned in the reports anymore as High priority.
Wed, Sep 23, 6:39 PM · challengeR (v1.0)
eisenman committed rCHALLENGER4edfe8f3555b: Complete overview of plots (authored by eisenman).
Complete overview of plots
Wed, Sep 23, 5:18 PM
eisenman created T27766: Update workflow figure.
Wed, Sep 23, 12:16 PM · webChallengeR
eisenman triaged T27765: Make sure that column specified for metric values has numeric values as High priority.
Wed, Sep 23, 12:12 PM · webChallengeR
eisenman triaged T27764: Detect early when insufficient number of columns are loaded as High priority.
Wed, Sep 23, 12:11 PM · webChallengeR
eisenman triaged T27761: Prevent generation of report if field "name" is empty as High priority.
Wed, Sep 23, 11:40 AM · webChallengeR
eisenman created T27758: Remove "next" button from page "Generate report".
Wed, Sep 23, 11:17 AM · webChallengeR
eisenman created T27757: Contradicting instruction on page "Uncertainty analysis".
Wed, Sep 23, 11:14 AM · webChallengeR

Mon, Sep 21

eisenman committed rCHALLENGERe3ef40f37895: Adapt sections (authored by eisenman).
Adapt sections
Mon, Sep 21, 11:48 PM
eisenman committed rCHALLENGER8ba1c489befb: Cite paper (authored by eisenman).
Cite paper
Mon, Sep 21, 11:48 PM
eisenman committed rCHALLENGER71695c6b7ffa: Use challenge name in description (authored by eisenman).
Use challenge name in description
Mon, Sep 21, 11:48 PM
eisenman added a comment to T27677: Update text in reports (feedback from Lena).

In section 3.2.1 "Visualizing bootstrap results", two visualization methods are mentioned, but only one is contained in the report: "To investigate which tasks separate algorithms well (i.e., lead to a stable ranking), two visualization methods
are recommended."

Mon, Sep 21, 7:01 PM · challengeR (v1.0)
eisenman claimed T27677: Update text in reports (feedback from Lena).
Mon, Sep 21, 6:58 PM · challengeR (v1.0)
eisenman added a comment to T27748: Decide on limit for algorithms in plot legends.

The plots can already be limited to the top x performing algorithms. This can be reused in that case. Then the question is how many legend items are we going to "guarantee" appearing nicely?

Mon, Sep 21, 9:32 AM · challengeR (v1.0)

Fri, Sep 18

eisenman committed rCHALLENGERcfb5ffa0fa05: Keep cases generic (authored by eisenman).
Keep cases generic
Fri, Sep 18, 7:01 PM
eisenman moved T27677: Update text in reports (feedback from Lena) from Backlog to In Progress on the challengeR (v1.0) board.
Fri, Sep 18, 6:55 PM · challengeR (v1.0)
eisenman moved T27474: Legend of line plot has missing legend title and algorithm from In Progress to Done on the challengeR (v1.0) board.

The 19 algorithms are shown correctly. The strategy for more algorithms is discussed in T27748.

Fri, Sep 18, 6:52 PM · challengeR (v1.0)
eisenman created T27748: Decide on limit for algorithms in plot legends.
Fri, Sep 18, 6:51 PM · challengeR (v1.0)
eisenman moved T27475: Legend of blob plot has missing legend title and algorithms from In Progress to Done on the challengeR (v1.0) board.

The 19 algorithms are shown correctly.

Fri, Sep 18, 6:39 PM · challengeR (v1.0)
eisenman added a comment to T27294: add multiple figure output formats in report with option clean=TRUE.

It works with the vector. The generated jpeg and png files have a quite low resolution and are as such not really reusable (e.g. in a presentation slide). Can you increase the resolution?

Fri, Sep 18, 5:42 PM · challengeR (v1.0)
eisenman moved T27658: LaTeX Warning: There were undefined references from In Progress to Done on the challengeR (v1.0) board.
Fri, Sep 18, 5:14 PM · challengeR (v1.0)
eisenman committed rCHALLENGERbf889b00d8e6: Merge branch 'feature/T27658-FixUndefinedReference' into develop (authored by eisenman).
Merge branch 'feature/T27658-FixUndefinedReference' into develop
Fri, Sep 18, 5:08 PM
eisenman committed rCHALLENGER91d75517df45: Add individual plot description without need of referencing (authored by eisenman).
Add individual plot description without need of referencing
Fri, Sep 18, 5:06 PM
eisenman added a comment to T27658: LaTeX Warning: There were undefined references.

The warning is raised because the label is defined when bootstrapping is used but also referenced when bootstrapping is not used.

Fri, Sep 18, 4:44 PM · challengeR (v1.0)
eisenman moved T27658: LaTeX Warning: There were undefined references from Backlog to In Progress on the challengeR (v1.0) board.
Fri, Sep 18, 2:29 PM · challengeR (v1.0)

Thu, Sep 17

eisenman added a comment to T27677: Update text in reports (feedback from Lena).

Almost all suggestions from Lena can be realized, apart from listing the metrics in the meta data which we don't know.

Thu, Sep 17, 3:41 PM · challengeR (v1.0)
eisenman placed T27360: compatibility with R4.0.0 up for grabs.
Thu, Sep 17, 2:55 PM · challengeR (v1.0)
eisenman moved T27740: Add missing imports from In Progress to Done on the challengeR (v1.0) board.
Thu, Sep 17, 2:53 PM · challengeR (v1.0)
eisenman moved T27252: Multiple cases per algorithm are not detected when missing data was added before from In Progress to Done on the challengeR (v1.0) board.
Thu, Sep 17, 2:52 PM · challengeR (v1.0)
eisenman committed rCHALLENGER6a9fa7994fc2: Merge branch 'feature/T27740-AddMissingImports' into develop (authored by eisenman).
Merge branch 'feature/T27740-AddMissingImports' into develop
Thu, Sep 17, 2:51 PM
eisenman committed rCHALLENGER5070907484ec: Add missing imports (authored by eisenman).
Add missing imports
Thu, Sep 17, 2:49 PM
eisenman moved T27740: Add missing imports from Backlog to In Progress on the challengeR (v1.0) board.
Thu, Sep 17, 2:48 PM · challengeR (v1.0)
eisenman triaged T27740: Add missing imports as High priority.
Thu, Sep 17, 2:48 PM · challengeR (v1.0)
eisenman added a comment to T27453: Create complete network plot in its function.

Is it not possible to create a plot first on a canvas that contains everything and as a second step add the scaled plot to the report?

Thu, Sep 17, 1:45 PM · challengeR (v1.0)

Wed, Sep 16

eisenman added a comment to T27338: Report contains misleading information on missing values.

I like the proposal!
"0 missing cases have been found in the data set. However, performance of not all algorithms has been observed for all cases. Therefor, missings have been inserted in the following cases:"

Wed, Sep 16, 9:43 AM · challengeR (v1.0)

Thu, Sep 10

eisenman moved T27715: Specify minimum required versions of dependencies from In Progress to Done on the challengeR (v1.0) board.
Thu, Sep 10, 4:50 PM · challengeR (v1.0)
eisenman committed rCHALLENGERc112251b10d2: Merge branch 'feature/T27715-MinimumDependencyVersions' into develop (authored by eisenman).
Merge branch 'feature/T27715-MinimumDependencyVersions' into develop
Thu, Sep 10, 4:50 PM
eisenman committed rCHALLENGERd4b4cf639b80: Specify minimum versions of dependencies (authored by eisenman).
Specify minimum versions of dependencies
Thu, Sep 10, 4:48 PM
eisenman renamed T27715: Specify minimum required versions of dependencies from Specify minimum required versions for dependencies to Specify minimum required versions of dependencies.
Thu, Sep 10, 4:27 PM · challengeR (v1.0)
eisenman added a comment to T27715: Specify minimum required versions of dependencies.

The minimum required versions were identified from the set of packages the developers (@wiesenfa , @reinkea , @aguilera , @eisenman ) are currently using.

Thu, Sep 10, 4:26 PM · challengeR (v1.0)
eisenman moved T27715: Specify minimum required versions of dependencies from Backlog to In Progress on the challengeR (v1.0) board.
Thu, Sep 10, 4:22 PM · challengeR (v1.0)
eisenman created T27715: Specify minimum required versions of dependencies.
Thu, Sep 10, 4:22 PM · challengeR (v1.0)
eisenman added a comment to T27338: Report contains misleading information on missing values.

I checked the report and I think the handling of missings is not clear: First saying no observations are missing, followed by algorithm performances are missing. The value of the replacement should not appear in the table.

Thu, Sep 10, 1:32 PM · challengeR (v1.0)
eisenman created T27714: Report contains weird R output.
Thu, Sep 10, 1:16 PM · challengeR (v1.0)
eisenman moved T27657: When missing algorithm performances are added as NAs, task is also NA from In Progress to Done on the challengeR (v1.0) board.
Thu, Sep 10, 1:04 PM · challengeR (v1.0)
eisenman committed rCHALLENGER44bdae582371: Merge branch 'feature/T27657-CompleteTaskForMissingAlgoPerformances' into… (authored by eisenman).
Merge branch 'feature/T27657-CompleteTaskForMissingAlgoPerformances' into…
Thu, Sep 10, 1:02 PM
eisenman committed rCHALLENGER580e32168af0: Add task for missing algorithm performances (authored by eisenman).
Add task for missing algorithm performances
Thu, Sep 10, 12:58 PM
eisenman added a comment to T27657: When missing algorithm performances are added as NAs, task is also NA.

Task is added as NA for both specified task name and dummy task name.

Thu, Sep 10, 12:16 PM · challengeR (v1.0)
eisenman renamed T27657: When missing algorithm performances are added as NAs, task is also NA from single task challenge treated as multi task in sanity checks gives unuseful warnings to When missing algorithm performances are added as NAs, task is also NA.
Thu, Sep 10, 12:09 PM · challengeR (v1.0)
eisenman moved T27385: Variables holding the aggregated value are named differently for single- and multi-task challenges from In Progress to Done on the challengeR (v1.0) board.
Thu, Sep 10, 11:48 AM · challengeR (v1.0)
eisenman moved T27253: Notify user when multi-task data set is interpreted as single-task data set from In Progress to Done on the challengeR (v1.0) board.
Thu, Sep 10, 11:47 AM · challengeR (v1.0)
eisenman moved T27694: Fix failing unit tests from In Progress to Done on the challengeR (v1.0) board.
Thu, Sep 10, 11:46 AM · challengeR (v1.0)
eisenman committed rCHALLENGERb6daf2265a26: Merge branch 'feature/T27694-FixUnitTests' into develop (authored by eisenman).
Merge branch 'feature/T27694-FixUnitTests' into develop
Thu, Sep 10, 11:45 AM
eisenman added a comment to T27253: Notify user when multi-task data set is interpreted as single-task data set.

Tests will be fixed in T27694.

Thu, Sep 10, 11:43 AM · challengeR (v1.0)
eisenman committed rCHALLENGER09dd8e82d5ac: Fix expected messages (authored by eisenman).
Fix expected messages
Thu, Sep 10, 11:40 AM

Wed, Sep 9

eisenman committed rCHALLENGER833f3c352f41: Fix expectations of value_FUN (authored by eisenman).
Fix expectations of value_FUN
Wed, Sep 9, 10:41 PM
eisenman committed rCHALLENGER71b760c11fe6: Let object returned from boxplot function inherit from ggplot again instead of… (authored by eisenman).
Let object returned from boxplot function inherit from ggplot again instead of…
Wed, Sep 9, 10:06 PM
eisenman committed rCHALLENGERbf8a099952d6: Fix expectations of rank_FUN (authored by eisenman).
Fix expectations of rank_FUN
Wed, Sep 9, 9:13 PM
eisenman updated subscribers of T27694: Fix failing unit tests.
Wed, Sep 9, 9:07 PM · challengeR (v1.0)
eisenman claimed T27694: Fix failing unit tests.
Wed, Sep 9, 9:06 PM · challengeR (v1.0)
eisenman added a comment to T27385: Variables holding the aggregated value are named differently for single- and multi-task challenges.

The tests in test-rankThenAggregate.R are failing because the expectation is "rank_FUN" but should be "rank_mean" or "rank_median" respectively after the changes. This will be fixed in T27694.

Wed, Sep 9, 9:04 PM · challengeR (v1.0)
eisenman moved T27694: Fix failing unit tests from Backlog to In Progress on the challengeR (v1.0) board.
Wed, Sep 9, 9:03 PM · challengeR (v1.0)

Fri, Sep 4

eisenman added a member for CAMI: wiesenfa.
Fri, Sep 4, 12:10 PM

Tue, Sep 1

eisenman triaged T27694: Fix failing unit tests as High priority.
Tue, Sep 1, 3:09 PM · challengeR (v1.0)

Mon, Aug 31

eisenman added a comment to T27474: Legend of line plot has missing legend title and algorithm.

@reinkea Do you have a dataset containing more algorithms?

Mon, Aug 31, 2:11 PM · challengeR (v1.0)

Jun 22 2020

eisenman added a comment to T27420: report text.

Item 3 and 4 are already mentioned in separate tasks: T27474 and T27475

Jun 22 2020, 8:03 PM · challengeR (v1.0)
eisenman added a comment to T27412: Description of variability of achieved rankings across tasks is missing in short report.

The error mentioned in the comment does not occur anymore in the latest develop branch.

Jun 22 2020, 7:40 PM · challengeR (v1.0)
eisenman moved T27396: Represent single-task challenge as multi-task challenge with one task from In Progress to Done on the challengeR (v1.0) board.
Jun 22 2020, 7:38 PM · challengeR (v1.0)
eisenman added a comment to T27396: Represent single-task challenge as multi-task challenge with one task.

Thanks @aguilera for extensive testing of report generation. The residual issues from this task will be handled in T27494.

Jun 22 2020, 7:37 PM · challengeR (v1.0)
eisenman created T27494: Check which files are still relevant.
Jun 22 2020, 7:35 PM · challengeR (v1.0)
eisenman committed rCHALLENGERb5cdf4dadc19: Merge branch 'feature/T27396-RepresentSingleTaskChallengeAsMultiTaskChallenge'… (authored by eisenman).
Merge branch 'feature/T27396-RepresentSingleTaskChallengeAsMultiTaskChallenge'…
Jun 22 2020, 7:22 PM

Jun 9 2020

eisenman committed rCHALLENGER49e1418ef12f: Show explanation of algorithm ordering only for multi-task data set (authored by eisenman).
Show explanation of algorithm ordering only for multi-task data set
Jun 9 2020, 6:14 PM
eisenman committed rCHALLENGERc4e1f3546b4f: Show title of podium plot only for multi-task data sets (authored by eisenman).
Show title of podium plot only for multi-task data sets
Jun 9 2020, 6:14 PM
eisenman committed rCHALLENGERd9becaa36315: Migrate reporting of top performing algorithms (authored by eisenman).
Migrate reporting of top performing algorithms
Jun 9 2020, 5:53 PM
eisenman committed rCHALLENGERe687c165c83d: Show task name in front of ranking list only for multi-task data sets (authored by eisenman).
Show task name in front of ranking list only for multi-task data sets
Jun 9 2020, 4:58 PM