Page MenuHomePhabricator

challengeRProject
ActivePublic

Details

Description

Methods and open-source toolkit for analyzing and visualizing challenge results

Recent Activity

Fri, Jun 17

aekavur closed T29214: challangeR Patch v1.0.4 as Resolved.
Fri, Jun 17, 1:23 PM · challengeR
aekavur created T29214: challangeR Patch v1.0.4.
Fri, Jun 17, 9:12 AM · challengeR
aekavur closed T28746: [dash] NaN values cause error as Resolved.
Fri, Jun 17, 8:52 AM · challengeR, webChallengeR
aekavur added a comment to T28746: [dash] NaN values cause error.

Warning messages when there are missing values in the data were reviewed as below:

Fri, Jun 17, 8:52 AM · challengeR, webChallengeR
aekavur closed T29167: [dash] Bars and dots don't match in podium plot as Resolved.
Fri, Jun 17, 8:46 AM · challengeR, webChallengeR
aekavur closed T28966: y-axis of blob plots always scaled to 5 as Resolved.
Fri, Jun 17, 8:45 AM · challengeR

Tue, Jun 14

aekavur added a comment to T28966: y-axis of blob plots always scaled to 5.

Hey everyone,

Tue, Jun 14, 11:56 AM · challengeR

Thu, Jun 9

eisenman added a comment to T28966: y-axis of blob plots always scaled to 5.

I like the results when the scales library is used! However, when we find a way to bring back the confidence intervals, also @wiesenfa's latest solution can be used.

Thu, Jun 9, 1:31 PM · challengeR

Tue, Jun 7

aekavur added a comment to T28966: y-axis of blob plots always scaled to 5.

Hey everyone,

Tue, Jun 7, 9:06 AM · challengeR
aekavur added a comment to T29167: [dash] Bars and dots don't match in podium plot.

I added tests in current future branch for checking class of “algorithm” column in challenge object.

Tue, Jun 7, 9:01 AM · challengeR, webChallengeR

Jun 3 2022

eisenman added a comment to T29167: [dash] Bars and dots don't match in podium plot.

First, I tried the fix with R 3.6 and can confirm that it does not break the functionality there.

Jun 3 2022, 1:00 PM · challengeR, webChallengeR

May 30 2022

aekavur added a comment to T29167: [dash] Bars and dots don't match in podium plot.

I have added object[[algorithm]] <- as.factor(object[[algorithm]]) to challengeR.R as you suggested. Now everything works without any problem. No need of stating stringsAsFactors anymore during CSV read.

May 30 2022, 10:55 AM · challengeR, webChallengeR

May 23 2022

wiesenfa added a comment to T29167: [dash] Bars and dots don't match in podium plot.

Thank you so much @aekavur ! It helps a lot to understand the reason finally!

May 23 2022, 7:34 AM · challengeR, webChallengeR

May 22 2022

aekavur added a comment to T29167: [dash] Bars and dots don't match in podium plot.

Hi again :)

May 22 2022, 7:51 PM · challengeR, webChallengeR

May 16 2022

wiesenfa added a comment to T29167: [dash] Bars and dots don't match in podium plot.

if the output is NULL, object[[by]] is not a factor, i.e. class(object[[by]]) is "character", in this case you need to use use unique() and probably your solution

May 16 2022, 10:21 AM · challengeR, webChallengeR
aekavur added a comment to T29167: [dash] Bars and dots don't match in podium plot.

Hi again,

May 16 2022, 9:05 AM · challengeR, webChallengeR

May 13 2022

wiesenfa added a comment to T29167: [dash] Bars and dots don't match in podium plot.

Thanks Emre!
Thats a weird change. I didn't find any mention in R changelog.
probably instead of

algorithms=factor(unique(object[[by]]))

it will be preferred

May 13 2022, 2:45 PM · challengeR, webChallengeR
scholzpa added a comment to T29167: [dash] Bars and dots don't match in podium plot.

Congrats for tracing this down!

May 13 2022, 1:46 PM · challengeR, webChallengeR
aekavur added a comment to T29167: [dash] Bars and dots don't match in podium plot.

Finally, I could find the source of the bug. 😊 It is caused by changed output type of unique() function in R:Base from R-3 to R-4.

May 13 2022, 1:40 PM · challengeR, webChallengeR
aekavur added a project to T29167: [dash] Bars and dots don't match in podium plot: challengeR.
May 13 2022, 1:14 PM · challengeR, webChallengeR

Feb 28 2022

aekavur added a comment to T28966: y-axis of blob plots always scaled to 5.

I am sharing my current test code with artificial data. Since there can be 4-5 blob plots in the report (depending on data, task number), I need to prepare a new test code for only blob plots. Until that, you may use the code I am sharing.

Feb 28 2022, 1:08 PM · challengeR
wiesenfa added a comment to T28966: y-axis of blob plots always scaled to 5.

thanks Emre. that's problematic, confidence intervals are missing. Could you share a code file for testing with artificial data (ideally not with the report as output but the plot itself)? Then I will try to look into it. or is this difficult for you?

Feb 28 2022, 11:16 AM · challengeR
aekavur added a comment to T28966: y-axis of blob plots always scaled to 5.

I have tried this approach. I just needed to remove minor_breaks=NULL, line since there is no such a config in R/scale-discrete-.r

Feb 28 2022, 10:42 AM · challengeR

Feb 24 2022

wiesenfa added a comment to T28966: y-axis of blob plots always scaled to 5.

I think the solution is to consider rank not as continuous but a factor (essentially a string)
That means first following

Feb 24 2022, 5:26 PM · challengeR

Feb 21 2022

wiesenfa added a comment to T28966: y-axis of blob plots always scaled to 5.

THanks Emre! This sounds like a lot of effort. Please give me some time to have a look at it

Feb 21 2022, 2:32 PM · challengeR
aekavur added a comment to T28966: y-axis of blob plots always scaled to 5.

I have tried many configurations just to force ggplot2 to start y-axis labels from "1" when choosing automatic scaling. However, it was not possible :/

Feb 21 2022, 10:20 AM · challengeR

Feb 14 2022

wiesenfa added a comment to T28966: y-axis of blob plots always scaled to 5.

I guess overall it's a matter of taste.
Fully automatic one has several problems: in case of the 30 algorithms, scale starts with 0 which is not sensible. I'm not sure what happens with something like 27 or 17 algorithms (a number which doesn't divide by 5). in case of the 7 alogirhtms it starts with 2 which I find a bit weird, I would expect a scale starting with 1. Thus, I would at least include the limits=c(1,max(...)) argument which however as said before may lead to sequences like 1,7,13,... but maybe this is not so much of a problem.

Feb 14 2022, 12:03 PM · challengeR
aekavur added a comment to T28966: y-axis of blob plots always scaled to 5.

Let's try the automatic config of ggplot :)

Feb 14 2022, 11:53 AM · challengeR
wiesenfa added a comment to T28966: y-axis of blob plots always scaled to 5.

If I remember correctly this didn't work layout-wise for large number of algorithms. Numbers will either overlap or need to get very small/size of figure will need to be increased.
try to test with something like 20 algorithms, how does the report look then?
what's the problem with 1,5,10,15,18? the scale isn't affected, so for me it wouldn't matter that it's not the same intervals. in principle you could also omit the 18, i.e. only 1,5,10,15. Instead of all integers, I would rather use the automatic choice.

Feb 14 2022, 11:07 AM · challengeR
aekavur added a comment to T28966: y-axis of blob plots always scaled to 5.

I agree with you. On the other hand, putting breaks according to a defined integer can be tricky. For example, let's assume that we have decided to define breaks on every 5th element. The y-axis will be a 1,5,10,15,18 for a challenge with 18 algorithms. The last portion of the sequence will have a different period. Therefore, I offer including all integer breaks for the [1, #algorithms] range. I am putting some examples here:

Feb 14 2022, 11:00 AM · challengeR

Feb 11 2022

wiesenfa added a comment to T28966: y-axis of blob plots always scaled to 5.

not sure whether this is a good idea. imagine a challenge with 18 algorithms. there will be only a 1 and an 18 and nothing in between, this may make it difficult to read. what do you think?

Feb 11 2022, 8:24 PM · challengeR
aekavur added a comment to T28966: y-axis of blob plots always scaled to 5.

I have tried suggested codes but they did not fix the problem. Besides, there caused additional issues. :)

Feb 11 2022, 2:02 PM · challengeR
wiesenfa added a comment to T28966: y-axis of blob plots always scaled to 5.

Could you try to replace "breaks" by "labels" in

Feb 11 2022, 11:52 AM · challengeR
eisenman added a comment to T28966: y-axis of blob plots always scaled to 5.

I have tested this with the provided data. The scaling of the y-axis seems to be correct now. But only the first rank is labeled on the y-axis. Can the other ranks be labeled as well?

Feb 11 2022, 11:26 AM · challengeR

Feb 8 2022

aekavur added a comment to T28746: [dash] NaN values cause error.

The problem is almost fixed by giving na.treat parameter in both as.challenge and ranking methods (except rankThenAggregate). Now we can generate reports for all ranking methods.

Feb 8 2022, 2:15 PM · challengeR, webChallengeR

Feb 7 2022

aekavur updated subscribers of T28966: y-axis of blob plots always scaled to 5.
Feb 7 2022, 3:00 PM · challengeR
aekavur added a comment to T28966: y-axis of blob plots always scaled to 5.

scale_y_continuous functions inside ./R/Stability.R file were modified. The problem seems solved. You can test it feature/T28966-YaxisOfBlobPlotsAlwaysScaledTo5 branch via the file at the attachment. (You can run it root folder of the challengeR code)

Feb 7 2022, 2:57 PM · challengeR
aekavur updated the task description for T28966: y-axis of blob plots always scaled to 5.
Feb 7 2022, 1:23 PM · challengeR
aekavur triaged T28966: y-axis of blob plots always scaled to 5 as Normal priority.
Feb 7 2022, 1:22 PM · challengeR
wiesenfa added a comment to T28746: [dash] NaN values cause error.

I guess na.treat it is only needed for the line plot for comparing to other ranking methods?
In this case, a message could be thrown when compiling the report saying something like "line plot comparing ranking methods omitted since na.treat is not specified. Specify na.treat in as.challenge() if inclusion of line plot is desired" and allow compilation of the report (excluding line plot).
(Note that you can define na.treat both in as.challenge() as well as in the ranking functions).

Feb 7 2022, 12:02 PM · challengeR, webChallengeR

Jan 28 2022

eisenman updated subscribers of T28746: [dash] NaN values cause error.

Thank you for investigating this! In challengeR it is covered in the way that a message is emitted saying "na.treat obligatory if report is intended to be compiled". In order to solve the mentioned issue 2, a strategy for the preferred way to handle it in VISSART should be defined. Should the user be guided to specify the NaN handling strategy? Should the user be able to generate a report but without the plots that require numeric values?

Jan 28 2022, 12:25 PM · challengeR, webChallengeR
aekavur added a comment to T28746: [dash] NaN values cause error.

Current status of the issue:

Jan 28 2022, 11:33 AM · challengeR, webChallengeR

Nov 29 2021

reinkea triaged T28879: Feature request: Plot hierarchical data structure as Wishlist priority.
Nov 29 2021, 1:08 PM · challengeR

Nov 16 2021

aekavur moved T28746: [dash] NaN values cause error from Backlog to In Progress on the challengeR board.
Nov 16 2021, 2:11 PM · challengeR, webChallengeR
aekavur added a project to T28746: [dash] NaN values cause error: challengeR.
Nov 16 2021, 2:11 PM · challengeR, webChallengeR

Oct 28 2021

eisenman moved T28484: [external] Improve documentation on significance ranking from Backlog to In Progress on the challengeR board.
Oct 28 2021, 4:03 PM · challengeR
eisenman updated subscribers of T28484: [external] Improve documentation on significance ranking.
Oct 28 2021, 4:03 PM · challengeR

Oct 15 2021

eisenman closed T28742: challangeR Patch v1.0.3 as Resolved.
Oct 15 2021, 2:12 PM · challengeR
eisenman closed T28710: [Bug] Ggplot warning in various places of the report as Resolved.
Oct 15 2021, 2:12 PM · challengeR
eisenman moved T28742: challangeR Patch v1.0.3 from In Progress to Done on the challengeR board.
Oct 15 2021, 2:12 PM · challengeR