Page MenuHomePhabricator

Multiple cases per algorithm are not detected when missing data was added before
Open, HighPublic

Description

Steps to reproduce:

  • Create a data set containing multiple cases per algorithm but also missing algorithm performance
  • Create a challenge object

Expected behavior: an error should be thrown

Example:

data=rbind(
    data.frame(alg_name="A1", value=0.8, case="C1"),
    data.frame(alg_name="A1", value=0.8, case="C1"),
    data.frame(alg_name="A2", value=0.6, case="C2"),
    data.frame(alg_name="A2", value=0.6, case="C2")
  )

  expect_error(as.challenge(data, algorithm="alg_name", case="case", value="value", smallBetter=FALSE),
               "Case(s) (C1, C2) appear(s) more than once for the same algorithm", fixed=TRUE)
}

Event Timeline

eisenman triaged this task as High priority.Mar 24 2020, 5:59 PM
eisenman created this task.
eisenman updated the task description. (Show Details)Mar 24 2020, 6:02 PM
eisenman added a subscriber: wiesenfa.
eisenman assigned this task to wiesenfa.Mar 30 2020, 4:28 PM
eisenman moved this task from Backlog to In Progress on the challengeR board.

The given example is already available as a unit test.

See test-challenge.R:185: cases cannot appear more than once per algorithm when missing data was added with sanity check enabled for single-task challenge

@eisenman: did you take over solving this task? Or should I still do something here?

This task still needs action. Could you work on this? The mentioned unit test is available in the develop branch.

wiesenfa added a comment.EditedAug 7 2020, 2:47 PM

should work now, tests successful.
However warning is

Performance of not all algorithms is observed for all cases in task 'dummyTask'

despite being single task challenge. see also separate task #T27657

@eisenman could you please check?

wiesenfa reassigned this task from wiesenfa to eisenman.Aug 7 2020, 2:49 PM

@eisenman error message might have changed, please check tests

eisenman moved this task from In Progress to Done on the challengeR (v1.0) board.Sep 17 2020, 2:52 PM