Page MenuHomePhabricator

Multiple cases per algorithm are not detected when missing data was added before
Closed, ResolvedPublic


Steps to reproduce:

  • Create a data set containing multiple cases per algorithm but also missing algorithm performance
  • Create a challenge object

Expected behavior: an error should be thrown


    data.frame(alg_name="A1", value=0.8, case="C1"),
    data.frame(alg_name="A1", value=0.8, case="C1"),
    data.frame(alg_name="A2", value=0.6, case="C2"),
    data.frame(alg_name="A2", value=0.6, case="C2")

  expect_error(as.challenge(data, algorithm="alg_name", case="case", value="value", smallBetter=FALSE),
               "Case(s) (C1, C2) appear(s) more than once for the same algorithm", fixed=TRUE)

Event Timeline

eisenman created this task.

The given example is already available as a unit test.

See test-challenge.R:185: cases cannot appear more than once per algorithm when missing data was added with sanity check enabled for single-task challenge

@eisenman: did you take over solving this task? Or should I still do something here?

This task still needs action. Could you work on this? The mentioned unit test is available in the develop branch.

should work now, tests successful.
However warning is

Performance of not all algorithms is observed for all cases in task 'dummyTask'

despite being single task challenge. see also separate task #T27657

@eisenman could you please check?

@eisenman error message might have changed, please check tests