Page MenuHomePhabricator

Notify user when multi-task data set is interpreted as single-task data set
Open, NormalPublic

Description

When a challenge object is instantiated with a multi-task data set and the parameter "by" is not specified, the data set is interpreted as a single-task data set. This can lead to subsequent errors.

Depending on the composition of the data set, the message "Performance of not all algorithms is observed for all cases in task x" or the error "Case(s) (y) appear(s) more than once for the same algorithm in task x" is shown. This makes sense, but is not intuitive at first glance.

The user can be warned of that scenario (including a suggestion on how to resolve the situation: specify the task column via parameter "by"). Further computation could also be stopped.

Related Objects

Event Timeline

eisenman created this task.Mar 25 2020, 1:33 PM

At the meeting on 30/03/2020 we agreed that the messages should be extended to warn the user of that scenario. Since the column containing the tasks is unknown when the parameter "by" is not specified, an automatic detection can only be realized with a lot of effort.

wiesenfa claimed this task.Apr 2 2020, 11:32 AM

if at least one quarter of cases have a duplicate, the error message is extended by "Or are you considering a multi-task challenge and forgot to specify argument 'by'? "
Otherwise the error message is now further improved, e.g. if it is a single task, the error message doesn't mention "in task dummyTask any more".
Example error message:

  The following cases appear more than once. Or are you considering a multi-task challenge and forgot to specify argument 'by'? 
Please revise. Cases : 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 32, 40, 41, 42, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 72, 73, 74, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 128, 129, 130, 131, 132, 135, 136, 137, 138, 139, 140, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 21

@eisenman could affect tests. please close when finished

Tests will be fixed in T27694.

eisenman moved this task from In Progress to Done on the challengeR (v1.0) board.Sep 10 2020, 11:47 AM