The function as.challenge has an argument "annotator". This is not used for the report and other use cases are not documented. Is it still working? Should the argument be dropped?
should be working but hasn't been really tested lately. only applies to rank then aggregate. originates from nature communications paper and should be explained there.
For testing I could again compare to Annette's code used at that time.
Wouldn't drop it but you are right that documentation is missing
added to documentation of as.challenge():
(arg annotator:) If multiple annotators annotated the test cases, a string specifying the name of the column that contains the annotator identifiers. Only applies to rang-then-aggregate. Use with caution: Currently not tested.
I would definitely not drop this feature, because this may be important to take into account. But I won't be able to properly validate this before Christmas. In principle, I can compare results to Annette's implementation for validation