Since this topic came up recently during a user workshop:
I would be benefical to have a simple plugin or alternatively an add-on on the existing segmentation plugin to:

• Load two segmentations (potential requirement: They must have the same geometry / orginate from the same image)
• Calculate and display all commonly used comparison metrics like Dice, Jaccard, Surface distance etc.
• Maybe also provide a Command line tool to enable batch processing

This was already implemented by many former PhD students but never really integrated.
I think recently @steint implemented something really nice for the browser that could be possibly recylced.