Masking an image with a PlanarFigure polyline results in an error when one or more of the polyline vertices lie outside of the image bounds.
The polyline should be clipped by the image bounds before generating the mask. Probably this can be easily done with a VTK filter.