During training, the thread handling saving of figures in the NumpyPlotFileLogger (show_value method, to be precise), raises an IndexError. The error is non-critical and happens in seemingly random intervals. Epochs after the error usually work fine again. Traceback:
Unhandled exception in thread started by <bound method Figure.savefig of <matplotlib.figure.Figure object at 0x7fe92a3b2e48>> Traceback (most recent call last): File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/figure.py", line 1573, in savefig self.canvas.print_figure(*args, **kwargs) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 2252, in print_figure **kwargs) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 545, in print_png FigureCanvasAgg.draw(self) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 464, in draw self.figure.draw(self.renderer) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/artist.py", line 63, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/figure.py", line 1144, in draw renderer, self, dsu, self.suppressComposite) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/image.py", line 139, in _draw_list_compositing_images a.draw(renderer) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/artist.py", line 63, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 2426, in draw mimage._draw_list_compositing_images(renderer, self, dsu) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/image.py", line 139, in _draw_list_compositing_images a.draw(renderer) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/artist.py", line 63, in draw_wrapper draw(artist, renderer, *args, **kwargs) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axis.py", line 1136, in draw ticks_to_draw = self._update_ticks(renderer) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axis.py", line 969, in _update_ticks tick_tups = [t for t in self.iter_ticks()] File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axis.py", line 969, in <listcomp> tick_tups = [t for t in self.iter_ticks()] File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axis.py", line 913, in iter_ticks majorTicks = self.get_major_ticks(len(majorLocs)) File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axis.py", line 1330, in get_major_ticks tick = self.majorTicks[i] IndexError: list index out of range
First step will be to find out which figure causes the error.