During training, the thread handling saving of figures in the NumpyPlotFileLogger (show_value method, to be precise), raises an IndexError. The error is non-critical and happens in seemingly random intervals. Epochs after the error usually work fine again. Traceback:
Unhandled exception in thread started by <bound method Figure.savefig of <matplotlib.figure.Figure object at 0x7fe92a3b2e48>>
Traceback (most recent call last):
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/figure.py", line 1573, in savefig
self.canvas.print_figure(*args, **kwargs)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 2252, in print_figure
**kwargs)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 545, in print_png
FigureCanvasAgg.draw(self)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 464, in draw
self.figure.draw(self.renderer)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/artist.py", line 63, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/figure.py", line 1144, in draw
renderer, self, dsu, self.suppressComposite)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/image.py", line 139, in _draw_list_compositing_images
a.draw(renderer)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/artist.py", line 63, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 2426, in draw
mimage._draw_list_compositing_images(renderer, self, dsu)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/image.py", line 139, in _draw_list_compositing_images
a.draw(renderer)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/artist.py", line 63, in draw_wrapper
draw(artist, renderer, *args, **kwargs)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axis.py", line 1136, in draw
ticks_to_draw = self._update_ticks(renderer)
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axis.py", line 969, in _update_ticks
tick_tups = [t for t in self.iter_ticks()]
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axis.py", line 969, in <listcomp>
tick_tups = [t for t in self.iter_ticks()]
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axis.py", line 913, in iter_ticks
majorTicks = self.get_major_ticks(len(majorLocs))
File "/home/jens/anaconda3/lib/python3.6/site-packages/matplotlib/axis.py", line 1330, in get_major_ticks
tick = self.majorTicks[i]
IndexError: list index out of rangeFirst step will be to find out which figure causes the error.