Page MenuHomePhabricator

Constant reflectances in ipcai training data set
Closed, ResolvedPublic

Description

In the following data set there is an error concerning the simulated reflectances: E130-Projekte/Biophotonics/Data/base_data/simulations/ipcai_train/ipcai_revision_colon_mean_scattering_train_all.txt

Rows 930 to 999 all contain the same reflectance value for each of the wavelength bands (namely 0.418774). This should not be as the physiological parameters vary quite a bit in this region. This is the only region in the file showing this behavior.

Event Timeline

adler triaged this task as Normal priority.Sep 18 2018, 2:01 PM
adler created this task.

Wrote a small script to check simulations, it is on:

susi/scripts/incubator/leo/check_simulations.py

You can give it a csv or txt file or a folder containing csv or txt files and a start and end (columns) in order to look over rows and see if there are some with the same reflectances.

adler added a comment.Sep 18 2018, 4:31 PM

Had a look at the data Lynton used and there is something strange going on in the same region. However, it is the exact oposite of before. If I compute the difference of neighboring reflectances the variance is huge in the 930 to 999 region and way smaller outside -.-

adler added a comment.Sep 18 2018, 4:34 PM

It seems that the physiological parameters in Lynton's dataset are the same as the ones in mine, but the reflectances in the region are not constant. But again, something is fishy. The variance spikes in the same region where before there were constant reflectances.

adler added a comment.Sep 19 2018, 1:31 PM

Resimulation of 70 affected rows is done. The updated file lies here:

E130-Projekte/Biophotonics/Data/base_data/simulations/ipcai_train/corrected_train.csv

The 1-off-difference now looks pretty uniform everywhere.

adler closed this task as Resolved.Jan 7 2019, 4:07 PM
adler claimed this task.