Page MenuHomePhabricator

Es gibt noch Funktionen in rttb, die nicht getestet sind
Closed, ResolvedPublic

Assigned To
Authored By
Nov 30 2016, 4:04 PM
  • Restricted Project
Referenced Files
F757469: AnalyzeCoverageUsage.PNG
Apr 27 2017, 4:03 PM
F757505: noColor.PNG
Apr 27 2017, 4:03 PM
F757526: allCoverage.PNG
Apr 27 2017, 4:03 PM
F757479: CoreTest.PNG
Apr 27 2017, 4:03 PM
F757497: TestsRun.PNG
Apr 27 2017, 4:03 PM
F757511: green.PNG
Apr 27 2017, 4:03 PM
F757500: htmlOverview.PNG
Apr 27 2017, 4:03 PM
F757515: colorful.PNG
Apr 27 2017, 4:03 PM


Es gibt noch Funktionen, die nicht getestet sind:

Die Stellen, an denen wir abs() benutzt haben, waren offenbar nicht durch einen Test abgedeckt bzw. haben sich nicht ausgewirkt:
• bool equalsAlmost(const OrientationMatrix& anOrientationMatrix, double errorConstant) const
• bool ScatterTester::doCheck(void)
• bool mapCompare(const std::map<double, double>& lhs, const std::map<double, double>& rhs)

Die erste Stelle ist in rttbBaseType.h, die anderen beiden sind in Tests.
Bei mapCompare schlägt der Test bei values zwischen 0 und 1 nie fehl: if (abs(a - b ) > errorConstant)

Event Timeline

zhangl created this object with edit policy "SIDT (Project)".

For this task it should be considered to look into Test Coverage / Code coverage and what Software can help us identify the parts of code that aren't tested yet.

Basic coverage criteria

With code coverage software there are several different criteria the tool can look out for. The main ones are (from "weakest" to the most strict one):

  • Function coverage: Has each function in the program been called by at least one test?
  • Statement coverage: Has each statement in the program been executed?
  • Branch coverage: Has each branch (i.e. if, else, case, etc.) been excecuted?
  • Condition coverage: Has each Boolean sub-expression evaluated both true and false at least once?

This is only a brief introduction to the different criterea. Anyone that is interested in a visual example can have a look here:

Common tools regarding Code Coverage

Wikipedia states several tools for a broad spectrum of programming languages. Right now the interest lies on C++ tools so the list reduces to:

Name of the toolProgramming languageCost / License
Bullseye CoverageC++800$ for a new license with 1 year subscription. After that 200 $ per year
Cantata++C, C++Commercial (No pricing information found)
froglogic's Squish CocoC, C++, C#, Tcl, QML, JavaScriptStarting at 3000€ depending on amount of users
gcovC, C++, Adafree, comes with gcc
LDRA TestbedC++Commercial (No pricing information found)
TessyC, C++Commercial (No pricing information found)
Testwell ctc++C, C++, Java, C#Commercial (No pricing information found)

This pretty much reduces the list of applicable software down to:

  • gcov
  • OpenCppCoverage

In the following each tool will be briefly introduced in order to select the best fitting one.

hentsch moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Apr 11 2017, 1:43 PM

1. OpenCppCoverage

I looked further into OpenCppCoverage due to the fact that it is also available as a visual studio extension and thus could be integrated in the daily workflow easily.

Basic information
Where it is foundDocumentationUp-to-dateness documentation provides little information about usageMost recent update on 23.02.2017 and support for vs 2017 planned this for year
  • It is available as a visual studio extension here.
  • It says to feature Jenkins support. This was not looked into though.
  • It provides GUI (Visual Studio extension) and CLI functionality
  • It offers html export
First usability test
  1. The Visual Studio extension was downloaded and installed
  2. The project containing the tests to run has to be selected as startup project
  3. Press Extras/Run OpenCppCoverage and you'll be presented with a GUI to set several settings
    Settings.PNG (402×593 px, 15 KB)
  4. You should most certainly be able to just hit "Run Coverage". OpenCppCoverage will do a lot on its own (like selecting the right directories etc.)
  5. You might be prompted to choose a Test. When you select one and everything works you will be presented the coverage information for the one particular test in Visual Studio.
    CoverageInfo.PNG (1×1 px, 81 KB)

If there are files included that you didn't want to include you can uncheck the specific module in the settings pane and retry.
Note that the coverage might be a little worse than it is in reality due to the fact that without countermeasures all "else" and "{" expressions get evaluated as not covered and thus reduce the percentage. More on that later.

First Results

The tool was fairly easy to set up but it doesn't really produce much valuable information.
If you for example have a project containing multiple test you are most ceartainly interested in whether those tests (summed up) cover all (or most) functionality of your productive project/module. With the GUI though, it is only possible to run one of those tests at a time so unless you can remember all previous coverage information and add it up yourself in your head you might ask if there is an workaround.
The short answer is yes, the realistic answer is not a good one.
If you go to the "Import / Export" tab of the GUI you are able to export the coverage data as binary files.
These in turn can be imported later and merged together.
So the workflow would be:

  1. Hit "Run OpenCppCoverage"
  2. Go to the "Import / Export" tab and set the export type to binary. Also you have to specify a path where the file should be saved and name the file.
  3. Hit "Run Coverage" and select the according test
  4. In "Import / Export" change the export filename to the name of the second test (of course you can also just use numbers then you don't have as much typing to do)
  5. Hit "Run Coverage" and select the second test
  6. REPEAT for every test of the module except the last one
  7. For the last test you have to specify each previously generated coverage file as input. (You can remove the export setting or set it to html to get an html coverage report)
  8. The test will be run and the coverage data will be merged. After this is done you should see the combined coverage report in Visual Studio

This is very tedious and takes a lot of time.

A way to improve the workflow efficiency by a little bit

For this "solution" you'll need the standalone executable of OpenCppCoverage. It is run through a command line interface which means that scripts can be written to automate some of the process.

Here I took the approach to write a simple batch script:

"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\0.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "DVHCalculatorTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\1.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "DVHSetTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\2.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "DVHTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\3.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "GenericDoseIteratorTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\4.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "GenericMaskedDoseIteratorTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\5.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "GeometricInfoTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\6.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "MaskVoxelTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\7.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "StrVectorStructureSetGeneratorTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=html:D:\OpenCppCoverage\coverageFiles\html 
                         --input_coverage=0.cov --input_coverage=1.cov --input_coverage=2.cov --input_coverage=3.cov 
                         --input_coverage=4.cov --input_coverage=5.cov --input_coverage=6.cov --input_coverage=7.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "StructureTest"

This script does nothing more than what I described above in the GUI:

  • Run every single test in the module and output a binary coverage report
  • With the last test all files are merged and a html report is created.

To simplify and shorten this script redundant information was moved to a CppConfig file (functionality of OpenCppCoverage)
This file looks like this and can be added to by will:


Theoretically the scripting could be further improved to a point where creating a coverage report only requires setting the directory that contains the test and further information is gathered by the script.
This was not done here. One the one hand because there was not enough time but more importantly because some of the RTTB test will just not work with OpenCppCoverage and there is no workaround I know of yet. An example is rttbDicomIOTests. When they are added to CTest several different parameters are specified without which the test can't be run. I found no way to tell OpenCppCoverage to use those parameters.


It is actually quite obvious how to add the parameters.
In the settings pane just add the test name and all the parameters in the arguments field. This runs like a charm for rttbDicomIOTests. The only thing that has to been done is to extract the parameters from the cmake files (either manually or automated) and add them to the batch file. With this in mind OpenCppCoverage could very well be imagined to become part of the toolchain at least if it gets more comfortable to use.
As a side note. The discussion page at is a great place for additional information. It also shows that the developer really cares about feedback and is quite willing to answer questions.

General Results

A coverage report for rttbCore was created and can be viewed as a html report aswell as in Visual Studio directly.

If I have missed something, you have questions or part of my description is difficult to understand please feel free to contact me and I will update this text.

Originally I was planning on reviewing all of the three coverage tools and present them here. This consumes a substantial amount of time though and I could further improve the efficiency of analyzing code coverage with OpenCppCoverage. These are the reasons why the other two tools won't be discussed here as of now.
Instead I'll try to give you a detailed description of what I think is the easiest way to analyze the code coverage with OpenCppCoverage.

Analyzation toolchain

I wrote a little executable to do the hard work (i.e. searching the Test names and additional parameters in the CMakeLists files of the test folder).

AnalyzeCoverageUsage.PNG (99×787 px, 6 KB)

So as we see we need "only" five arguments for the tool to work.

OpenCppExePathThe absolute path to OpenCppCoverage.exe
PathToSourceFilesThe absolute path to the directory containing the sources (Similar to: D:\dev\RTTB\trunk\code\core)
PathToTestFilesSame as above but with the testing dir (Like D:\dev\RTTB\trunk\testing\core)
PathToTestExe(Like D:\dev\RTTB_build\trunk\bin\Debug\rttbCoreTests.exe) Important: Use the Debug folder. OpenCppCoverage needs the .pdb files
TestDataRootDirThe root data directory containing the additional files for testing (In my case: D:\dev\RTTB\trunk\testing\data)

Done. Now if we run the exe we get:

CoreTest.PNG (441×1 px, 21 KB)

We can see two things happening:

  1. In the cmd prompt we see all the tests that were found (here we would also see additional parameters but the core tests have none)
  2. A folder was created for us called like the executable

In there you see should two files: AnalyzeCoverage.bat and config.txt.
Run the batch file and you'll see the magic happening:

TestsRun.PNG (544×1 px, 45 KB)

The htmlReport_TEST_NAME folder is of interest to us. It contains a visual presentation of the coverage situation:
htmlOverview.PNG (349×1 px, 29 KB)

This is great but when we head into a certain file we see no color coded coverage information:
noColor.PNG (643×838 px, 32 KB)

The information is there though it is just not visible. To see it we have modify the html code like this:
green.PNG (447×943 px, 48 KB)
and this:
red.PNG (454×807 px, 38 KB)
After we are presented with this lovely view:
colorful.PNG (843×848 px, 39 KB)

And thats it there is your coverage report.

Additional Thoughts

As we see manipulating the html report is actually the most time consuming task. It should be easy though to automate the text replacement. If someone actually does this I would be happy if you let me know so we can hand the script out to everyone.

When I created the coverage reports I didn't write them directly in the console but rather into a batch file:

allCoverage.PNG (642×1 px, 98 KB)

This allows me now to easily get all coverage reports about all the tests with one mouseclick and only a few seconds.

This wraps it up for now.
As always I'm happy about any kind of constructive feedback.

The files I used and the coverage reports created can be found on the network drive E071-Daten

jungsi moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.May 2 2017, 9:50 AM
jungsi added a subscriber: jungsi.
hentsch claimed this task.