For this task it should be considered to look into Test Coverage / Code coverage and what Software can help us identify the parts of code that aren't tested yet.

Basic coverage criteria

With code coverage software there are several different criteria the tool can look out for. The main ones are (from "weakest" to the most strict one):

• Function coverage: Has each function in the program been called by at least one test?
• Statement coverage: Has each statement in the program been executed?
• Branch coverage: Has each branch (i.e. if, else, case, etc.) been excecuted?
• Condition coverage: Has each Boolean sub-expression evaluated both true and false at least once?

This is only a brief introduction to the different criterea. Anyone that is interested in a visual example can have a look here:
https://www.johner-institut.de/blog/iec-62304-medizinische-software/code-coverage/

Common tools regarding Code Coverage

 
Wikipedia states several tools for a broad spectrum of programming languages. Right now the interest lies on C++ tools so the list reduces to:

This pretty much reduces the list of applicable software down to:

• gcov
• OpenCppCoverage
• COVTOOL

In the following each tool will be briefly introduced in order to select the best fitting one.

1. OpenCppCoverage

I looked further into OpenCppCoverage due to the fact that it is also available as a visual studio extension and thus could be integrated in the daily workflow easily.

Basic information

Extras

• It is available as a visual studio extension here.
• It says to feature Jenkins support. This was not looked into though.
• It provides GUI (Visual Studio extension) and CLI functionality
• It offers html export

First usability test

1. The Visual Studio extension was downloaded and installed
2. The project containing the tests to run has to be selected as startup project
3. Press Extras/Run OpenCppCoverage and you'll be presented with a GUI to set several settings
   
4. You should most certainly be able to just hit "Run Coverage". OpenCppCoverage will do a lot on its own (like selecting the right directories etc.)
5. You might be prompted to choose a Test. When you select one and everything works you will be presented the coverage information for the one particular test in Visual Studio.
   

If there are files included that you didn't want to include you can uncheck the specific module in the settings pane and retry.
Note that the coverage might be a little worse than it is in reality due to the fact that without countermeasures all "else" and "{" expressions get evaluated as not covered and thus reduce the percentage. More on that later.

First Results

The tool was fairly easy to set up but it doesn't really produce much valuable information.
If you for example have a project containing multiple test you are most ceartainly interested in whether those tests (summed up) cover all (or most) functionality of your productive project/module. With the GUI though, it is only possible to run one of those tests at a time so unless you can remember all previous coverage information and add it up yourself in your head you might ask if there is an workaround.
The short answer is yes, the realistic answer is not a good one.
If you go to the "Import / Export" tab of the GUI you are able to export the coverage data as binary files.
These in turn can be imported later and merged together.
So the workflow would be:

1. Hit "Run OpenCppCoverage"
2. Go to the "Import / Export" tab and set the export type to binary. Also you have to specify a path where the file should be saved and name the file.
3. Hit "Run Coverage" and select the according test
4. In "Import / Export" change the export filename to the name of the second test (of course you can also just use numbers then you don't have as much typing to do)
5. Hit "Run Coverage" and select the second test
6. REPEAT for every test of the module except the last one
7. For the last test you have to specify each previously generated coverage file as input. (You can remove the export setting or set it to html to get an html coverage report)
8. The test will be run and the coverage data will be merged. After this is done you should see the combined coverage report in Visual Studio

This is very tedious and takes a lot of time.

A way to improve the workflow efficiency by a little bit

For this "solution" you'll need the standalone executable of OpenCppCoverage. It is run through a command line interface which means that scripts can be written to automate some of the process.

Here I took the approach to write a simple batch script:

"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\0.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "DVHCalculatorTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\1.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "DVHSetTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\2.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "DVHTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\3.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "GenericDoseIteratorTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\4.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "GenericMaskedDoseIteratorTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\5.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "GeometricInfoTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\6.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "MaskVoxelTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=binary:D:\OpenCppCoverage\coverageFiles\7.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "StrVectorStructureSetGeneratorTest"
"../OpenCppCoverage.exe" --config_file=openCppConfig.txt --export_type=html:D:\OpenCppCoverage\coverageFiles\html 
                         --input_coverage=0.cov --input_coverage=1.cov --input_coverage=2.cov --input_coverage=3.cov 
                         --input_coverage=4.cov --input_coverage=5.cov --input_coverage=6.cov --input_coverage=7.cov 
                         "D:\dev\RTTB_build\T21862\bin\Debug\rttbCoreTests.exe" "StructureTest"

This script does nothing more than what I described above in the GUI:

• Run every single test in the module and output a binary coverage report
• With the last test all files are merged and a html report is created.

To simplify and shorten this script redundant information was moved to a CppConfig file (functionality of OpenCppCoverage)
This file looks like this and can be added to by will:

excluded_line_regex="else.*" 
excluded_line_regex="\{.*"
sources=D:\dev\RTTB\T21862\code\core
sources=D:\dev\RTTB\T21862\testing\core
working_dir=D:\dev\RTTB_build\T21862\Testing\core\\

Theoretically the scripting could be further improved to a point where creating a coverage report only requires setting the directory that contains the test and further information is gathered by the script.
This was not done here. One the one hand because there was not enough time but more importantly because some of the RTTB test will just not work with OpenCppCoverage and there is no workaround I know of yet. An example is rttbDicomIOTests. When they are added to CTest several different parameters are specified without which the test can't be run. I found no way to tell OpenCppCoverage to use those parameters.

Update

It is actually quite obvious how to add the parameters.
In the settings pane just add the test name and all the parameters in the arguments field. This runs like a charm for rttbDicomIOTests. The only thing that has to been done is to extract the parameters from the cmake files (either manually or automated) and add them to the batch file. With this in mind OpenCppCoverage could very well be imagined to become part of the toolchain at least if it gets more comfortable to use.
As a side note. The discussion page at https://opencppcoverage.codeplex.com/discussions is a great place for additional information. It also shows that the developer really cares about feedback and is quite willing to answer questions.

General Results

A coverage report for rttbCore was created and can be viewed as a html report aswell as in Visual Studio directly.

If I have missed something, you have questions or part of my description is difficult to understand please feel free to contact me and I will update this text.

Originally I was planning on reviewing all of the three coverage tools and present them here. This consumes a substantial amount of time though and I could further improve the efficiency of analyzing code coverage with OpenCppCoverage. These are the reasons why the other two tools won't be discussed here as of now.
Instead I'll try to give you a detailed description of what I think is the easiest way to analyze the code coverage with OpenCppCoverage.

Analyzation toolchain

I wrote a little executable to do the hard work (i.e. searching the Test names and additional parameters in the CMakeLists files of the test folder).
Usage:

Done. Now if we run the exe we get:

1. In the cmd prompt we see all the tests that were found (here we would also see additional parameters but the core tests have none)
2. A folder was created for us called like the executable

In there you see should two files: AnalyzeCoverage.bat and config.txt.
Run the batch file and you'll see the magic happening:

Additional Thoughts

As we see manipulating the html report is actually the most time consuming task. It should be easy though to automate the text replacement. If someone actually does this I would be happy if you let me know so we can hand the script out to everyone.

When I created the coverage reports I didn't write them directly in the console but rather into a batch file:

This wraps it up for now.
As always I'm happy about any kind of constructive feedback.

The files I used and the coverage reports created can be found on the network drive E071-Daten