Page MenuHomePhabricator

Different coverage results with OpenCppCoverage and Gcov
Open, NormalPublic

Description

I did a code coverage with OpenCppCoverage and compared my output with http://cdash.mitk.org/viewCoverage.php?buildid=723463&status=2.
In my case I got an coverage of 70% but on CDash - which is based on Gcov I think - it's 82%.

Does anybody know what could be the reason for that?

Event Timeline

j762e triaged this task as Normal priority.Apr 3 2020, 1:47 PM
j762e created this task.
kalali added a subscriber: nolden.Apr 3 2020, 2:01 PM

Unfortunately I cannot see any colored lines in the CDash html but it's ok since I'm able to see the output in the provided link.
@nolden can you shed any light on this?

nolden added a comment.Apr 7 2020, 9:51 PM

If I look at the outputs side by side there is e.g. a difference in the SetSecondaryMask method: it's called 24 times in the CDash run but in the OpenCppCoverage it's red. Which tests did you execute? I think if I remember correctly the numbers in the CTest/CDash result are aggregated over all tests in the default config. Could this make the difference?

kalali added a project: Restricted Project.Apr 8 2020, 10:27 AM
j762e added a comment.Apr 8 2020, 11:59 AM

I executed all tests in files.cmake, which are: mitkImageStatisticsCalculatorTest, mitkPointSetStatisticsCalculatorTest, mitkPointSetDifferenceStatisticsCalculatorTest, mitkImageStatisticsTextureAnalysisTest, mitkImageStatisticsContainerManagerTest and mitkImageStatisticsHotspotTest.
But I think the reason for the different results might be that opencppcoverage does not use any testfiles and cannot load pictures because it cannot find these and there is no option to pass them to opencppcoverage.

If I look at the outputs side by side there is e.g. a difference in the SetSecondaryMask method: it's called 24 times in the CDash run but in the OpenCppCoverage it's red. [...]

That's exactly what we werelooking at. Difference in percentage is because of the different number of lines / functions not hit by the tests. The question was, why are the functions not reached by OpenCppCoverage.

But I think the reason for the different results might be that opencppcoverage does not use any testfiles and cannot load pictures because it cannot find these and there is no option to pass them to opencppcoverage.

Ok, I think this is the reason for the difference and I think this has to be fixed to make the output of OpenCppCoverage meaningful. Many tests just don't do anything useful without test images. You can find the proper command line to start the test if you execute "ctest -V -R <test-name>" in the MITK-build directory, or look it up on CDash and adapt the paths to your local setup. E.g. look here and click "Show command line":

http://cdash.mitk.org/testDetails.php?test=13770846&build=724301