Page MenuHomePhabricator

CTest returns error code if subproject does not have tests
Closed, ResolvedPublic

Assigned To
Authored By
kislinsk
Jan 21 2020, 4:18 PM
Tags
Subscribers
Tokens
"Party Time" token, awarded by kislinsk.

Description

Subprojects without tests lead to CTest returning 255 instead of 0, resulting in wringly declared failed builds in CI.

Event Timeline

kislinsk triaged this task as Normal priority.Jan 21 2020, 4:18 PM
kislinsk created this task.

Reverted the removal as CTest still fails, probably because of no tests in superbuild projects. I think the best way to solve this is to contribute to CMake.

kislinsk renamed this task from Remove Subprojects without tests to CTest returns error code if subproject does not have tests.Jan 22 2020, 5:50 AM

In CMake, Source/CTest/cmCTestTestHandler.cxx, cmCTestTestHandler::ProcessHandler(), there is a check if the sum of passed and failed tests equal zero. After another pre-check (see below), "No tests were found!!!" is logged as ERROR_MESSAGE. In Source/cmCTest.cxx, cmCTest::Log(), cmSystemTools::SetErrorOccured() is called if the log type is ERROR_MESSAGE.

The other pre-check I wrote above is "if not cmCTest::GetShowOnly() and not cmCTest::ShouldPrintLabels()". The former condition is set to true if -N/--show-only[=...] is passed as argument to CTest. The latter condition is set to true if --print-labels is passed as argument to CTest.

-N/--show-only[=...] disables the actual execution of tests.

--print-labels prints all available test labels and does not run any tests.

The only option I see is to contribute to CMake. To stay backward-compatible, we probably cannot simply change the log type from ERROR_MESSAGE to WARNING_MESSAGE or similar. I think I will propose a new argument for CTest like --no-tests=<[error|ignore]>.

Here's my CMake merge request: https://gitlab.kitware.com/cmake/cmake/merge_requests/4258

Just for documentation purposes, this is how I ran my tests:

CMake-build/Tests$ RunCMake_TEST_FILTER="^ignore-no-tests" ctest -V -C Release -R '^RunCMake\.CTestCommandLine'

Merged! 🎉 Will be available in CMake 3.17.

CMake 3.17 is out now. I configured the MITK Nightlies to use CMake 3.17 from tonight on and added the new --no-tests=ignore argument to the call of CTest.

kislinsk added a project: Restricted Project.Mar 24 2020, 2:10 PM
kislinsk moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.
kislinsk added a project: Restricted Project.Mar 25 2020, 8:29 AM
kislinsk moved this task from In Progress to Done on the CI board.

It worked. The nightlies are sunny now and the documentation was uploaded to docs.mitk.org. I changed every MITK and eMITK job to use CMake 3.17 now and passed --no-tests=ignore to CTest.

kislinsk moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.
kislinsk moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.