Page MenuHomePhabricator

Short-range plan to prevent Differentials from unnecessarily building the superbuild
Closed, ResolvedPublic

Description

All the Differential/code review builds currently clog our Jenkins build queue. In particular for Windows we have a single executor for all MITK build jobs which leads to the situation, that even the nightly build will probably be delayed tonight. On the other side we already experienced that it is not a good idea to do Differential builds in continuous/incremental mode, as it will require manual interference now and then on the build clients.

However, I think it should be possible to write a script that checks if anything in CMakeExternals/ changed compared to the last Differential build. If this is the case, do a clean build, but otherwise just delete MITK-build instead of the whole workspace. This would already save as many hours of build time.

Event Timeline

kislinsk created this task.
kislinsk updated the task description. (Show Details)
kislinsk updated the task description. (Show Details)

Possible entry points in Jenkions job configuration:

  • Build EnvironmentDelete workspace before build startsAdvanced...External Deletion Command is possibly too early as it is hard to retrieve information at that point to check what files the Diff will affect.
  • Build EnvironmentApply Phabricator Differential has various options on how to apply the patch like creating a commit, applying it to master, or creating a branch. We may use the easiest option to later check affected files in a new build step based on a script.

We also have the possibility to modify the Herald rule that triggers the jobs. For example, if the revision summary contains a certain keyword, trigger a complete rebuild instead of an incremental build. Two disadventages are: Revision creators have to be aware of this possibility, and this is not enough as there may be also some other changes on the master branch since the last build on the build client that affect critical files.

The latter is a general problem. It is easy to query the files affected by the Differential on Jenkins side by calling git diff --cached --name-only. But it is hard to query all affected files since the last build. These may include files from the last Diff and files from other commits on the master in between.

I just thought about it could be a good idea to actually do this in our CMake script right in MITK. We can check all sorts of things there before and after external project processing.

External projects have an install_manifest.txt file in their build folder listing all installed files, by the way.

I activated MITK_AUTOCLEAN_EXTERNAL_PROJECTS on Differential and Continuous jobs. Fingers crossed.