Page MenuHomePhabricator
Create Task
Maniphest T22656

Boost binaries provoke a compile error in superbuild when built for multiple configurations
Closed, ResolvedPublic
Actions

Description

If one makes superbuilds for several configurations (e.g. debug and release) with boost binaries activated the superbuild will fail.

Problem seems to root in a conflict with the already existing boost binary directory. The cmake tries to copy the second build dir to the same destination and fails if the destination already exists. This should be addressed appropriatly.

Example error message:

2> Building Custom Rule D:/Dev/MITK/src/CMakeLists.txt
2> CMake does not need to re-run because D:\Dev\MITK\build\CMakeFiles\generate.stamp is up-to-date.
2> Creating directories for 'Boost-download'
2> Performing download step (download, verify and extract) for 'Boost-download'
2> -- verifying file...
2> file='D:/Dev/MITK/build/ep/src/boost_1_60_0.7z'
2> -- File already exists and hash match (skip download):
2> file='D:/Dev/MITK/build/ep/src/boost_1_60_0.7z'
2> MD5='7ce7f5a4e396484da8da6b60d4ed7661'
2> -- extracting...
2> src='D:/Dev/MITK/build/ep/src/boost_1_60_0.7z'
2> dst='D:/Dev/MITK/build/ep/src/Boost-download'
2> -- extracting... [tar xfz]
2> -- extracting... [analysis]
2> -- extracting... [rename]
2> CMake Error at Boost-download-stamp/extract-Boost-download.cmake:51 (file):
2> file RENAME failed to rename
2>
2> D:/Dev/MITK/build/ep/src/ex-Boost-download1234/boost_1_60_0
2>
2> to
2>
2> D:/Dev/MITK/build/ep/src/Boost-download
2>
2> because: Directory not empty if one

Related Objects
Search...

Event Timeline

floca created this task.Mar 23 2017, 2:24 PM
hentsch added a subscriber: hentsch.Mar 27 2017, 12:45 PM
• goch added a subscriber: • goch.Mar 29 2017, 10:05 AM

This is flagged as an MITK bug, does this happen on a regular MITK superbuild?

floca added a comment.Mar 30 2017, 11:24 AM

Depends on what you deem "regular". It should happen with any Superbuild as soon as you inforce (somehow) that boost binaries should be build.

As long as you use boost header only, you have no problems.

kislinsk triaged this task as Normal priority.Apr 27 2017, 11:20 AM
kislinsk claimed this task.Apr 28 2017, 10:32 AM
kislinsk removed kislinsk as the assignee of this task.Apr 28 2017, 11:20 AM
kislinsk added a subscriber: kislinsk.

Couldn't reproduce with MSVC 2015. Set MITK_USE_Boost_LIBRARIES to chrono;system;thread and batch built Boost in release and debug mode.

floca removed a project: MITK.Sep 21 2017, 4:46 PM
floca added a project: Restricted Project.Sep 21 2017, 8:33 PM

Gathered some more insights. If I do not use the "super-super-build", but directly just build the MITK super-build everything works fine (so both, release and debug, in one solution). So I assume it is a problem of the super-super-build solution?!?

hentsch added a comment.Sep 22 2017, 8:24 AM

One reason more to change eMITK architecture to AwesomeProject

• goch added a comment.EditedSep 26 2017, 5:04 PM

Awesome project seems to have the same issue. I pulled it from Github and did the following change:

diff --git a/CMakeExternals/MITK.cmake b/CMakeExternals/MITK.cmake
index 1d0a0d1..f30ae72 100644
--- a/CMakeExternals/MITK.cmake
+++ b/CMakeExternals/MITK.cmake
@@ -123,7 +123,7 @@ if(NOT MITK_DIR)

   set(MITK_SOURCE_DIR "" CACHE PATH "MITK source code location. If empty, MITK will be cloned from MITK_GIT_REPOSITORY")
   set(MITK_GIT_REPOSITORY "https://phabricator.mitk.org/source/mitk.git" CACHE STRING "The git repository for cloning MITK")
-  set(MITK_GIT_TAG "releases/2016-11" CACHE STRING "The git tag/hash to be used when cloning from MITK_GIT_REPOSITORY")
+  set(MITK_GIT_TAG "origin/personal/floca/DIPP-2017-10" CACHE STRING "The git tag/hash to be used when cloning from MITK_GIT_REPOSITORY")
   mark_as_advanced(MITK_SOURCE_DIR MITK_GIT_REPOSITORY MITK_GIT_TAG)

   #-----------------------------------------------------------------------------
@@ -150,6 +150,8 @@ if(NOT MITK_DIR)
   if(MITK_INITIAL_CACHE_FILE)
     list(APPEND additional_mitk_cmakevars "-DMITK_INITIAL_CACHE_FILE:INTERNAL=${MITK_INITIAL_CACHE_FILE}")
   endif()
+
+  list(APPEND additional_mitk_cmakevars "-DMITK_USE_Boost_LIBRARIES:STRING=filesystem$<SEMICOLON>regex$<SEMICOLON>date_time$<SEMICOLON>program_options$<SEMICOLON>system$<SEMICOLON>thread$<SEMICOLON>chrono")

   if(MITK_USE_SUPERBUILD)
     set(MITK_BINARY_DIR ${proj}-superbuild)

Then batch build it for Debug and Release with MSVS 2015 and CMake 3.8.2 (below is the release step)

...updated 11957 targets...
-- Moving lib file from bin to lib directory: D:/projects/awesome/bin/MITK-superbuild/ep/bin/boost_chrono-vc140-mt-1_60.lib
-- Moving lib file from bin to lib directory: D:/projects/awesome/bin/MITK-superbuild/ep/bin/boost_date_time-vc140-mt-1_60.lib
-- Moving lib file from bin to lib directory: D:/projects/awesome/bin/MITK-superbuild/ep/bin/boost_filesystem-vc140-mt-1_60.lib
-- Moving lib file from bin to lib directory: D:/projects/awesome/bin/MITK-superbuild/ep/bin/boost_program_options-vc140-mt-1_60.lib
-- Moving lib file from bin to lib directory: D:/projects/awesome/bin/MITK-superbuild/ep/bin/boost_regex-vc140-mt-1_60.lib
-- Moving lib file from bin to lib directory: D:/projects/awesome/bin/MITK-superbuild/ep/bin/boost_system-vc140-mt-1_60.lib
-- Moving lib file from bin to lib directory: D:/projects/awesome/bin/MITK-superbuild/ep/bin/boost_thread-vc140-mt-1_60.lib
Completed 'Boost'

Debug then

  Das Projekt "D:\projects\awesome\bin\MITK-superbuild\Boost.vcxproj" (4) erstellt "D:\projects\awesome\bin\MITK-superbuild\Boost-download.vcxproj" (5) auf Knoten "1" (Standardziele).
PrepareForBuild:
  Das Verzeichnis "x64\Debug\Boost-download\" wird erstellt.
  Das Verzeichnis "x64\Debug\Boost-download\Boost-download.tlog\" wird erstellt.
InitializeBuildStatus:
  "x64\Debug\Boost-download\Boost-download.tlog\unsuccessfulbuild" wird erstellt, da "AlwaysCreate" angegeben wurde.
ComputeCustomBuildOutput:
  Das Verzeichnis "D:\projects\awesome\bin\MITK-superbuild\ep\src\Boost-download-stamp\Debug\" wird erstellt.
CustomBuild:
  Building Custom Rule D:/projects/awesome/bin/MITK/CMakeLists.txt
  CMake does not need to re-run because D:/projects/awesome/bin/MITK-superbuild/CMakeFiles/generate.stamp is up-to-date.
  Creating directories for 'Boost-download'
  Performing download step (download, verify and extract) for 'Boost-download'
  -- verifying file...
         file='D:/projects/awesome/bin/MITK-superbuild/ep/src/boost_1_60_0.7z'
  -- File already exists and hash match (skip download):
    file='D:/projects/awesome/bin/MITK-superbuild/ep/src/boost_1_60_0.7z'
    MD5='7ce7f5a4e396484da8da6b60d4ed7661'
  -- extracting...
       src='D:/projects/awesome/bin/MITK-superbuild/ep/src/boost_1_60_0.7z'
       dst='D:/projects/awesome/bin/MITK-superbuild/ep/src/Boost-download'
  -- extracting... [tar xfz]
  -- extracting... [analysis]
  -- extracting... [rename]
  CMake Error at Boost-download-stamp/extract-Boost-download.cmake:51 (file):
    file RENAME failed to rename

      D:/projects/awesome/bin/MITK-superbuild/ep/src/ex-Boost-download1234/boost_1_60_0

    to

      D:/projects/awesome/bin/MITK-superbuild/ep/src/Boost-download

    because: Directory not empty



C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" wurde mit dem Code 1 beendet. [D:\projects\awesome\bin\MITK-superbuild\Boost-download.vcxproj]
Die Erstellung des Projekts "D:\projects\awesome\bin\MITK-superbuild\Boost-download.vcxproj" ist abgeschlossen (Standardziele) -- FEHLER.
Die Erstellung des Projekts "D:\projects\awesome\bin\MITK-superbuild\Boost.vcxproj" ist abgeschlossen (Standardziele) -- FEHLER.
hentsch added a parent task: Restricted Maniphest Task.Oct 11 2017, 4:33 PM
floca added a comment.Feb 14 2018, 9:38 AM

I can reproduce the error on windows systems on the current master with the mitk superbuild itself. I use the new extension feature on windows. If I build release and then debug (in the same VS session; don't know yet if this is important). I get the following error:

4>------ Erstellen gestartet: Projekt: Boost, Konfiguration: Debug x64 ------
4>Building Custom Rule D:/Dev/DIPP/MITK-source/CMakeLists.txt
4>CMake does not need to re-run because D:/Dev/DIPP/build/CMakeFiles/generate.stamp is up-to-date.
4>Creating directories for 'Boost'
4>Performing download step (download, verify and extract) for 'Boost'
4>-- verifying file...
4> file='D:/Dev/DIPP/build/ep/src/boost_1_65_1.7z'
4>-- File already exists and hash match (skip download):
4> file='D:/Dev/DIPP/build/ep/src/boost_1_65_1.7z'
4> MD5='72ab92cb936f93d33b8b313aee2dd47a'
4>-- extracting...
4> src='D:/Dev/DIPP/build/ep/src/boost_1_65_1.7z'
4> dst='D:/Dev/DIPP/build/ep/src/Boost'
4>-- extracting... [tar xfz]
4>-- extracting... [analysis]
4>-- extracting... [rename]
4>CMake Error at Boost-stamp/extract-Boost.cmake:51 (file):
4> file RENAME failed to rename
4>
4> D:/Dev/DIPP/build/ep/src/ex-Boost1234/boost_1_65_1
4>
4> to
4>
4> D:/Dev/DIPP/build/ep/src/Boost
4>
4> because: Directory not empty
4>
4>
4>
4>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" wurde mit dem Code 1 beendet.
4>Die Erstellung des Projekts "Boost.vcxproj" ist abgeschlossen -- FEHLER.

I also geht the error if I permutate the build types. So first debug and then release.
I now test if the problem persists when I don't use the extension feature.

hentsch added a comment.Feb 14 2018, 10:01 AM

I compiled the current MITK master without compiling errors on Debug and Release.
When enabling the extension feature, no boost libraries are built.
External projects complain:

CMake Error at C:/Program Files/CMake/share/cmake-3.10/Modules/FindBoost.cmake:1928 (message):
  Unable to find the requested Boost libraries.

  Boost version: 1.65.1

  Boost include path: D:/dev/MITK-bin/m-1802/ep/src/Boost

  Could not find the following Boost libraries:

          boost_filesystem
          boost_system
          boost_thread
          boost_date_time

  Some (but not all) of the required Boost libraries were found.  You may
  need to install these additional Boost libraries.  Alternatively, set
  BOOST_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT
  to the location of Boost.
kislinsk added a comment.Feb 14 2018, 10:32 AM

I can confirm Ralfs error. As it worked in the past, it might have something to do with merging the Boost-download and Boost projects to a single project. I'll check.

floca added a comment.Feb 14 2018, 11:08 AM
In T22656#122810, @hentsch wrote:

I compiled the current MITK master without compiling errors on Debug and Release.

Strange.

When enabling the extension feature, no boost libraries are built.
External projects complain:

Have you set the "MITK_USE_Boost_LIBRARIES" accordingly?

hentsch added a comment.Feb 14 2018, 11:12 AM

Have you set the "MITK_USE_Boost_LIBRARIES" accordingly?

No. Obviously, MITK only needs boost header libraries per default.
Regarding eMITK: I suppose it's set by RTTB, but I'll check.

floca added a comment.Feb 14 2018, 11:13 AM

I tried, but failed so far. I am in the process to clarify with Stefan, where the problem is.

kislinsk added a parent task: T24615: Make Boost build robust.Oct 12 2018, 2:02 PM
kislinsk added a comment.Oct 19 2018, 1:19 PM

I think I just figured it out, kind of. Long story short, we have to separate the Boost-download step again into an extra project before the actual Boost project.

floca added a comment.Oct 19 2018, 2:09 PM

that would be great.

kislinsk added a comment.Oct 24 2018, 3:25 PM

Okay, this is nothing we can really fix. I'm not even sure if CMake can fix this. The problem is that if two file-related commands are executed in a row, there's a race condition. For example, the extract script calls a remove_recurse on the destination folder and then renames the actual extraction folder to the destination folder. The second command fails, because the destination folder is still there, even though the first command returned.

I also noticed very strange behavior in a case of a copy FOO to BACKUP followed by a copy REPLACEMENT to FOO. The replacement ended up of being copied to the backup.

I think, it may be Symantec EndPoint Protection as it is heavily doing stuff while building/extracting. I added security risk and SONAR exceptions for my build drive and until now I didn't experience any of these errors anymore.

kislinsk closed this task as Resolved.Oct 25 2018, 12:20 PM
kislinsk claimed this task.
MITK · MIC · IMSY · DKFZ