CMake add_subdirectory and recompiling

Question

I've just migrated a somewhat large project from Visual Studio solutions to CMake and I've noticed a weird behavior. I have something like the following structure:

project/CMakeLists.txt
project/code/CMakeLists.txt

project/code/library-1/CMakeLists.txt
project/code/library-1/*.hpp
project/code/library-1/*.cpp
project/code/library-2/CMakeLists.txt
project/code/library-2/*.hpp
project/code/library-2/*.cpp
...
project/code/library-n/CMakeLists.txt
project/code/library-n/*.hpp
project/code/library-n/*.cpp

project/demo/CMakeLists.txt
project/demo/demo-1/CMakeLists.txt
project/demo/demo-1/*.hpp
project/demo/demo-1/*.cpp
project/demo/demo-2/CMakeLists.txt
project/demo/demo-2/*.hpp
project/demo/demo-2/*.cpp
...
project/demo/demo-n/CMakeLists.txt
project/demo/demo-n/*.hpp
project/demo/demo-n/*.cpp

1. The root CMakeLists.txt file configures the compilation flags, macro definitions, etc. and uses CMake's add_subdirectory() to include targets defined by the libraries and demo projects.
2. The code sub-folder contains a flat list of sub-folders with each containing source code for a static library (as well as its target defined in a CMakeLists.txt file).
3. The demo sub-folder contains a flat list of sub-folders. Each contains source code for an executable and associated CMakeLists.txt file.
4. Each library is a standalone component and builds independently from all other libraries and demo projects.
5. Each demo program depends on one or more of the different libraries in the code sub-folder.

This setup is really nice. If I want to change build options, I only need to modify the root CMakeLists.txt and everything re-compiles with the new settings. If I modify any source code anywhere in the tree, the appropriate libraries, if any, are recompiled and all dependent demo programs are also re-built.

However, if I modify any CMakeLists.txt file anywhere in the tree, the entire tree of libraries and programs is re-compiled without respect of dependencies. To give an idea of what I mean, here a few parts of the CMake build scripts.

---

project/demo/CMakeLists.txt

# Resolve libraries built in `code` sub-folder.
link_directories(${LIBRARY_OUTPUT_PATH})

set(demo-projects
  demo-1
  demo-2
  ...
  demo-n
)
foreach(demo-project ${demo-projects})
  add_subdirectory(${demo-project})
endforeach()

---

project/demo/demo-n/CMakeLists.txt

# Find all source code in the same folder.
file(GLOB ${demo-project}_headers
  ${CMAKE_CURRENT_SOURCE_DIR}/*.hpp
)
file(GLOB ${demo-project}_sources
  ${CMAKE_CURRENT_SOURCE_DIR}/*.cpp
)

# Select libraries to link with.
set(${demo-project}_libraries
  library-1
  library-2
  library-5
)

# Build the demo program.
add_executable(${demo-project}
  ${${demo-project}_headers}
  ${${demo-project}_sources}
)
if(${demo-project}_libraries)
  target_link_libraries(${demo-project} ${${demo-project}_libraries})
endif()

# Manually register some dependencies on other targets.
if(${demo-project}_dependencies)
  add_dependencies(${demo-project} ${${demo-project}_dependencies})
endif()

---

If I happen to modify project/demo/demo-n/CMakeLists.txt to add an extra library, like this:

set(${demo-project}_libraries
  library-1
  library-2
  library-5
  library-6
)

Then the entire source code for all libraries and demo programs in the project is re-compiled. Why is this so? Is there a better way to structure my scripts in order to avoid this?

Answer 1

It happens that my problem was cause by a totally unrelated issue. I applied Bill Hoffman's suggestion and the modifying any "CMakeLists.txt" file in the project ended up modifying the CXX_FLAGS (C++ compiler flags) variable in all generated Makefiles.

I traced that back to my root "CMakeLists.txt" file, which had something like the following:

if(MSVC)
  # ...
  set(CMAKE_CXX_FLAGS_DEBUG
    "${CMAKE_CXX_FLAGS_DEBUG} /WX /wd4355" // depends on cached value.
    CACHE STRING "Debug compiler flags" FORCE)
  # ...
endif()

I changed it to the following.

if(MSVC)
  # ...
  set(CMAKE_CXX_FLAGS_DEBUG
    "/DWIN32 /D_WINDOWS /WX /wd4355" // no longer depends on cached value.
    CACHE STRING "Debug compiler flags" FORCE)
  # ...
endif()

CMake no longer repeats the /WX /wd4355 flags when updating the build scripts and my project no longer re-compiles from scratch at each modification!

Answer 2

Hi did you try to remove the

link_directories(${LIBRARY_OUTPUT_PATH})

statement from project/demo/CMakeLists.txt

I do not see why this should be necessary. All the target you are linking to should be specified by target_link_libraries(...)

Answer 3

The first thing you want to do is figure out what changes. You can use git to help you do that if you have it installed.

1. Run cmake on your project with an out of source build
2. cd to build directory
3. create a git repo out of the build tree git add . git commit -m "add build tree"
4. change the cmakefile that causes the rebuild
5. re-run cmake on the build tree cmake .
6. run git diff and see what changed.

Answer 4

It is true that if any CMakeLists.txt file changes (or any input to it, such as the "source" file of a configure_file call) then CMake will re-run at the top level of your build tree, and regenerate the solution files and project files that have changed.

However, it should only regenerate files that are different from the last time that it ran... so based on what you've shown in your question, I do not have a good explanation for why everything is rebuilding.

On the other hand, CMake does leave it up to Visual Studio to decide what to re-build when a "build solution" is triggered. We do not express any dependencies for VS, other than putting the sources and headers in the right projects, setting include directories properly, and trusting VS to analyze the includes and rebuild things properly when headers and source files change.

You do not show any include_directories calls. Do you make those in your top level CMakeLists.txt file such that all sub-directories have the same set of include values? If so, perhaps that's the thing that's triggering a rebuild of everything.

We certainly do our best to make CMake produce build systems that minimize rebuild times.

Is your project public? Can I see the full source code for it and try to reproduce the problem on my own machine?

Answer 5

I'm not familiar with the way CMake generates Visual Studio solution/project files, but I'm guessing it's generating one (set?) for your entire project. Due to the nature of the way CMake works (I can't find good documentation on this; maybe the "Mastering CMake" book?), if the subdirectory project files are not separate, it will regenerate them any time a CMakeLists.txt file is changed. Then, Visual Studio would be responsible for rebuilding the whole thing or not (I'm guessing that it rebuilds it when it sees that the project file has changed).

The documentation for add_subdirectory does say

The CMakeLists.txt file in the specified source directory will be processed immediately by CMake before processing in the current input file continues beyond this command.

That seems to indicate that all the subdirectories are processed as part of the main project.

That same documentation mentions the use of the project command in a subdirectory. It's talking about when you use the EXCLUDE_FROM_ALL argument, but it seems possible (worth a try) that it could help in your situation anyway. For example, maybe if CMake generates the demo code as its own project, the main code won't get rebuilt when the demo project file changes.

I think that, to get a definitive answer, you may need to post on the CMake mailing list (and if you get a good answer, post it here).