Isolate Bazel C++ build from system include directories

Question

I'd like to make my Bazel C++ build independent of /usr/include, /usr/local/include, etc. By default, as described in this thread, all files in /usr/include will be available when compiling C++ programs, not just standard ones.

What is the best way to achieve this?

One option would be to put the standard header files into a tarball and host it somewhere over HTTP, then add a new_http_repository and run gcc with -nostdinc -isysroot.

Answer 1

The solution consists of three parts:

• creating a custom C++ toolchain definition
• convincing Bazel to use that toolchain
• convincing the compiler to ignore its default system headers and use yours

Here's the reference you'll need:

https://docs.bazel.build/versions/master/crosstool-reference.html https://docs.bazel.build/versions/master/tutorial/crosstool.html

Here's an especially good tutorial:

https://github.com/bazelbuild/bazel/wiki/Yet-Another-CROSSTOOL-Writing-Tutorial

And some more tutorials, I don't know how up-to-date they are:

https://github.com/bazelbuild/bazel/wiki/About-the-CROSSTOOL https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain

---

Here's how I got this working on Linux with GCC:

1. Create a new workspace with a dummy cc_binary rule in it.

   mkdir /tmp/bar && cd /tmp/bar
   touch WORKSPACE
   echo "cc_binary(name='x',srcs=['x.cc'])" > BUILD
   echo -e "#include <stdio.h>\nint main(void) { return 0; }" > x.cc
   

2. Build the dummy target.

   bazel build //:x
   

   Since it's a cc_* rule, Bazel will initialize the C++ toolchain. This includes autogenerating a CROSSTOOL file. This is a text file that describes the compiler's interface so Bazel knows how to talk to it.

3. Create a package for your custom crosstool, copy the autogenerated CROSSTOOL and BUILD files there.

   mkdir /tmp/bar/my_toolchain
   touch /tmp/bar/my_toolchain/WORKSPACE
   cat $(bazel info output_base)/external/local_config_cc/CROSSTOOL > /tmp/bar/my_toolchain/CROSSTOOL
   cat $(bazel info output_base)/external/local_config_cc/BUILD > /tmp/bar/my_toolchain/BUILD
   

   We'll use the autogenerated files as templates to start from.

4. Add an external repository rule to the WORKSPACE file.

   echo "local_repository(name='my_toolchain', path='/tmp/bar/my_toolchain')" >> WORKSPACE
   

   This way you can reference your custom toolchain by a Bazel build label.

5. Try using your custom crosstool.

   bazel build --crosstool_top=@my_toolchain//:toolchain //:x
   

   Since it's the same as the autogenerated one, the build should succeed.

6. Edit system header paths from the "local" toolchain in the custom CROSSTOOL.

   Note: You may need to edit a different toolchain, if that's what will be selected. See toolchain selection here: https://docs.bazel.build/versions/master/crosstool-reference.html#toolchain-selection

   In a text editor, open /tmp/bar/my_toolchain/CROSSTOOL, find the "toolchain" record whose "toolchain_identifier" is "local", and comment out its "cxx_builtin_include_directory" entries (the comment character is "#").

   Then add the new "cxx_builtin_include_directory" for your toolchain directory:

   cxx_builtin_include_directory: "/tmp/bar/my_toolchain"
   

   You've just told Bazel where system headers are, so when it validates header inclusions, it will know that headers under this path are system headers and are allowed to be included.

7. Add a compiler flag to declare the location of system headers.

   Below the just-added "cxx_builtin_include_directory" line, add the following:

   compiler_flag: "-isystem=/tmp/bar/my_toolchain"
   

   You've just told Bazel to tell the compiler where it should look up system headers from.

8. Add the headers to the default set of inputs of compile actions.

   In a text editor, open /tmp/bar/my_toolchain/BUILD, find the "compiler_deps" filegroup, and edit its "srcs" attribute for example like so:

   srcs = glob(["**/*.h"]),
   

   You've just told Bazel to always include these files in C++ compilation actions, so the build will work with sandboxing and remote execution, and Bazel will rebuild C++ rules if any of the system headers (the ones you just added to srcs) change.

9. Create a mock stdc-predef.h.

   This is a header file the compiler always wants to include. Fortunately, the compiler will first look at your "-isystem" value before falling back to its own default path (which I don't know if you can tell it to ingore), so we can mock out this file like so:

   touch /tmp/bar/my_toolchain/std-predef.h
   

10. Try building with the custom crosstool.

    $ bazel build --crosstool_top=@my_toolchain//:toolchain //:x
    INFO: Build options have changed, discarding analysis cache.
    INFO: Analysed target //:x (0 packages loaded, 61 targets configured).
    INFO: Found 1 target...
    ERROR: /tmp/bar/BUILD:1:1: undeclared inclusion(s) in rule '//:x':
    this rule is missing dependency declarations for the following files included by 'x.cc':
      '/usr/include/stdio.h'
      (...)
    Target //:x failed to build
    Use --verbose_failures to see the command lines of failed build steps.
    INFO: Elapsed time: 0.332s, Critical Path: 0.09s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
    INFO: 0 processes.
    FAILED: Build did NOT complete successfully
    

    Since there's no stdio.h in our toolchain directory, but there is one in the compiler's own default directories, it will find the file and complain about missing headers. You can create a mock stdio.h to silence this.

    You can create a fake stdio.h to make the build work, or you can remove the #include <stdio.h> line from x.cc.

11. Create a fake system header in the custom toolchain.

    We'll use it to verify that Bazel picks up files from the custom toolchain.

    echo 'int foo_func(int x) { return x*2; }' > /tmp/bar/my_toolchain/foo.h
    

12. Rewrite the source file to include the custom system header.

    echo -e '#include <foo.h>\nint main(int argc, char**) { return foo_func(argc); }' > x.cc
    

13. Build again with the custom toolchain. Everything should work now.

    bazel build --crosstool_top=@my_toolchain//:toolchain //:x
    

14. Try running the built binary with 2 args (so argc=3):

    bazel-bin/x hello world ; echo $?
    6