Why bazel limit 24 actions running even if build option "--jobs=240" have set?

Question

I have configured a toolchain for distcc, parameters were forwarded through a wrapper script to distcc.

distcc_wrapper_gcc.sh:

#!/bin/bash
set -euo pipefail
ccache distcc g++ "$@"

I want to start 240 parallel tasks like 'make -j240' before. the build command is:

bazel build --action_env=HOME --action_env=DISTCC_HOSTS="***" --config=joint_compilation --jobs=240 target

But the output is： 238 actions, 24 running

If I set --jobs lower than 24, the number of running actions will equal to it, else there are up to 24 processes running no matter what value in the parameters.

It really takes a long time to compile if there are only 24 actions running. Is there a hard limit of the running actions? (This computer has 12 cpus and 2 threads per core)

Is there a way to break or ignore this limit?

---

below is the config content.

.bazelrc

# Create a new CROSSTOOL file for our toolchain.

build:joint_compilation --crosstool_top=//toolchain:distcc

# Use --cpu as a differentiator.

build:joint_compilation --cpu=joint_compilation

# Specify a "sane" C++ toolchain for the host platform.

build:joint_compilation --host_crosstool_top=@bazel_tools//tools/cpp:toolchain

toolchain/BUILD

package(default_visibility = ['//visibility:public'])
cc_toolchain_suite(
    name = "distcc",
    toolchains = {
        "joint_compilation": ":joint_compilation_toolchain",
        "distcc|joint_compilation": ":joint_compilation_toolchain",
    },
)
filegroup(name = "empty")

filegroup(
name = "all",
srcs = [
   "distcc_wrapper_gcc.sh",
],
)

cc_toolchain(
    name = "joint_compilation_toolchain",
    toolchain_identifier = "joint_compilation-toolchain",
    all_files = ":all",
    compiler_files = ":all",
    cpu = "distcc",
    dwp_files = ":empty",
    dynamic_runtime_libs = [":empty"],
    linker_files = ":all",
    objcopy_files = ":empty",
    static_runtime_libs = [":empty"],
    strip_files = ":empty",
    supports_param_files = 0,
)

toolchain/CROSSTOOL

major_version: "1"
minor_version: "0"
default_target_cpu: "joint_compilation"

toolchain {
  toolchain_identifier: "joint_compilation-toolchain"
  host_system_name: "i686-unknown-linux-gnu"
  target_system_name: "joint_compilation-unknown-distcc"
  target_cpu: "joint_compilation"
  target_libc: "unknown"
  compiler: "distcc"
  abi_version: "unknown"
  abi_libc_version: "unknown"
  tool_path {
      name: "gcc"
      path: "distcc_wrapper_gcc.sh"
  }
   tool_path {
      name: "g++"
      path: "distcc_wrapper_gcc.sh"
  }
  tool_path {
      name: "ld"
      path: "/usr/bin/ld"
  }
  tool_path {
      name: "ar"
      path: "/usr/bin/ar"
  }
  tool_path {
      name: "cpp"
      path: "distcc_wrapper_gcc.sh"
  }
  tool_path {
      name: "gcov"
      path: "/usr/bin/gcov"
  }
  tool_path {
      name: "nm"
      path: "/usr/bin/nm"
  }
  tool_path {
      name: "objdump"
      path: "/usr/bin/objdump"
  }
  tool_path {
      name: "strip"
      path: "/usr/bin/strip"
  }


  cxx_builtin_include_directory: "/usr/lib/gcc/"
  cxx_builtin_include_directory: "/usr/local/include"
  cxx_builtin_include_directory: "/usr/include"
}

Answer 1

Unless there's some integration between distcc and bazel that I'm not aware of, bazel thinks it is executing everything on the local machine and is therefore limited by the local machine's resources. There is a local resources arg that can be tweaked, but instead I strongly recommend using bazel as intended. When building remotely, this means using a REAPI-capable buildfarm.

At least two exist:

• https://github.com/bazelbuild/bazel-buildfarm

  • the official impl, started by Uber, used by many
  • two components in the architecture: server (scheduler) and worker
  • cache can be stored on the workers, or in a grpc-based cache
    • the latter seems to have seen little use so far
  • written in java

• https://github.com/EdSchouten/bazel-buildbarn

  • written in go
  • somewhat different architecture (frontend/scheduler/worker)
  • more flexible caching

I've tried the former a little, and am about to try the latter - partially due to the caching, and partially due to the language: I find go far easier to read (and write) than java.