CODEWITHSUNDEEP
pythonbuildbazel
user17648384
user17648384

Reputation: 13

How can I specify which Python toolchain to use in Bazel?

How can I configure Bazel to pick one toolchain over the other? I am okay with defining which toolchain to use via command-line argument or specifying which should be used in a specific target.

There are currently two toolchains being defined in my WORKSPACE file. I have two Python toolchains. One of them builds Python from source and includes it in the executable .zip output, and the other one does not.

When building, the toolchain that gets used is always the first toolchain which is registered. In this case, python3_tooolchain is used even though the build target imports requirement from hermetic_python3_toolchain.

# WORKSPACE

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("@rules_python//python:pip.bzl", "pip_install")

http_archive(
    name = "rules_python",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.5.0/rules_python-0.5.0.tar.gz",
    sha256 = "cd6730ed53a002c56ce4e2f396ba3b3be262fd7cb68339f0377a45e8227fe332",
)

# Non-hermetic toolchain 
register_toolchains("//src:python3_toolchain")

pip_install(
   quiet = False,
   name = "python_dependencies",
   requirements = "//:requirements.txt",
   python_interpreter = "/usr/bin/python3"
)

load("@python_dependencies//:requirements.bzl", "requirement")




# Hermetic toolchain 

_py_configure = """
if [[ "$OSTYPE" == "darwin"* ]]; then
    ./configure --prefix=$(pwd)/bazel_install --with-openssl=$(brew --prefix openssl)
else
    ./configure --prefix=$(pwd)/bazel_install
fi
"""

http_archive(
    name = "hermetic_interpreter",
    urls = ["https://www.python.org/ftp/python/3.11.0/Python-3.11.0.tar.xz"],
    sha256 = "a57dc82d77358617ba65b9841cee1e3b441f386c3789ddc0676eca077f2951c3",
    strip_prefix = "Python-3.11.0",
    patch_cmds = [
        "mkdir $(pwd)/bazel_install",
        _py_configure,
        "make",
        "make install",
        "ln -s bazel_install/bin/python3 python_bin",
    ],
    build_file_content = """
exports_files(["python_bin"])
filegroup(
    name = "files",
    srcs = glob(["bazel_install/**"], exclude = ["**/* *"]),
    visibility = ["//visibility:public"],
)
""",
)

pip_install(
    name = "hermetic_python3_dependencies",
    requirements = "//:requirements.txt",
    python_interpreter_target = "@hermetic_interpreter//:python_bin",
)
load("@hermetic_python3_dependencies//:requirements.bzl", "requirement")




load("@rules_python//python:defs.bzl", "py_binary")
load("@rules_python//python:defs.bzl", "py_library")

register_toolchains("//src:hermetic_python3_toolchain")


# src/BUILD

load("@bazel_tools//tools/python:toolchain.bzl", "py_runtime_pair")

# Non-hermetic toolchain 
py_runtime(
    name = "python3_runtime",
    interpreter_path = "/usr/bin/python3",
    python_version = "PY3",
    visibility = ["//visibility:public"],
)

py_runtime_pair(
    name = "python3_runtime_pair",
    py2_runtime = None,
    py3_runtime = ":python3_runtime",
)

toolchain(
    name = "python3_toolchain",
    toolchain = ":python3_runtime_pair",
    toolchain_type = "@bazel_tools//tools/python:toolchain_type",
)

# Hermetic toolchain

py_runtime(
    name = "hermetic_python3_runtime",
    files = ["@hermetic_interpreter//:files"],
    interpreter = "@hermetic_interpreter//:python_bin",
    python_version = "PY3",
    visibility = ["//visibility:public"],
)

py_runtime_pair(
    name = "hermetic_python3_runtime_pair",
    py2_runtime = None,
    py3_runtime = ":hermetic_python3_runtime",
)

toolchain(
    name = "hermetic_python3_toolchain",
    toolchain = ":hermetic_python3_runtime_pair",
    toolchain_type = "@bazel_tools//tools/python:toolchain_type",
)


package(default_visibility = ["//visibility:public"])


# /src/some_tool/BUILD

load("@hermetic_python3_dependencies//:requirements.bzl", "requirement") # Can load this rule from either `hermetic_python3_dependencies` or `python3_dependencies`, but does not seem to make a difference 

py_binary(
    name = "some-tool",
    main = "some_tool.py",
    srcs = ["some_tool_file.py"],
    python_version = "PY3",
    srcs_version = "PY3",
    deps = [
        requirement("requests"),
        "//src/common/some-library:library",
    ]
)

package(default_visibility = ["//visibility:public"])

Upvotes: 1

Views: 4735

Answers (1)

lummax
lummax

Reputation: 373

Consider upgrading rules_python, as that ruleset includes a hermetic python toolchain since https://github.com/bazelbuild/rules_python/releases/tag/0.7.0.

If that is not an option:

Currently you are registering two toolchains in your WORKSPACE.bazel file and bazel will use its toolchain resolution to pick one of them. You can debug that resolution with the --toolchain_resolution_debug=regex flag to see what is going on.

If you want to force the entire build to use one of the toolchains, remove registering the toolchains from the WORKSPACE.bazel file and create a .bazelrc:

build:hermetic_python --extra_toolchains=//src:hermetic_python3_toolchain
build:system_python --extra_toolchains=//src:python3_toolchain

Now you can switch between these toolchains by using bazel build --config=hermetic_python or bazel build --config=system_python.

Beware however, that this does not influence which of the python toolchains was used to run the pip_parse(). You need to take extra care from which you load the requirement() function. Simply by load()ing the function you force the evaluation of the pip_parse() and therefor the fetching/compilation of the corresponding python interpreter.

Upvotes: 1

Related Questions