CODEWITHSUNDEEP
bazel
jxramos
jxramos

Reputation: 8266

Effect of --test_env and --test_arg on bazel cache

I'm naively passing along some variable test metadata to some py_test targets to inject that metadata into some test result artifacts that later get uploaded to the cloud. I'm doing so using either the --test_env or --test_arg values at the bazel test invocation.

Would this variable data negatively affect the way test results are cached such that running the same test back to back would effectively disturb the bazel cache?

Upvotes: 0

Views: 563

Answers (1)

jxramos
jxramos

Reputation: 8266

Command Line Inputs

Command line inputs can indeed disturb cache hits. Consider the following set of executions

BUILD file

py_test(
    name = "test_inputs",
    srcs = ["test_inputs.py"],
    deps = [
        ":conftest",
        "@pytest",
    ],
)

py_library(
    name = "conftest",
    srcs = ["conftest.py"],
    deps = [
        "@pytest",
    ],
)

Test module

import sys
import pytest

def test_pass():
    assert True

def test_arg_in(request):
    assert request.config.getoption("--metadata")

if __name__ == "__main__":
    args = sys.argv[1:]
    ret_code = pytest.main([__file__, "--log-level=ERROR"] + args)
    sys.exit(ret_code)

First execution

$ bazel test //bazel_check:test_inputs --test_arg --metadata=abc
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.40s
INFO: Critical path 0.57s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.72s (preparation 0.12s, execution 0.60s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs                                                  PASSED in 0.4s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions

Second execution: same argument value, cache hit!

$ bazel test //bazel_check:test_inputs --test_arg --metadata=abc
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 1 process: 1 internal (100.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.00s
INFO: Critical path 0.47s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.61s (preparation 0.12s, execution 0.49s)
INFO: Build completed successfully, 1 total action
//bazel_check:test_inputs                                         (cached) PASSED in 0.4s

Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action

Third execution: new argument value, no cache hit

$ bazel test //bazel_check:test_inputs --test_arg --metadata=kk
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 93 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.30s
INFO: Critical path 0.54s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.71s (preparation 0.14s, execution 0.57s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs                                                  PASSED in 0.3s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions

Fourth execution: reused same argument as first two runs

Interestingly enough there is no cache hit despite the result being cached earlier. Somehow it did not persist.

$ bazel test //bazel_check:test_inputs --test_arg --metadata=abc
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.34s
INFO: Critical path 0.50s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.71s (preparation 0.17s, execution 0.55s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs                                                  PASSED in 0.3s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions

Environment Inputs

The same exact behavior applies for --test_env inputs

import os
import sys
import pytest

def test_pass():
    assert True

def test_env_in():
    assert os.environ.get("META_ENV")

if __name__ == "__main__":
    args = sys.argv[1:]
    ret_code = pytest.main([__file__, "--log-level=ERROR"] + args)
    sys.exit(ret_code)

First execution

$ bazel test //bazel_check:test_inputs --test_env META_ENV=33
INFO: Build option --test_env has changed, discarding analysis cache.
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 7285 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.29s
INFO: Critical path 0.66s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 1.26s (preparation 0.42s, execution 0.84s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs                                                  PASSED in 0.3s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions

Second execution: same env value, cache hit!

$ bazel test //bazel_check:test_inputs --test_env META_ENV=33
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 1 process: 1 internal (100.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.00s
INFO: Critical path 0.49s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 0.67s (preparation 0.15s, execution 0.52s)
INFO: Build completed successfully, 1 total action
//bazel_check:test_inputs                                         (cached) PASSED in 0.3s

Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action

Third execution: new env value, no cache hit

$ bazel test //bazel_check:test_inputs --test_env META_ENV=44
INFO: Build option --test_env has changed, discarding analysis cache.
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 7285 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.29s
INFO: Critical path 0.62s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 1.22s (preparation 0.39s, execution 0.83s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs                                                  PASSED in 0.3s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions

Fourth execution: reused same env value as first two runs

$ bazel test //bazel_check:test_inputs --test_env META_ENV=33
INFO: Build option --test_env has changed, discarding analysis cache.
INFO: Analyzed target //bazel_check:test_inputs (0 packages loaded, 7285 targets configured).
INFO: Found 1 test target...
INFO: 2 processes: 1 internal (50.00%), 1 local (50.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.28s
INFO: Critical path 0.66s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 1.25s (preparation 0.40s, execution 0.85s)
INFO: Build completed successfully, 2 total actions
//bazel_check:test_inputs                                                  PASSED in 0.3s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions

Upvotes: 1

Related Questions