Under what conditions can a remotely cached test result be drawn from in bazel?

Question

I'm trying to understand why these test executions I'm running never secure a remote cache hit on the first try. These tests are configured to pull in some remote cache configuration when executed and yet never windup presenting a cache hit on the first try.

Consider the following: starting from a completely fresh bazel cache, the test runs and passes

$ ls ~/.cache/bazel
ls: cannot access /home/USERX/.cache/bazel: No such file or directory
$
$
$ bazel test --config=remote //foo:bar
Starting local Bazel server and connecting to it...
INFO: Invocation ID: c3aca338-7093-4d17-aafb-928411575ae1
INFO: Analyzed target //foo:bar (327 packages loaded, 33610 targets configured).
INFO: Found 1 test target...
INFO: 11902 processes: 9086 remote cache hit (76.34%), 2815 internal (23.65%), 1 local (0.01%).
INFO: Cache hit rate for remote actions: 100.00% (9086 / 9086)
INFO: Total action wall time 1.35s
INFO: Critical path 3.56s (setup 2.08s, action wall time 1.35s)
INFO: Elapsed time 109.13s (preparation 78.29s, execution 30.84s)
INFO: Build completed successfully, 11902 total actions
//foo:bar                          PASSED in 1.4s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 11902 total actions

Executing a second time finally do we see the (cached) designation pop up

$ bazel test --config=remote //foo:bar
INFO: Invocation ID: 253212e5-e5a9-435b-96db-2ac23d5510ea
INFO: Analyzed target //foo:bar (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: 1 process: 1 internal (100.00%).
INFO: Cache hit rate for remote actions: -- (0 / 0)
INFO: Total action wall time 0.00s
INFO: Critical path 0.79s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 1.32s (preparation 0.42s, execution 0.90s)
INFO: Build completed successfully, 1 total action
//foo:bar                 (cached) PASSED in 1.4s

Am I mistaken in assuming that I could ever have a (cached) test result land on the very first test execution of a never before built or tested bazel workspace? Under what conditions if any would it be possible to land a cached test result on the very first run? Would it help to understand where the bazel retains the state necessary to determine a (cached) hit? Where does that information reside, on the filesystem, in the working memory of the bazel server process, in an environment variable?

---

Context

I'm running some HIL tests on some dedicated hardware that operate off a bamboo plan whereby bamboo creates its own workspace from scratch on an agent, clones the repo to it, and executes some bazel test commands in that workspace. The goal was to leverage the bazel remote cache to avoid even running tests which are not warranted, but it seems that can only be possible if the tests were executed at least once on that tester machine.

Answer 1

Apparently an exclusive tagged test target cannot contribute to the remote cache. A bit about that tag from the docs...

Bazel modifies test running behavior if it finds the following keywords in the tags attribute of the test rule:

exclusive will force the test to be run in the "exclusive" mode, ensuring that no other tests are running at the same time. Such tests will be executed in serial fashion after all build activity and non-exclusive tests have been completed. Remote execution is disabled for such tests because Bazel doesn't have control over what's running on a remote machine.

So the follow up question is how remote execution relates to the remote cache and why they would be coupled together in this aspect of bazel. The motivation behind why the remote execution would be disabled makes some sense. I actually needed exclusive tagging of the test case I'm working on because it's using system resources that would need further synchronization support to run these tests in parallel. Implementing such support is simply not worth it when there's a means to force such tests to run single session in serial fashion.

There's apparently a bug filed on it https://github.com/bazelbuild/bazel/issues/3791

---

(earlier explorations)

I leaned on what is effectively a hello world test case that happens to be in our repo, literally a c++ main that std::cout's a message and returns 0.

$ rm -rf  ~/.cache/bazel
$ bazel shutdown
Extracting Bazel installation...
$ bazel test --config=remote //foo:test_hello_world
Starting local Bazel server and connecting to it...
INFO: Invocation ID: a9f78e49-1145-47f6-971c-4d8ed0d97e0f
INFO: Analyzed target //foo:test_hello_world (46 packages loaded, 4287 targets configured).
INFO: Found 1 test target...
INFO: 7 processes: 3 remote cache hit (42.86%), 4 internal (57.14%).
INFO: Cache hit rate for remote actions: 100.00% (3 / 3)
INFO: Total action wall time 0.00s
INFO: Critical path 0.91s (setup 0.00s, action wall time 0.00s)
INFO: Elapsed time 54.69s (preparation 53.73s, execution 0.96s)
INFO: Build completed successfully, 7 total actions
//foo:test_hello_world                                    (cached) PASSED in 0.0s

The first two commands that delete the bazel cache and shutdown the bazel server should substantiate that one does not need to have local state persisted on the test machine to be capable of drawing a test result from the remote cache. Therefore there may be interference relating to the specific test target itself that for some reason does not mesh well with the remote cache.

Command line and Environment Inputs

You should also be aware that using --test_arg and --test_env inputs to the test target execution can perturb the bazel cache lookup as well. They can still participate in the remote cache so long as you always input the same values across runs. Using these mechanisms for injecting variable metadata or any other variable valued argument will amount to shooting yourself in the foot in leveraging the bazel cache. See https://stackoverflow.com/a/72638202/1330381