Sharding with Google Firebase Test Lab to speed automation testing

Question

I am trying to shard tests using Flank with Firebase Test Lab to speed up the automated testing. However, I am not able to shard my tests despite using the parameter max-test-shards

Reference: https://github.com/TestArmada/flank

Code: Run test_runner/flank.yml with flank to verify Android execution is working.

cd test_runner/
./gradlew clean build fatJar
java -jar ./build/libs/flank-*.jar firebase test android run

There is no sharding happening and test is taking the same time to execute even after specifying the max-test-shards

flank.yml:

# -- GcloudYml --

## The name of a Google Cloud Storage bucket where raw test results will be stored # results-bucket: tmp_flank

## The name of a unique Google Cloud Storage object within the results bucket where raw test results will be stored ## (default: a timestamp with a random suffix). # results-dir: tmp

## Enable video recording during the test. Enabled by default, use --no-record-video to disable. # record-video: true

## The max time this test execution can run before it is cancelled (default: 15m). ## It does not include any time necessary to prepare and clean up the target device. ## The maximum possible testing time is 30m on physical devices and 60m on virtual devices. ## The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. # timeout: 30m

## Invoke a test asynchronously without waiting for test results. # async: false

## The history name for your test results (an arbitrary string label; default: the application's label from the APK manifest). ## All tests which use the same history name will have their results grouped together in the Firebase console in a time-ordered test history list. # results-history-name: android-history

## Experimental! ## The number of times a TestExecution should be re-attempted if one or more\nof its test cases fail for any reason. ## The maximum number of reruns allowed is 10. Default is 0, which implies no reruns. # flaky-test-attempts: 0

# -- AndroidGcloudYml --

## The path to the application binary file. ## The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. ## Android App Bundles are specified as .aab, all other files are assumed to be APKs. app: /Users/akanshaverma/test_flank/flank/test_runner/espresso_long.apk

## The path to the binary file containing instrumentation tests. ## The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. test: /Users/akanshaverma/test_flank/flank/test_runner/espresso_long_test.apk

## Automatically log into the test device using a preconfigured Google account before beginning the test. ## Enabled by default, use --no-auto-google-login to disable. auto-google-login: true

## Whether each test runs in its own Instrumentation instance with the Android Test Orchestrator ## (default: Orchestrator is used). Disable with --no-use-orchestrator. ## See https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator # use-orchestrator: true

## A comma-separated, key=value map of environment variables and their desired values. This flag is repeatable. ## The environment variables are mirrored as extra options to the am instrument -e KEY1 VALUE1 … command and ## passed to your test runner (typically AndroidJUnitRunner) # environment-variables: # coverage: true # coverageFilePath: /sdcard/ # clearPackageData: true

## A list of paths that will be copied from the device's storage to the designated results bucket after the test ## is complete. These must be absolute paths under /sdcard or /data/local/tmp # directories-to-pull: # - /sdcard/

## Monitor and record performance metrics: CPU, memory, network usage, and FPS (game-loop only). ## Enabled by default, use --no-performance-metrics to disable. # performance-metrics: true

## The fully-qualified Java class name of the instrumentation test runner ## (default: the last name extracted from the APK manifest). # test-runner-class: com.foo.TestRunner

## A list of one or more test target filters to apply (default: run all test targets). ## Each target filter must be fully qualified with the package name, class name, or test annotation desired. ## Supported test filters by am instrument -e … include: ## class, notClass, size, annotation, notAnnotation, package, notPackage, testFile, notTestFile ## See https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner for more information. # test-targets: # - class com.example.app.ExampleUiTest#testPasses

## A list of DIMENSION=VALUE pairs which specify a target device to test against. ## This flag may be repeated to specify multiple devices. ## The four device dimensions are: model, version, locale, and orientation. # device: # - model: NexusLowRes # version: 28 # locale: en # orientation: portrait # - model: NexusLowRes # version: 27 device: - model: hero2lte version: 23 locale: en orientation: portrait

flank: # -- FlankYml --

## test shards - the amount of groups to split the test suite into ## set to -1 to use one shard per test. default: 1 max-test-shards: 10

## shard time - the amount of time tests within a shard should take ## when set to > 0, the shard count is dynamically set based on time up to the maxmimum limit defined by max-test-shards ## 2 minutes (120) is recommended. ## default: -1 (unlimited) # shard-time: -1 shard-time: 2

## repeat tests - the amount of times to run the tests. ## 1 runs the tests once. 10 runs all the tests 10x # repeat-tests: 1

## Google cloud storage path to store the JUnit XML results from the last run. # smart-flank-gcs-path: gs://tmp_flank/flank/test_app_android.xml

## Disables smart flank JUnit XML uploading. Useful for preventing timing data from being updated. ## Default: false # smart-flank-disable-upload: false

## Disables sharding. Useful for parameterized tests. # disable-sharding: false disable-sharding: false

## always run - these tests are inserted at the beginning of every shard ## useful if you need to grant permissions or login before other tests run # test-targets-always-run: # - class com.example.app.ExampleUiTest#testPasses

## regex is matched against bucket paths, for example: 2019-01-09_00:13:06.106000_YCKl/shard_0/NexusLowRes-28-en-portrait/bugreport.txt # files-to-download: # - .*.mp4$

## The billing enabled Google Cloud Platform project name to use # project: delta-essence-114723

## Local folder to store the test result. Folder is DELETED before each run to ensure only artifacts from the new run are saved. # local-result-dir: flank

## Include additional app/test apk pairs in the run. If app is omitted, then the top level app is used for that pair. # additional-app-test-apks: # - app: ../test_app/apks/app-debug.apk # test: ../test_app/apks/app-debug-androidTest.apk

# - test: ../test_app/apks/app-debug-androidTest.apk

Answer 1

FYI, Firebase now natively supports sharding via gcloud beta firebase test android run, e.g. --num-uniform-shards works great.