Android Instrumentation Tests Paritally Failing with androidx orchestrator

Question

I am attempting to get our tests running under androidx orchestrator.

It seems to be partially working but not all tests are being found and run.

I have extracted the relevant testing code into a helper method I call on the projects build.gradle.

ext.enableTesting = { Project project ->
    logger.lifecycle "Enable Testing: " + project.displayName

    project.android {
        defaultConfig {
            testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"

            // runs pm clear after each test invocation
            testInstrumentationRunnerArguments clearPackageData: 'true'
        }

        testOptions {
            execution 'ANDROIDX_TEST_ORCHESTRATOR'
            animationsDisabled true

            unitTests {
                includeAndroidResources = true
            }
        }

        useLibrary 'android.test.runner'
        useLibrary 'android.test.base'
        useLibrary 'android.test.mock'
    }

    project.dependencies {
        // Tests - Including NoOp
        testImplementation 'junit:junit:4.12'

        androidTestUtil 'androidx.test:orchestrator:1.2.0'

        // Core library
        androidTestImplementation 'androidx.test:core:1.2.0'

        // AndroidJUnitRunner and JUnit Rules
        androidTestImplementation 'androidx.test:runner:1.2.0'
        androidTestImplementation 'androidx.test:rules:1.2.0'

        // Assertions
        androidTestImplementation 'androidx.test.ext:junit:1.1.1'
        androidTestImplementation 'androidx.test.ext:truth:1.2.0'

        // Mockito
        testImplementation "org.mockito:mockito-core:${project.ext.mockitoVersion}"
        testImplementation "org.mockito:mockito-android:${project.ext.mockitoVersion}"
        androidTestImplementation "org.mockito:mockito-core:${project.ext.mockitoVersion}"
        androidTestImplementation "org.mockito:mockito-android:${project.ext.mockitoVersion}"

    }
    // 'androidx.test.ext:truth:1.2.0' pulls in some conflicting
    // versions with other things we need. In particular it has a more
    // recent version of guava which then screws up grpc stuff.
    project.configurations.all {
        resolutionStrategy {
            force 'com.google.errorprone:error_prone_annotations:2.1.2',
                'com.google.code.findbugs:jsr305:3.0.2',
                'com.google.guava:guava:26.0-android',
                'org.checkerframework:checker-compat-qual:2.5.3'
        }
    }
}

18 of our tests pass and logcat has the following for those tests:

06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.LogRunListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.CoverageListener
06-14 03:13:10.135: I/TestRunner(17920): run started: 1 tests
06-14 03:13:10.139: I/TestRunner(17920): started: testSetAvailable(app.impl.AvailabilityManagerTest)
06-14 03:13:11.914: I/TestRunner(17920): finished: testSetAvailable(app.impl.AvailabilityManagerTest)
06-14 03:13:11.924: I/TestRunner(17920): run finished: 1 tests, 0 failed, 0 ignored

However, a test in another package fails with:

06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.LogRunListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.CoverageListener
06-14 03:15:07.676: I/TestRunner(28814): run started: 1 tests
06-14 03:15:07.676: E/TestRunner(28814): failed: Test mechanism
06-14 03:15:07.677: E/TestRunner(28814): ----- begin exception -----
06-14 03:15:07.677: E/TestRunner(28814): java.lang.IllegalStateException: Unable to send notification, callback is null
06-14 03:15:07.677: E/TestRunner(28814):    at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.sendTestNotification(OrchestratedInstrumentationListener.java:163)
06-14 03:15:07.677: E/TestRunner(28814):    at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.testRunStarted(OrchestratedInstrumentationListener.java:94)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.notification.SynchronizedRunListener.testRunStarted(SynchronizedRunListener.java:35)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier$1.notifyListener(RunNotifier.java:91)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier.fireTestRunStarted(RunNotifier.java:88)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.JUnitCore.run(JUnitCore.java:136)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
06-14 03:15:07.677: E/TestRunner(28814):    at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
06-14 03:15:07.677: E/TestRunner(28814):    at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
06-14 03:15:07.677: E/TestRunner(28814):    at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2106)
06-14 03:15:07.677: E/TestRunner(28814): ----- end exception -----
06-14 03:15:07.680: I/TestRunner(28814): started: testMatcherWithMatchFound(app.impl.matcher.MatcherTest)
06-14 03:15:07.681: E/TestRunner(28814): failed: Test mechanism
06-14 03:15:07.681: E/TestRunner(28814): ----- begin exception -----
06-14 03:15:07.681: E/TestRunner(28814): java.lang.IllegalStateException: Unable to send notification, callback is null
06-14 03:15:07.681: E/TestRunner(28814):    at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.sendTestNotification(OrchestratedInstrumentationListener.java:163)
06-14 03:15:07.681: E/TestRunner(28814):    at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.testStarted(OrchestratedInstrumentationListener.java:114)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.notification.SynchronizedRunListener.testStarted(SynchronizedRunListener.java:49)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier$3.notifyListener(RunNotifier.java:121)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier.fireTestStarted(RunNotifier.java:118)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.internal.runners.model.EachTestNotifier.fireTestStarted(EachTestNotifier.java:42)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:323)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.Suite.runChild(Suite.java:128)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.Suite.runChild(Suite.java:27)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
06-14 03:15:07.681: E/TestRunner(28814):    at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
06-14 03:15:07.681: E/TestRunner(28814):    at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
06-14 03:15:07.681: E/TestRunner(28814):    at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2106)
06-14 03:15:07.681: E/TestRunner(28814): ----- end exception -----

After this no more tests are run and firebase test lab times out the build.

Any suggestions on what to change to get all our tests to run?

Thanks in advance!

Answer 1

It turns out this is a bug in the Orchestration framework, and the current best solution is to simply not use Orchestration. I created an issue for this:

https://issuetracker.google.com/issues/135574731