Why am I unable to find an easy way to run instrumentation tests on release build type APKs?

Question

I am trying to run instrumentation tests on Android when its build variant is release instead of debug. The idea was automate tests and run them after proguard/R8 has shrunk the code in order to ensure that the final APK include everything covering our use-cases and nothing important has been removed. But I am unable to find an easy way. I am currently under the impression that the build tools are not designed to work this way. But I cannot understand the reason. What am I missing?

I tried to do it in different ways (none of them worked).

1. Just creating a new Android project in Android Studio (mine is version 3.4.2) with an empty Activity, an Espresso test is provided. If I run it in debug, it seems to work as expected. When I change the build variant to release, all references to org.junit cannot be resolved. I cannot understand the reason, so the compilation is failing. These are the dependencies on the app/build.gradle

   dependencies {
       implementation fileTree(dir: 'libs', include: ['*.jar'])
       implementation 'androidx.appcompat:appcompat:1.0.2'
       implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
       testImplementation 'junit:junit:4.12'
       androidTestImplementation 'androidx.test:runner:1.2.0'
       androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
   }
   

2. Executing the Gradle task assemble, both debug and release build variant APKs are generated. However if I run assembleAndroidTest, only the debug one is created. In fact, assembleReleaseAndroidTest task does not exist (why?).

3. Using adb, I can install any of the assembled APKs and run the instrumentation using am. But I cannot run the instrumentation when the release build variant app is installed, because the certificate signatures does not match.

   adb install app-release.apk
   adb install app-debug-androidTest.apk
   adb shell am instrument <test-package>
   

I only found one solution to run the test for the release version, but it is complex. In order to achieve it I have to:

1. Create 2 modules, each being a different application. For simplicity, let assign com.example.myapp as the application id for the production code app, and com.example.myapp.test to the second app, that will be actually the test.

2. In the test app, I include the following AndroidManifest:

   <manifest xmlns:android="http://schemas.android.com/apk/res/android"
             package="com.example.myapp.test">
   
       <instrumentation
           android:name="com.example.myapp.test.Instrumentation"
           android:targetPackage="com.example.myapp"/>
   
   </manifest>
   

3. I include an Instrumentation implementation matching the name within the test module (I guess I could also point to the Espresso test runner as well instead of creating a custom one, but I did not tried it yet).

   package com.example.myapp.test;
   
   import android.os.Bundle;
   import static android.app.Activity.RESULT_OK;
   
   public final class Instrumentation extends android.app.Instrumentation {
   
       @Override
       public void onCreate(Bundle arguments) {
           super.onCreate(arguments);
           start();
       }
   
       @Override
       public void onStart() {
           super.onStart();
           performTest();
           finish(RESULT_OK, new Bundle());
       }
   
       private void performTest() {
           // Implement some tests
       }
   }
   

4. Include a suitable build.gradle in the test module where both debug and release are signed with the same key used in the app release. Something like:

   buildTypes {
       debug {
           signingConfig signingConfigs.targetSign
       }
   
       release {
           signingConfig signingConfigs.targetSign
       }
   }
   

5. Build and install both resulting APKs in the device (the app in release build variant, and the test in debug, both are now sharing the same key).

6. Run the instrumentation from command line:

   adb shell am instrument com.example.myapp.test
   

And it seems to be working as expected, even if the solution is complex.

The main question here would be if there is any sensible reason (that I am obviously missing) to prevent developers on running tests on the release build variant APK and force them to do it in the debug one?

On the other side, does anyone know a better/simpler solution to achieve the same?

Answer 1

You can set your testBuildType to release. For example in your app module set:

android {
  testBuildType = "release"
}

Then when you run

./gradlew app:assembleAndroidTest

This will generate app-release-androidTest.apk for you.

https://developer.android.com/studio/build/gradle-tips#change-the-test-build-type