Make Android jenkins build fail if tests fail

Question

I have seen several post regarding making a build in Jenkins fail if the unit test execution fail (e.g. this one). It turns out that by default Jenkins reports builds with failing tests as unstable and some people do not like that. This, however, will be perfectly fine for me. I just want to be able to easily differentiate builds with passing tests from such with failing tests.

And here is the catch: I am developing for Android so my build is configured following this page. Basically the tests are run with the following command:

ant all clean emma debug install test

As result coverage report is generated and published in Jenkins.

All posts I have read about configuring the Jenkins result according to tests results were dealing with ant task manipulation. However, if we look at the android build.xml the Android tests are run with adb command: adb shell am instrument .... I don't know how to configure this command to print the tests results. It can be configured to print the coverage report. I have already done that, but was never able to make the build fail according to the coverage report.

I hope somebody else also faced the same problem and managed to solve it. Any guidance will be most appreciated.

Answer 1

This is possible, configure the server using the following documentation:

https://wiki.jenkins-ci.org/display/JENKINS/Android+Emulator+Plugin

See "Parsing monkey output" in the link.

Addition, Jenkins can test instrumentation, below have a link of a good place to start in the learning of instrumentation:

http://developer.android.com/resources/tutorials/testing/helloandroid_test.html

I hope it is useful.

Answer 2

Consider using 'the-missing-android-xml-junit-test-runner' - the idea being to produce jUnit reports that Jenkins understands. Disclaimer: I do have no personal experience with adb. However, I do know from experience that Jenkins deals with properly generated jUnit reports very gracefully.