CODEWITHSUNDEEP
mavenjenkinsjunitmaven-surefire-plugin
supercalifragilistichespirali
supercalifragilistichespirali

Reputation: 1309

Jenkins, Junit and Flaky Test Handler plugin

I'm trying to get some visibility of flaky tests running in Jenkins.

In order to do so I've installed https://wiki.jenkins-ci.org/display/JENKINS/Flaky+Test+Handler+Plugin which seems to have this purpose.

All the requirements specified in the plugin page are met by my installation (Jenkins version, Junit plugin, etc...)

After having installed the plugin, the suggest to

check on “Publish JUnit flaky test reports” under “Additional test report features.

and their screenshot shows

enter image description here

My Maven Project configuration does not give me the option of specifying the Test report XMLs, and looks like this instead:

enter image description here

Although, when I run the Jenkins build for the configured project, all I get is an empty table:

enter image description here

The Build section of the project configuration is using the -Dsurefire.rerunFailingTestsCount=3 option.

I'm a bit confused whether the plugin is not behaving with my Jenkins installation or I'm missing something.

EDIT: I've been trying version 1.0.3 which contains the fix mentioned in one of the answers, and at the end of the job execution I get the following error:

ERROR: Publisher 'Publish JUnit test result report' aborted due to exception: 
java.io.IOException: Unable to serialize com.google.jenkins.flakyTestHandler.junit.FlakyTestResult@55c1d6e9
    at hudson.remoting.UserRequest.serialize(UserRequest.java:169)
    at hudson.remoting.UserRequest.perform(UserRequest.java:129)
    at hudson.remoting.UserRequest.perform(UserRequest.java:49)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to some.slave.com(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1413)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
    at hudson.remoting.Channel.call(Channel.java:778)
    at com.google.jenkins.flakyTestHandler.plugin.JUnitFlakyTestDataPublisher.getTestData(JUnitFlakyTestDataPublisher.java:49)
    at hudson.tasks.junit.TestDataPublisher.contributeTestData(TestDataPublisher.java:62)
    at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:166)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:75)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726)
    at hudson.model.Build$BuildExecution.post2(Build.java:185)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671)
    at hudson.model.Run.execute(Run.java:1766)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:408)
Caused by: java.io.NotSerializableException: com.google.jenkins.flakyTestHandler.junit.FlakyCaseResult$FlakyRunInformation
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at java.util.ArrayList.writeObject(ArrayList.java:762)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at java.util.ArrayList.writeObject(ArrayList.java:762)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at java.util.ArrayList.writeObject(ArrayList.java:762)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at hudson.remoting.UserRequest._serialize(UserRequest.java:158)
    at hudson.remoting.UserRequest.serialize(UserRequest.java:167)
    at hudson.remoting.UserRequest.perform(UserRequest.java:129)
    at hudson.remoting.UserRequest.perform(UserRequest.java:49)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

This seem to happen only when the job is executed on a slave, it seem to work fine on master.

EDIT 2: 1.0.4 fixed the issue above.

EDIT 3: Even though 1.0.4 fixed the serialisation issue, the plugin reporting still seem to be broken. In the following situation for example:

wrong reporting

The report of the project is saying that there is one flaky test with one Flakes, 0 Fails and 0 Passes. I would expect the 2 Passes, 0 Fails and 1 Flakes instead.

Am I missing something?

Upvotes: 2

Views: 3399

Answers (2)

Qingzhou Luo
Qingzhou Luo

Reputation: 23

If I remember correctly, we define a flaky test to be:

For the same code revision, it both fails and passes.

Therefore, if you never modify the code, even if you run it 100 times and the result is 50 passes and 50 failures, we still count it as one flake; if it passes all the 100 times, then it counts as one pass; otherwise if it fails all 100 times, it is one failure.

If you made some changes and submit another version, and then build upon on that, then it will be a new number.

I believe if you click the test, then you can see the detailed passes/fails, which is what you want?

see https://wiki.jenkins-ci.org/display/JENKINS/Flaky+Test+Handler+Plugin ​"​Aggregating Tests Statistics over Revisions​"​

Upvotes: 2

bezda
bezda

Reputation: 36

I had the same problem. Looking into master logs it turned out that the plugin doesn't expect builds to be executed on slaves. There's pull request which fixes the issue for me - https://github.com/jenkinsci/flaky-test-handler-plugin/pull/2

Upvotes: 2

Related Questions