Cobertura code coverage report in Jenkins seem to give the same statistics no matter of the extension of the tests

Question

Problem

My code coverage report created from my scripts and Cobertura seem to give me the same coverage no matter of how many tests that are run. Jenkins' Cobertura plug-in always show me the same graph, should I run all or none of the tests.

Problem description

I am a beginner to Jenkins, Cobertura and the whole of code coverage in large. I have a simple Java project (see Code: Program below) that runs a few simple tests in TestNG (see Code: Test below). A few scripts are handling the compilation and testing (see Code: Scripts below), scripts that are given to Jenkins in it's Build: Execute Shell in the following order:

clean.sh
compile.sh
test-coverage.sh
(test.sh)

Now, in test-coverage.sh, Cobertura instruments the program's source code (the Code: Program part), creates a coverage.xml that should (and seemingly does) contain the coverage for my code after the tests are run.

However, if I should remove a test (or all, it does not matter) and run it again, the coverage.xml seem to give me the same statistics as if I ran it with all tests intact.

What am I doing wrong? Have I missunderstood the concept? Should I also instrument my test classes?

Code: Program

The program is an automated, simple version of the FizzBuzz math game, where each player takes turns to count up, replacing each number that is evenly denomainated by a said denominator by another word (ex. "Fizz" but in this example "Cowabunga").

main.java

public class main {
    public static void main(String[] args){
        StartGame game = new StartGame();
        game.start();
    }
}

StartGame.java

public class StartGame {
    public static void start() {
        GameRules gr = new GameRules(3,5);
        gr.startGame();
    }
}

GameRules.java

public class GameRules {
    private int currentNumber;
    private int currentPlayer;
    private int playUntilThisNumber;
    private int dividerToSkip;
    private List playerlist;


    public GameRules(int numberOfPlayers, int divider){
        playerlist = new ArrayList();
        currentNumber=1;
        currentPlayer=3;
        playUntilThisNumber = 100;
        dividerToSkip = divider;
        fillPlayerList(numberOfPlayers);
    }

    public int getCurrentNumber() {
        return currentNumber;
    }

    public void increaseCurrentNumber() {
        this.currentNumber++;
    } 

    private void fillPlayerList(int numberOfPlayers){
        for(int i = 0 ; i < numberOfPlayers ; i++){
            Player p = new Player(this, i+1,dividerToSkip);
            playerlist.add(p);
        }
    }

    public void startGame(){
        while(currentNumber<=playUntilThisNumber){
            System.out.println(playerlist.get(currentPlayer++%playerlist.size()).play());
        }
    }
}

Player.java

public class Player {
    private GameRules rules;
    private int dividerToSkip;
    private int playerNumber;


    public Player(GameRules r, int playerNr, int divider){
        rules = r;
        dividerToSkip = divider;
        playerNumber = playerNr;
    }

    public String play() {
        String s;
        if(rules.getCurrentNumber()%dividerToSkip==0){
            s = "Player " + playerNumber + ": Cowabunga";
        }
        else {
            s = "Player " + playerNumber + ": " +rules.getCurrentNumber();
        }
        rules.increaseCurrentNumber();
        return s;
    }
}

Code: Tests

GameRuleTest.java

public class GameRulesTest {
        private int nrOfPLayers;
        private int divider;
        private GameRules gr;

        @BeforeMethod
        public void setUp(){
            nrOfPLayers = 35;
            divider = 13;
            gr = new GameRules(nrOfPLayers,divider);
        }

        @Test
        public void testCurrentNumber(){
            assert (gr.getCurrentNumber()>0);
        }

        @Test
        public void testIncreaseCurrentNumber() {
            int cn = gr.getCurrentNumber();
            gr.increaseCurrentNumber();
            assert(gr.getCurrentNumber()==cn+1);
        } 
    }

PlayerTest.java

public class PlayerTest {
    private int nrOfPLayers;
    private int divider;
    private GameRules gr;
    private Player player;

    @BeforeMethod
    public void setUp(){
        nrOfPLayers = 35;
        divider = 13;
        gr = new GameRules(nrOfPLayers,divider);
        player = new Player(gr,100,divider);
    }

    @Test
    public void testReturnString() {
        String res = player.play();

        assert "Player 100: 1".equals(res) : "Expected correct return string, got " + res;

        assertEquals("Player 100: 1", res);
    }
}

PerformanceTest.java

public class PerformanceTest {
    private StartGame game;

    @BeforeMethod
    public void setUp(){
        game = new StartGame();
    }

    @Test
    public void testExecutionTime(){
        long startTime = ManagementFactory.getThreadMXBean().getCurrentThreadCpuTime();
        game.start();
        long finnishTime = ManagementFactory.getThreadMXBean().getCurrentThreadCpuTime();
        System.out.println(startTime);
        System.out.println(finnishTime);
        assert (finnishTime-startTime<0.0000005);
    }
}

Code: Scripts

clean.sh

#!/bin/bash

find . -name '*.class' | xargs rm
rm -rf instrumented cobertura.ser cobertura_report test-output

compile.sh

#!/bin/bash
javac src/MyProject/*.java
javac -cp src/:testng-6.8/testng-6.8.jar test/*.java

test.sh

#!/bin/bash
java -cp testng-6.8/testng-6.8.jar:test:src org.testng.TestNG -verbose 2 test/test.xml

test-coverage.sh

#!/bin/bash

COBERTURA=cobertura-2.0.3
INSTRUMENTED=instrumented
REPORTDIR=cobertura_report

mkdir -p $INSTRUMENTED
mkdir -p $REPORTDIR

# Instrument the classes that we want to check coverage on
$COBERTURA/cobertura-instrument.sh src/MyProject/*.class --destination $INSTRUMENTED || exit

# Run the tests
java -cp $COBERTURA/$COBERTURA.jar:$INSTRUMENTED:testng-6.8/testng-6.8.jar:test org.testng.TestNG -verbose 2 test/test.xml

# Generate report
$COBERTURA/cobertura-report.sh --format xml --destination $REPORTDIR src

# Check coverage
#$COBERTURA/cobertura-check.sh --branch 0

test.xml

coverage.xml (As generated)

src

Saeldin · Accepted Answer

Mystery solved.

The instrumented classes get coverage for all reading done from the test classes. This means any call from these will result in (if but a bit) of coverage. In this case, I run a couple of unit tests and after having produced the code I wrote the PerformanceTest.java just to try out a higher level of testing.

Now, the problem was that the performance test ran the program to calculate execution time. This meant that almost the entire code of the program was run during this test, something that made the unit tests obsolete and what the unit tests were supposed to cover was covered anyway.

Sumary

Use smaller tests like unit tests or module tests to see your code coverage. Higher level of testing renders most of the smaller ones obsolete when it comes to see if a row of code was run or not.