CODEWITHSUNDEEP
javajacoco
Sergii S.
Sergii S.

Reputation: 449

Java code coverage with Jacoco. Merge exec files collected from different application versions

I am trying to establish a process of collecting QA tests coverage and aggregating this information into a single report. We have a big team and code changes very frequently, so my main problem was related to impossibility to collect coverage from the single app version. According to the documentation Jococo should warn about all classes where execution data does not match and report them as not covered.

[WARN] Execution data for class com/application/package/ClassName does not match.

Related docs:

However, when I merged exec files collected from different releases of application (millions of lines of code in total and thousands changed lines) Jacoco reported WARNs only about four classes resulting in 12 lines of code. JAR files used for report were taken from the latest release that was part of merge.

So, I am just trying to understand how it is possible and whether I can trust this report or not?

Upvotes: 1

Views: 3260

Answers (2)

Sergii S.
Sergii S.

Reputation: 449

Just decided to visualize previous answer for better perception (percentage numbers are made up and serve illustration purposes).

Merge process

Upvotes: 2

Godin
Godin

Reputation: 10574

Consider following example.

Version 1 consists of following source files:

src/Example.java

class Example {
  public static void main(String[] args) {
    C.print(A.getPrefix() + B.getSuffix());
  }
}

src/A.java

class A {
  static String getPrefix() {
    return "Hello, ";
  }
}

src/B.java

class B {
  static String getSuffix() {
    return "World";
  }
}

src/C.java

class C {
  static void print(String msg) {
    if ("Hello, World".equals(msg)) {
      System.out.println(msg + "!");
    } else {
      System.out.println(msg);
    }
  }
}

Let's compile and execute version 1:

# javac src/A.java src/B.java src/C.java src/Example.java -d v1

# java \
    -javaagent:jacoco-0.8.4/lib/jacocoagent.jar=destfile=1.exec,sessionid=v1 \
    -cp v1 \
    Example
Hello, World!

Version 2:

src/Example.java modified

class Example {
  public static void main(String[] args) {
    C.print("Hello");
  }
}

src/A.java modified

class A {
  static String getPrefix() {
    return "";
  }
}

src/B.java and src/C.java not modified

Let's compile and execute version 2:

# javac src/A.java src/B.java src/C.java src/Example.java -d v2

# java \ 
    -javaagent:jacoco-0.8.4/lib/jacocoagent.jar=destfile=2.exec,sessionid=v2 \
    -cp v2 \
    Example
Hello

Note that Example.class and A.class are different, while B.class and C.class are the same in both versions:

# diff --report-identical-files v1/Example.class v2/Example.class
Binary files v1/Example.class and v2/Example.class differ

# diff --report-identical-files v1/A.class v2/A.class
Binary files v1/A.class and v2/A.class differ

# diff --report-identical-files v1/B.class v2/B.class
Files v1/B.class and v2/B.class are identical

# diff --report-identical-files v1/C.class v2/C.class
Files v1/C.class and v2/C.class are identical

And so ids computed for these class files:

# java -jar jacoco-0.8.4/lib/jacococli.jar classinfo v1
  INST   BRAN   LINE   METH   CXTY   ELEMENT
     8      0      3      2      2   class 0xa170badd641f5a31 Example
     5      0      2      2      2   class 0x45b9146c94e31f23 B
     5      0      2      2      2   class 0xb8f01b5012761c26 A
    16      2      5      2      3   class 0xaf857eca353b9073 C

# java -jar jacoco-0.8.4/lib/jacococli.jar classinfo v2
  INST   BRAN   LINE   METH   CXTY   ELEMENT
     6      0      3      2      2   class 0x5915f0accdd77c81 Example
     5      0      2      2      2   class 0x45b9146c94e31f23 B
     5      0      2      2      2   class 0xa529ea9ab9745b77 A
    16      2      5      2      3   class 0xaf857eca353b9073 C

And so ids recorded in execution data:

# java -jar jacoco-0.8.4/lib/jacococli.jar execinfo 1.exec
[INFO] Loading exec file 1.exec.
CLASS ID         HITS/PROBES   CLASS NAME
Session "v1": Fri Jul 05 20:39:50 CEST 2019 - Fri Jul 05 20:39:50 CEST 2019
b8f01b5012761c26    1 of   2   A
a170badd641f5a31    1 of   2   Example
45b9146c94e31f23    1 of   2   B
af857eca353b9073    3 of   5   C

# java -jar jacoco-0.8.4/lib/jacococli.jar execinfo 2.exec
[INFO] Loading exec file 2.exec.
CLASS ID         HITS/PROBES   CLASS NAME
Session "v2": Fri Jul 05 20:39:50 CEST 2019 - Fri Jul 05 20:39:50 CEST 2019
af857eca353b9073    2 of   5   C
5915f0accdd77c81    1 of   2   Example

Let's merge, what combines execution data for classes with same name and same id:

# java -jar jacoco-0.8.4/lib/jacococli.jar merge 1.exec 2.exec --destfile merged.exec
[INFO] Loading execution data file /private/tmp/j/1.exec.
[INFO] Loading execution data file /private/tmp/j/2.exec.
[INFO] Writing execution data to /private/tmp/j/merged.exec.

# java -jar jacoco-0.8.4/lib/jacococli.jar execinfo merged.exec
[INFO] Loading exec file merged.exec.
CLASS ID         HITS/PROBES   CLASS NAME
Session "v1": Fri Jul 05 20:39:50 CEST 2019 - Fri Jul 05 20:39:50 CEST 2019
Session "v2": Fri Jul 05 20:39:50 CEST 2019 - Fri Jul 05 20:39:50 CEST 2019
b8f01b5012761c26    1 of   2   A
a170badd641f5a31    1 of   2   Example
45b9146c94e31f23    1 of   2   B
af857eca353b9073    4 of   5   C
5915f0accdd77c81    1 of   2   Example

Let's generate report using merged execution data and class files of version 2:

# java \
    -jar jacoco-0.8.4/lib/jacococli.jar \
    report merged.exec \
    --classfiles v2 \
    --sourcefiles src \
    --html report
[INFO] Loading execution data file /private/tmp/j/merged.exec.
[WARN] Some classes do not match with execution data.
[WARN] For report generation the same class files must be used as at runtime.
[WARN] Execution data for class A does not match.
[INFO] Analyzing 4 classes.

For src/Example.java report will show data about execution of version 2, because id for v2/Example.class is 5915f0accdd77c81:

Example.java

For src/A.java report won't show anything, because in merged.exec there is no data that corresponds to id of v2/A.class that is a529ea9ab9745b77:

A.java

With a message similar to warning during generation of report

A class

For src/B.java report will show data about execution of version 1, because in merged.exec there is data from 1.exec that corresponds to id of v2/B.class - 45b9146c94e31f23:

B.java

For src/C.java report will show combined data about execution of both versions, because in merged.exec there is data from both 1.exec and 2.exec that corresponds to id of v2/C.class - af857eca353b9073:

C.java

Above report is correct in a sense that it absolutely correctly represents merge of two executions in respect to the individual class files that were provided for generation of report:

  • v2/Example.class was executed
  • v2/A.class was not executed
  • B.class was executed in version 1
  • both branches in C.class were executed - one in version 1, another in version 2

Without usage of class ids report will be absolutely incorrect without ability to detect this

  • v2/A.class will be considered as executed, whereas this never happened
  • third line in A.java will be shown as executed, whereas this never happened

However in respect to all classes together above report does not represent final version, because in final version

  • B.class will never be executed
  • only one branch in C.class will be executed

Upvotes: 5

Related Questions