Looking for ideas on how to improve groovy performance in the specific test

Question

I've encountered some rather significant performance issues while converting an existing batch process from Java to Groovy. The existing batch process written in Java runs periodically reading the data from different data sources and performing some data transformation. What has been discovered is a significant performance degradation with unexpectedly high gap of 10+ times after converting Java code to Groovy.

The code at https://github.com/nicolas-martinez/grava-speed-test is a simplified example that shows one of the problems discovered with a simple loop and filtering using collection closures. It's setup as Maven project which can be easily cloned locally and executed.

Below is the highlight of the Groovy code:

    List items = (0..length).collect()
    List even = items.findAll { item -> item > 0 && item.longValue() % 2 == 0 }

and the Java code:

    List<Long> items = new ArrayList(length);
    for (int i = 0; i < length; i++) {
        items.add(Long.valueOf(i + 1));
    }

    List<Long> even = new ArrayList<Long>();
    for(Long item : items){
        if (item > 0 && item % 2 == 0) {
            even.add(item);
        }
    }

The results of the test are 342ms for Groovy and under 30ms for Java:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running tst.speedtest.GroovyFilterTest
testFilter: 500000 elapsed: 342
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.637 sec
Running tst.speedtest.JavaFilterTest
testFilterUsingInterface: 500000 elapsed: 29
testFilter: 500000 elapsed: 27
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 sec

Please let me know if you have a suggestion on how to improve Groovy performance. Our team is considering moving to Groovy because of some advanced features it offers but it's hard to justify it because of such a high gap in performance we have encountered so far.

Below is my hardware profile as reported by system_profiler SPHardwareDataType:

Hardware Overview:

  Model Name: MacBook Pro
  Model Identifier: MacBookPro11,3
  Processor Name: Intel Core i7
  Processor Speed: 2.5 GHz
  Number of Processors: 1
  Total Number of Cores: 4
  L2 Cache (per Core): 256 KB
  L3 Cache: 6 MB
  Memory: 16 GB
  Boot ROM Version: MBP112.0138.B11
  SMC Version (system): 2.19f12

And here is the Java version:

java version "1.7.0_72"
Java(TM) SE Runtime Environment (build 1.7.0_72-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode)

The Groovy version is 2.3.7 as defined in pom.xml.

UPDATE.

1. Made suggested modifications to Groovy code:

   List items = (0..length)
   List even = items.findAll { int item -> item > 0 && item % 2 == 0 }
   

2. Added repeat of the test method call to warm up the test

I ran ./speed-test.sh which runs groovy and java tests separately. The startup of the jvm was never included in the tests.

Below are the best results that I was able to see running the same method 10 times within the same jvm process allowing for the warm up:

/speed-test.sh
Java test
Java testUsingInterface: 500000 elapsed: 44
Java testUsingInterface: 500000 elapsed: 43
Java testUsingInterface: 500000 elapsed: 28
Java testUsingInterface: 500000 elapsed: 11
Java testUsingInterface: 500000 elapsed: 31
Java testUsingInterface: 500000 elapsed: 10
Java testUsingInterface: 500000 elapsed: 9
Java testUsingInterface: 500000 elapsed: 11
Java testUsingInterface: 500000 elapsed: 19
Java testUsingInterface: 500000 elapsed: 19
JavaTest: for testSize=1000000 and repeat=10 total elapsed: 226

Groovy Test
GroovyTest: 500000 elapsed: 199
GroovyTest: 500000 elapsed: 76
GroovyTest: 500000 elapsed: 91
GroovyTest: 500000 elapsed: 80
GroovyTest: 500000 elapsed: 58
GroovyTest: 500000 elapsed: 83
GroovyTest: 500000 elapsed: 91
GroovyTest: 500000 elapsed: 58
GroovyTest: 500000 elapsed: 58
GroovyTest: 500000 elapsed: 67
GroovyTest: for testSize=1000000 and repeat=10 total elapsed: 1073

As @blackdrag indicated, Groovy takes longer to warm up. After the warm up cycle, it still takes ~5 times longer to execute (even if the initial warm up cycle is excluded). The updated code is on the branch feature/option-1 if anyone wants to review it.

Answer 1

I have roughly those guide lines for performance tests in general:

• Ensure you measure more than 1s, to avoid timing errors from the computer clock.
• Always give the Groovy and Java versions as well as the computer specs to be able to compare things
• Always have a long enough warm-up phase
• Don't run multiple micro-benchmarks together
• Measuring multiple iterations to get an average is to be preferred over measuring a single iteration.

Since performance testing is a really really wide field, and with micro-benchmarks especially (since you may not test what you think you test). I am also giving some hints for your case, but going into all details is probably just too much for this platform.

First of all, you should think about what you want to test. Is it peak performance, average performance or initial performance? With or without startup cost? As you may know the JVM uses partially interpreted and partially runtime compiled code. When and how the interpreted code is transformed into compiled code depends for example on the number of iterations the method containing the code has been called (and the types used, code size and a lot of other things)

If you go by peak performance, then junit is not the right tool. JMH for example would be better here, since it handles not only warm-up times, but also stops in a stabilization phase.

There is for example a lot of class loading done the first time the groovy runtime is used, in which the default groovy methods are loaded. This alone can easily take half of the time you observed, and effectively no code had been executed yet at that point.

@CompileStatic can help, but we can not yet always prevent the loading of the groovy meta class system. So even with that, there might be this warm-up cost. Not to mention the JVM itself has a warm-up cost.

The original code needs on my computer for the original code about 752ms. Adding a warm-up of only one single iteration this goes down to 14-20ms.

And there are some logical disconnects as well... List items = (0..length).collect() the range is already a list, so there is no need to call collect here. This will only produce a new list by copying over every element. And collect() is not going to transform the elements into longs. Since we are dealing with Integer objects, there is really no need to convert to long, by calling longValue(). Correcting those two things alone would already reduce the execution time to half (on my computer at least, and without a warm-up phase). But the warm-up phase is really doing a difference here. So with warm-up and those corrections I get to 10ms already (50k elements). To compare it, the Java version needs here 5ms. Which I find already to short to test it. So if I redo the test with 1 million elements I see 73ms (Java) vs. 200ms (Groovy). Of course I changed the Java version to use Integer as well.

Adding a type hint to enable primitive optimizations List even = items.findAll {int item -> item > 0 && item % 2 == 0 } will improve performance to about 120ms.

In other case @CompileStatic or running using invokedynamic (performance of the invokedynamic version depends extremely on the JVM version!) may also help improving performance. They are not going to do much in this test I assume.