CODEWITHSUNDEEP
javaperformancejvm
Pavel Ryzhov
Pavel Ryzhov

Reputation: 5142

How do to perform a good performance comparations test?

To write a good comparations test test you have to run it several thousands (millions) times. It will level (in most cases) other programs' influence.

But if a JVM can influence on the results. For example:

First solution is:

    final StringBuilder stringBuilder = new StringBuilder();
    stringBuilder.append(getStrOne());
    stringBuilder.append(getStrTwo());
    final String result1 = stringBuilder.toString();

And second is:

    final String result2 = getStrOne() + getStrTwo();

I do not know which one is better because JVM can influence on the results. How to know which one is better?

UPDATE: I don't mean exactly that appending comporation test. I'm asking about such a hard to test situation.

Upvotes: 6

Views: 122

Answers (2)

Duncan Jones
Duncan Jones

Reputation: 69329

I recently performed some benchmark tests that relied on the excellent IBM article found here: http://www.ibm.com/developerworks/java/library/j-benchmark1/index.html.

The article describes many of the pitfalls that can affect accuracy of results, for example:

  • Runtime optimisation/re-compilation of your code.
  • Dead code elimination (i.e. unused results can cause your test code to be removed)
  • Garbage collection
  • Caching
  • ...

Finally, the article links to a site where a framework can be downloaded. The framework does a very good job of spooling up a test method, looking for evidence of recompilation and waiting for the execution time to settle.

Upvotes: 2

Thomas
Thomas

Reputation: 88707

In case of 2 strings, performance differences are negligible, but try the following:

String s = "";
for( int i = 0; i < 10000; i++ ) {
  s += i;
}

vs.

StringBuilder b = new StringBuilder();
for( int i = 0; i < 10000; i++ ) {
  b.append(i);
}

You'll find that the second loop is way faster. Why? Because string concatenation will create a new String object in every iteration, which wastes CPU cycles as well as memory.

I'll cite you:

To write a good comparations test test you have to test it several thousands (millions) times. It will level (in most cases) other programs' influence.

The same applies to tests within a single VM: test your code multiple times, use larger data, larger loops etc. Comparing only small portions doesn't make sense due to timing precision errors and other influences (e.g. garbage collection running in between).

Upvotes: 0

Related Questions