Performance evaluation of synchronized Hashmap vs ConcurrentHashMap in Java

Question

I have written a small test based on this https://dzone.com/articles/java-7-hashmap-vs to test out which approach performs better

threadSafeMap2 = new HashMap<String, Integer>(2);
threadSafeMap2 = Collections.synchronizedMap(threadSafeMap2);

or

threadSafeMap3 = new ConcurrentHashMap<String, Integer>(2)

Here is my junit test:

package com.bm.framework.concurrent;

import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import org.junit.Test;

class WorkerThread implements Runnable 
{

    static final int WORKER_ITERATION_COUNT = 500000;
    private Map<String, Integer> map = null;

    public WorkerThread(Map<String, Integer> assignedMap) 
    {

          this.map = assignedMap;
    }

    @Override
    public void run() 
    {
          for (int i=0; i < WORKER_ITERATION_COUNT; i++) 
          {

                 // Return 2 integers between 1-1000000 inclusive

                 Integer newInteger1 = (int) Math.ceil(Math.random() * 1000000);

                 Integer newInteger2 = (int) Math.ceil(Math.random() * 1000000);                    

                 // 1. Attempt to retrieve a random Integer element

                 Integer retrievedInteger = map.get(String.valueOf(newInteger1));

                 // 2. Attempt to insert a random Integer element

                 map.put(String.valueOf(newInteger2), newInteger2);                
          }
    }
}



public class BmfThreadPoolTest 
{
    private static final int NB_THREADS = 3;
    private static final int NB_TEST_ITERATIONS = 50;


    private static Map<String, Integer> nonThreadSafeMap = null;
    private static Map<String, Integer> threadSafeMap2 = null;
    private static Map<String, Integer> threadSafeMap3 = null;


    @Test
    public void testMapPerformance() throws InterruptedException 
    {
        // this is a test between two styles we see
        // one is to use Collections.synchronizedMap()
        // and the other one is to use directly ConcurrentHashMap which 
        // one is faster?
        // Plain old HashMap (since JDK 1.2)
        nonThreadSafeMap = new HashMap<String, Integer>(2);

        // Fully synchronized HashMap
        threadSafeMap2 = new HashMap<String, Integer>(2);
        threadSafeMap2 = Collections.synchronizedMap(threadSafeMap2);

        // ConcurrentHashMap (since JDK 1.5)
        threadSafeMap3 = new ConcurrentHashMap<String, Integer>(2);
        System.out.println("ConcurrentHashMap");
        beginTest(threadSafeMap3);
        // the second one is always performing poor no matter whether it is hashmap or concurrenthashmap why????
        System.out.println("Collections.synchronizedMap()");
        beginTest(threadSafeMap2);

    }

    private void beginTest(final Map<String, Integer> assignedMapForTest)
    {
        for (int i=0; i<NB_TEST_ITERATIONS; i++) 
        {
            long timeBefore = System.currentTimeMillis();
            long timeAfter = 0;

            Float totalProcessingTime = null;

            ExecutorService executor = Executors.newFixedThreadPool(NB_THREADS);

            for (int j = 0; j < NB_THREADS; j++) 
            {

                   /** Assign the Map at your convenience **/

                   Runnable worker = new WorkerThread(assignedMapForTest);
                   executor.execute(worker);              

            }

            // This will make the executor accept no new threads

            // and finish all existing threads in the queue

            executor.shutdown();

            // Wait until all threads are finish

            while (!executor.isTerminated()) 
            {

            }

            timeAfter = System.currentTimeMillis();

            totalProcessingTime = new Float( (float) (timeAfter - timeBefore) / (float) 1000);

            System.out.println("All threads completed in "+totalProcessingTime+" seconds");
        }
    }   
}

The problem is among the two beginTest()call, the second one always performs bad, i.e., if I run like this

beginTest(threadSafeMap3);
beginTest(threadSafeMap2); 

the last one takes longer to complete pointing to the fact that ConcurrentHashMap is faster. Again if I swap the order like this

beginTest(threadSafeMap2);
beginTest(threadSafeMap3); 

the last one takes longer to complete pointing to the fact that ConcurrentHashMap is slower. Why am I getting conflicting results based on the order the maps are used in the test?

If I comment out one of them and run the test in two separate runs (one for synchronized hash map and one for ConcurrentHashMap), then I always get consistent result with ConcurrentHashMap as the winner.

Answer 1

Simple explaination is provided here - https://javaho.wordpress.com/2012/12/11/concurrent-hashmap-vs-synchronized-hashmap/

Answer 2

1. You should be testing this with your actual usage pattern: quite likely the overhead of maps is negligible compared to the rest of your program - and even if it isn't, the way the maps are used may matter. Therefore, it often better to measure your entire application than to micro benchmark individual components.
2. In a micro-benchmark, you should warm up the code under test, so no code path is taken first during the timing phase, to ensure that you are not benchmarking interpreted code or the just-in-time compilation itself
3. You should separate the tests into different methods. This would permit the garbage collector to free the memory consumed by the first test run, so both tests run with the same heap size and garbage collection overhead. (In your code, the JVM may not realize that the first map is no longer used once the second test has begun ...)

Background reading: How do I write a correct micro-benchmark in Java?