differences of performances between NSDictionary (obj-c) and HashMap (java)

Question

I made some perfomance tests with these two datastructures and these two different languages. The results are not what I've expected. I thought that obj-c program would be faster than the java one. My tests says that java TreeMap is faster than cocoa NSDictionary. the code used to test is that:
obj-c NSDictionary:

#import <Foundation/Foundation.h>
NSString * getRandomString();
int main (int argc, const char * argv[]) {
    NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
    NSMutableDictionary * dict = [NSMutableDictionary dictionary];
    unsigned long i;
    NSString * string1;
    NSString * string2;
    NSString * string3;
    NSString * lastString;
    //dictionary with 100'000 elements 
    srand(time(NULL));
    for (i=0;i<100000;i++){
        NSString * aString = getRandomString();
        [dict setObject:aString forKey:aString];
        if (i == 100) 
            string1 = aString;
        if (i == 1000)
            string2 = aString;
        if (i == 10000)
            string3 = aString;
        if (i == 100000-1)
            lastString = aString;
    }
    NSDate * now;

    now = [NSDate date];
    [dict objectForKey:string1];
    NSTimeInterval interval = [now timeIntervalSinceNow];
    NSLog(@"%f",interval *-1000);

    now = [NSDate date];
    [dict objectForKey:string2];
    interval = [now timeIntervalSinceNow];
    NSLog(@"%f",interval *-1000);


    now = [NSDate date];
    [dict objectForKey:string3];
    interval = [now timeIntervalSinceNow];
    NSLog(@"%f",interval *-1000);

    now = [NSDate date];
    [dict objectForKey:lastString];
    interval = [now timeIntervalSinceNow];
    NSLog(@"%f",interval *-1000);


    [pool drain];
    return 0;
}
NSString * getRandomString(){
    NSString * tmp = [NSString string];
for (int i = 0 ; i < 10;i++){
    tmp = [NSString stringWithFormat:@"%@%c",tmp,rand()%128];
}
return tmp;
}

The command-line output is this:

2011-07-12 13:11:48.299 TestBench[1178:a0f] 0.008047
2011-07-12 13:11:48.302 TestBench[1178:a0f] 0.005007
2011-07-12 13:11:48.302 TestBench[1178:a0f] 0.003040
2011-07-12 13:11:48.303 TestBench[1178:a0f] 0.003994

Java TreeSet:

import java.util.Date;
import java.util.TreeMap;


public class Main {
    public static void main(String[] args) {
        String string1="",string2="",string3="",lastString="";
        TreeMap<String,String> map = new TreeMap<String,String>();
        for (long i=0;i<100000;i++){
            String aString = getRandomString();
            map.put(aString, aString);
            if (i == 100) 
                string1 = aString;
            if (i == 1000)
                string2 = aString;
            if (i == 10000)
                string3 = aString;
            if (i == 100000-1)
                lastString = aString;
        }
        Date start,end;

        start = new Date();
        map.get(string1);
        end = new Date();
        System.out.println(end.getTime()-start.getTime());

        start = new Date();
        map.get(string2);
        end = new Date();
        System.out.println(end.getTime()-start.getTime());

        start = new Date();
        map.get(string3);
        end = new Date();
        System.out.println(end.getTime()-start.getTime());

        start = new Date();
        map.get(lastString);
        end = new Date();
        System.out.println(end.getTime()-start.getTime());
    }
    public static String getRandomString(){
        String toRet = "";
        for (int i=0;i<10;i++){
            toRet+=(char)(Math.random()*128);
        }
        return toRet;
    }

}

and the command line output is this:

0
0
0
0

obviously in milliseconds as for obj-c. Why is TreeMap so fast? or... Why is NSDictionary so slow? Can anyone explain that to me?? Sorry for my very bad english Thank you very much.

**ADDING QUESTION******** I made modifications to code like this:
obj-c

#import <Foundation/Foundation.h>
NSString * getRandomString();
int main (int argc, const char * argv[]) {
    NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
    NSMutableDictionary * dict = [NSMutableDictionary dictionary];
    long int i;
    NSString * string1;
    NSString * string2;
    NSString * string3;
    NSString * lastString;
    //dictionary with 100'000 elements 
    srand(time(NULL));
    double sum = 0;
    for (int j = 0 ;j<10;j++){
        NSDate * now;
        now = [NSDate date];
        for (i=0;i<100000;i++){
            NSString * aString = getRandomString();
            [dict setObject:aString forKey:aString];
            if (i == 100) 
                string1 = aString;
            if (i == 1000)
                string2 = aString;
            if (i == 10000)
                string3 = aString;
            if (i == 100000-1)
                lastString = aString;
        }
        NSLog(@"Finished adding elements in %f ms",[now timeIntervalSinceNow]*-1000);
        now = [NSDate date];
        for (int i = 0;i<1000000;i++)
            [dict objectForKey:string1];
        for (int i = 0;i<1000000;i++)
            [dict objectForKey:string2];
        for (int i = 0;i<1000000;i++)
            [dict objectForKey:string3];
        for (int i = 0;i<1000000;i++)
            [dict objectForKey:lastString];
        NSTimeInterval interval = [now timeIntervalSinceNow];
        sum+=interval;
    }
    NSLog(@"medium lookup time: %f ms",sum/10/4*-1000);
    [pool drain];
    return 0;
}
NSString * getRandomString(){
    NSString * tmp = [NSString string];
    for (int i = 0 ; i < 10;i++){
        tmp = [NSString stringWithFormat:@"%@%c",tmp,rand()%128];
    }
    return tmp;
}

the output:

2011-07-12 14:48:36.519 TestBench[974:a0f] Finished adding elements in 1950.287998 ms
2011-07-12 14:48:38.722 TestBench[974:a0f] Finished adding elements in 1899.537027 ms
2011-07-12 14:48:41.340 TestBench[974:a0f] Finished adding elements in 1939.461946 ms
2011-07-12 14:48:43.681 TestBench[974:a0f] Finished adding elements in 1991.870999 ms
2011-07-12 14:48:45.854 TestBench[974:a0f] Finished adding elements in 1857.455015 ms
2011-07-12 14:48:48.636 TestBench[974:a0f] Finished adding elements in 2205.457032 ms
2011-07-12 14:48:50.782 TestBench[974:a0f] Finished adding elements in 1866.232991 ms
2011-07-12 14:48:53.106 TestBench[974:a0f] Finished adding elements in 1847.414017 ms
2011-07-12 14:48:55.537 TestBench[974:a0f] Finished adding elements in 1982.506990 ms
2011-07-12 14:49:00.629 TestBench[974:a0f] Finished adding elements in 4536.152005 ms
2011-07-12 14:49:00.962 TestBench[974:a0f] medium lookup time: 107.704024 ms

Java

import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.TreeMap;


public class Main {

    /**
     * @param args
     */

    public static void main(String[] args) {
        String string1="",string2="",string3="",lastString="";
        Map<String,String> map = new HashMap<String,String>();
        long sum=0;
        for (int j=0;j<10;j++){
            Date start,end;
            start = new Date();
            for (long i=0;i<100000;i++){
                String aString = getRandomString();
                map.put(aString, aString);
                if (i == 100) 
                    string1 = aString;
                if (i == 1000)
                    string2 = aString;
                if (i == 10000)
                    string3 = aString;
                if (i == 100000-1)
                    lastString = aString;
            }
            end = new Date();
            System.out.println("Finished adding elements in "+(end.getTime()-start.getTime())+" ms");
            start = new Date();
            for (int i = 0;i<1000000;i++)
                map.get(string1);
            for (int i = 0;i<1000000;i++)
                map.get(string2);
            for (int i = 0;i<1000000;i++)
                map.get(string3);
            for (int i = 0;i<1000000;i++)
                map.get(lastString);
            end = new Date();
            sum+=end.getTime()-start.getTime();
        }
        System.out.println("medium lookup time: "+sum/10/4+" ms");
    }
    public static String getRandomString(){
        String toRet = "";
        for (int i=0;i<10;i++){
            toRet+=(char)(Math.random()*128);
        }
        return toRet;
    }

}

result for hashmap:

Finished adding elements in 314 ms
Finished adding elements in 275 ms
Finished adding elements in 263 ms
Finished adding elements in 285 ms
Finished adding elements in 309 ms
Finished adding elements in 284 ms
Finished adding elements in 270 ms
Finished adding elements in 395 ms
Finished adding elements in 320 ms
Finished adding elements in 1804 ms
medium lookup time: 8 ms

result for treemap

Finished adding elements in 400 ms
Finished adding elements in 430 ms
Finished adding elements in 474 ms
Finished adding elements in 581 ms
Finished adding elements in 562 ms
Finished adding elements in 599 ms
Finished adding elements in 654 ms
Finished adding elements in 625 ms
Finished adding elements in 638 ms
Finished adding elements in 1750 ms
medium lookup time: 194 ms

So i think NSDictionary is not made with hash function but with a tree. And why it takes so much time to add elements in NSDictionary? Is there a map in cocoa with similar performances to java hashset? thank you

Answer 1

Java is very good at optimising code which doesn't do anything. In this regard it is often much faster than C/C++. However when it comes to doing real work the difference is usually much smaller.

map.get() doesn't do anything but new Date() does and this is much slower than System.currentTimeMillis() If you want to time short operations you are better off using System.nanoTime() and calculating results which are not immediately discarded.

Additionally, it is worth running a test for about 2 seconds to get the best result.

---

import java.util.HashMap;
import java.util.Map;
import java.util.TreeMap;

public class Main {
    public static void main(String[] args) {
        testMap(new TreeMap<String, String>());
        testMap(new HashMap<String, String>());
    }

    private static void testMap(Map<String, String> map) {
        System.out.println("Using " + map.getClass().getSimpleName());

            String[] strings = new String[100000];
            for (int i = 0; i < 100000; i++)
                strings[i] = getRandomString();

            {
                long start = System.nanoTime();
                for (int i = 0; i < 100000; i++) {
                    String aString = strings[i];
                    map.put(aString, aString);
                }
                long time = System.nanoTime() - start;
                System.out.printf("... added elements in %.6f ms%n", time / 1e6);
            }
            String string1 = strings[100], string2 = strings[1000], string3 = strings[10000], string4 = strings[100000 - 1];
        final int runs = 3000000;
        {
            long start = System.nanoTime();
            for (int i = 0; i < runs; i++)
                string1 = map.get(string1);
            long time = System.nanoTime() - start;
            System.out.printf("... average get() time was %.6f ms%n", time / 1e6 / runs);
        }
        {
            long start = System.nanoTime();
            for (int i = 0; i < runs; i++)
                string2 = map.get(string2);
            long time = System.nanoTime() - start;
            System.out.printf("... average get() time was %.6f ms%n", time / 1e6 / runs);
        }
        {
            long start = System.nanoTime();
            for (int i = 0; i < runs; i++)
                string3 = map.get(string3);
            long time = System.nanoTime() - start;
            System.out.printf("... average get() time was %.6f ms%n", time / 1e6 / runs);
        }
        {
            long start = System.nanoTime();
            for (int i = 0; i < runs; i++)
                string4 = map.get(string4);
            long time = System.nanoTime() - start;
            System.out.printf("... average get() time was %.6f ms%n", time / 1e6 / runs);
        }
    }

    public static String getRandomString() {
        String toRet = "";
        for (int i = 0; i < 10; i++) {
            toRet += (char) (Math.random() * 128);
        }
        return toRet;
    }
}

prints

Using TreeMap
... added elements in 85.856512 ms
... average get() time was 0.000095 ms
... average get() time was 0.000121 ms
... average get() time was 0.000124 ms
... average get() time was 0.000119 ms
Using HashMap
... added elements in 20.189437 ms
... average get() time was 0.000016 ms
... average get() time was 0.000015 ms
... average get() time was 0.000012 ms
... average get() time was 0.000012 ms

Answer 2

You can't really compare the values, because in Java you have longs and in Cocoa you have doubles. If you would cast the NSTimeInterval (which is a double) to long, you'd get the exact same result.

Also, to get useful results, timing just one call of the method isn't enough. Make a loop and call it a few thousand times at least.