CODEWITHSUNDEEP
javaperformancekoloboke
Gaurav Agarwal
Gaurav Agarwal

Reputation: 66

Performance issues when copying between Koloboke sets

due to the way the iteration is performed and new entries are added, If one iterates over one set and copies to another set, the performance is very slow. Consider the following code snippet:

        final int num = (int) (1024 * 1024 * 2.1);
        final HashLongSet set1 = HashLongSets.newMutableSet();
        for (int i = 0; i < num; i++) {
            final long oid = r.nextLong();
            set1.add(oid);
        }

        System.out.println("populated first set..");

        final HashLongSet set2 = HashLongSets.newMutableSet();
        final LongCursor cursor = set1.cursor();
        while (cursor.moveNext()) {
            set2.add(cursor.elem());
        }
        System.out.println("populated first set..");

Is there any way to accelerate the population of second set in this case? I understand that if I knew the expected set size upfront, I could have used it on second set construction and made things faster - but that is not always possible - I could have inserted some conditions in between that determined which output set the value needs to be inserted to, or thrown away completely.

Upvotes: 0

Views: 95

Answers (1)

Georg Leber
Georg Leber

Reputation: 3580

Is it faster if you create the second HashLongSet by using the first set as parameter in the creating method:

 final HashLongSet set2 = HashLongSets.newMutableSet(set1);

UPDATE
Depending on your comment, what if you do something like:

  1. Create as many HashLongSets you need (countSets) with an initialCapacity of (int) set1.size() / countSets
  2. Then run your loop for dividing your data of set1 onto the other sets. In every loop you have to check if the initialCapacity is reached and expand the corresponding HashLongSet with another initialCapacity: set2.ensureCapacity(set2.size() + initialCapacity)

Upvotes: 1

Related Questions