CODEWITHSUNDEEP
javamemorymemory-managementjvm
Ryan Her
Ryan Her

Reputation: 1117

Measuring and monitoring the size of large HashMap

What are the best options of measuring or at least somewhat accurately estimating the total memory consumption by a hash map holding custom objects?

Background:

Our system has multiple in-memory stores (ConcurrentHashMap's) that holds 1K-200K objects of different types. We'd like to periodically monitor the memory footprint of these.

Constraints/Expectations:

We cannot afford frequent GC or full iteration of these hash maps that would have negative impact on the latency. As far as I know, there's no sizeof() in java, so we are okay with estimation.

Considering these options:

Have a thread that reports the memory usage # for each hashmap by:

  1. Measure the size of object of type X using something like classmexer and multiply it by the size of hashmap. This ignores memory used by hashmap itself assuming that it's relatively small compared to the its content.
  2. Create another hashmap, then add N objects of the type, measure the size of this, then extrapolate the number to approximate the full size.
  3. Externally measure, then feed this to the map. Thinking of jmap or similar, but won't this be very heavy operation as it measures everything in jvm?
  4. Take delta in heap size as we add objects to the hashmap. Not sure if this is feasible as it's multi-threaded
  5. Any other awesome option?

Upvotes: 4

Views: 2064

Answers (2)

Eugene
Eugene

Reputation: 120848

First of all, there is this great article from Alexey Shipilev (a former JVM engineer at Oracle, now at Redhat) that explain that this is not that easy.

Every Object in Java has two headers, their sizes can depend on the platform or on the configuration that jvm is started with (UseCompressedOops).

Then, there is the padding and alignment between fields and Objects them-selves (8 bytes alignment). Then, there is space that a JVM simply does not show because it can't or does have need to.

All these things make it a bit un-trivial to compute how much size will a certain Object have in heap; fortunately JOL exists. It even has lots of samples too... Here is one small example, suppose you have such a class:

static class MyValue {

    private final int left;
    private final String right;

    public MyValue(int left, String right) {
        this.left = left;
        this.right = right;
    }
}

And you create a HashMap:

    Map<String, MyValue> map = new HashMap<>();
    System.out.println("empty map = " + GraphLayout.parseInstance(map).totalSize());

    MyValue one = new MyValue(1, "one");
    System.out.println("one = " + GraphLayout.parseInstance(one).totalSize());

    map.put("one", one);
    System.out.println("map after one = " + GraphLayout.parseInstance(map).totalSize());

    MyValue two = new MyValue(1, "two");
    map.put("two", two);
    System.out.println("map after two = " + 
    GraphLayout.parseInstance(map).totalSize());

Upvotes: 3

Sambit
Sambit

Reputation: 8001

If my understanding is correct, you want to know the size of the concurrent HashMap. I think, the following steps may help.

  1. Create a background thread (not foreground).
  2. Serialize the concurrent HashMap using ByteArrayOutputStream.
  3. Get the byte array using toByteArray().
  4. Finally calculate the size in kb/mb.
  5. Log the size using Logger.

Other option is use profiler like JProfiler to see the object graph and its size.

Upvotes: 1

Related Questions