CODEWITHSUNDEEP
javaperformancemathmemory-managementcollections
simlei
simlei

Reputation: 165

Lightweight Map implementation Java (little memory overhead)

I am currently writing some code in java meant to be a little framework for a project which revolves around a database with some billions of entries. I want to keep it high-level and the data retriueved from the database shoud be easily usable for statistic inference. I am resolved to use the Map interface in this project.

a core concept is mapping the attributes ("columns in the database") to values ("cells") when handling single datasets (with which I mean a columns in the database) for readable code: I use enum objects (named "Attribute") for the attribute types, which means mapping <Attribute, String>, because the data elements are all String (also not very large, maximum 40 characters or so). There are 15 columns, so there are 15 enums, and the maps will have only so much entries, or less.

So it appears, I will be having a very large number of Map objects floating around, at times, but with comparatively little payload (15-). My goal is to not make the memory explode due to the implementation memory overhead, compared to the actual payload. (Stretch goal: do the same with cpu usage ;] )

I was not really familiar with all the different implementations of Java Collections to date, and when the problem dawned at me today, I looked into my to-date all-time favorite 'HashMap', and was not happy with how much memory overhead there was declared. I am sure, that additonal to the standard implementations, there are a number of implementations not shipped with Java. Googling my case brought not up much of a result, So I am asking you:

Do you know a good implementation of Map for my use case (low entry count, low value size, enumerable keys, ...)

I hope I made my use case clear, and am anxious for your input =) Thanks a lot!

Stretch answer goal, absolutely optional and only if you got the time and knowledge: What other implementations of collections are suitable for:

Upvotes: 3

Views: 7907

Answers (3)

Philipp Wendler
Philipp Wendler

Reputation: 11433

Use EnumMap, this is the best map implementation if you have enums as key, for both performance and memory usage.

The trick is that this map implementation is the only one that that does not store the keys, it only needs a single array with the values (similar to an ArrayList of the values). There is only a little bit of overhead if there are keys that are not mapped to a value, but in most cases this won't be a problem because enums usually do not have too many instances.

Compared to HashMap, you additionally get a predictable iteration order for free.

Upvotes: 7

Tom Carchrae
Tom Carchrae

Reputation: 6486

Since you start off saying you want to store lots of data, eventually, you'll also want to access/modify that data. There are many high performance libraries out there.

Look at

When you find a bottleneck, you can switch to using a lower level API (more efficient)

You'll many more choices if look a bit more: What is the most efficient Java Collections library?

EDIT: if your strings are not unique, you could save significant amounts of memory using String.intern() : Is it good practice to use java.lang.String.intern()?

Upvotes: 5

Jilles van Gurp
Jilles van Gurp

Reputation: 8294

You can squeeze out a bit of memory with a simple map implementation that uses two array lists (keys and values). For larger maps, that is going to mean insertion and look up speeds become much slower because you have to scan the entire list. However, for small maps it is actually faster this way since you don't have to calculate any hashcodes and only have to look at a small number of entries.

If you need an implementation, take a look at my SimpleMap in my jsonj project: https://github.com/jillesvangurp/jsonj/blob/master/src/main/java/com/github/jsonj/SimpleMap.java

Upvotes: 3

Related Questions