Storing many objects using Java

Question

I'm reaching out to the community here to understand if there is a way to store many objects in a Java map structure (> 500K). I know at some point my application will exceed its memory footprint if I use the Java Collections API and I'm looking for other solutions (outside of distributed caching). My intention is to store something to the effect of Map<String, List<String>>.

Requirements:

• Structure will be immutable (one time load into the map). Only read access.
• Access to the map needs to be fairly quick but not looking for low latency retrievals, more concerned about storing and retaining these objects in memory.

Does anyone know of such a library I can utilize that can achieve this or has anyone ever come across a solution in which they were required to load many objects in memory in Java? I'd be very interested in hearing your feedback.

Thanks.

Answer 1

MVStore feature of H2 Database Engine

While I have not tried it, the H2 Database Engine offers a specialized key-value store besides its relational database engine. This store is called MVStore. While used internally by H2, the documentations suggest you may use it directly for key-value storage.

MVStore lets you interact through the java.util.Map interface, without using JDBC or SQL.

Example usage, taken from the doc:

import org.h2.mvstore.*;

// open the store (in-memory if fileName is null)
MVStore s = MVStore.open(fileName);

// create/get the map named "data"
MVMap<Integer, String> map = s.openMap("data");

// add and read some data
map.put(1, "Hello World");
System.out.println(map.get(1));

// close the store (this will persist changes)
s.close();

Read the thorough documentation to learn about its performance profile, concurrency, and ability to cache in-memory while also persisting to storage.

Answer 2

Have you considered a read-only database, such as a java cdb: http://www.strangegizmo.com/products/sg-cdb/

Answer 3

I'm with skaffman. In addition to the overflow to disk EhCache offers you to place the cache in-process but off-heap. If your cache is really big, this may have a very positive impact on performance since it reduces stress on the GC.

This specific feature however must be payed for, but otherwise EhCache is free.

Instead of EhCache, there are a couple of other caches in Java that offer similar or sometimes even more advanced options like Infinispan or OsCache.

Answer 4

EhCache would be perfect for this. At its most basic, it offers a key-value map, with optional overflow to disk, and optional persistence over JVM restarts. It will keep elements in memory that are most frequently used.