CODEWITHSUNDEEP
javadatasetcomparison
bluelhf
bluelhf

Reputation: 101

Compare values in large dataset with ones in small dataset quickly in Java

Goal

I'm attempting to create a Skyblock-like island plugin for Minecraft, and I'd like to be able to calculate an "island level" for every player's own three-dimensional island area.

Situation

I have a small hash map that gives me a decimal value for some subset of the possible combinations between the types of blocks and the states that they can be in.

The dataset of blocks is large: there are approximately 16777216 blocks, and I need to calculate the sum of each of their "values", as given by the aforementioned map.

A naive (and very slow) implementation in "Pseudo-Java"

double total = 0;
for (BlockData block : blocks) {
    for (Entry<Key, Double> entry : map) {
        Key key = entry.getKey();

        // Type check
        if (!key.getString().equals(block.getString()) continue;
        
        // States check (Only ones explicitly defined by entry must match)
        States blockStates = block.getStates();
        States keyStates = key.getStates();
        for (Entry<String, String> state : keyStates) {
            if (!state.getValue().equals(blockStates.get(state.getKey()))
                continue;
        }

        total += entry.getValue();
    }
}

How could I implement the level calculation efficiently?

P.S. Delta encoding isn't viable in this environment, since I can't listen to a block's setter due to API restrictions and obfuscation.

Upvotes: 0

Views: 226

Answers (1)

bluelhf
bluelhf

Reputation: 101

I've managed to trim this down to sub-1 second times on the first run, even for 67 million blocks. I figured I'd share my solution here.

Naive approach

My first attempt was to simply run the naive approach on multiple threads (equal to or 2x the amount of logical processors available to the program). This was slow, taking about 12 seconds or so even with a small amount of blocks to process.

Finding the bottlenecks

IntelliJ IDEA, the IDE I'm using, has a great tool pre-installed called Java Flight Recorder. Particularly, the call tree was immensely helpful in finding and eradicating bottlenecks.

The call tree lists the percentage of program time spent on any specific method calls. It's a nice, easy-to-read view and helps find bottlenecks quickly.

For my specific case, the block data objects were provided by an external API that did not have a way to read specific states. Initially, I tried parsing the string representation of the block data, but this turned out to be a terrible idea from a performance perspective. My solution was to write a wrapper class for the block data, QuickBlockData, which had a customised equals and hashCode method.

The equals method accessed the external API's internal block data instead of the one presented publicly by the API. It only compared state keys that were present in both block data objects.

The hashCode method could not utilise any states without breaking the hash code contract, so it just returned the type enum of the block data instead of looking at any states.

FastUtil

I decided on FastUtil's Object2DoubleOpenHashMap for my value map, since it most closely represented the data structure I wanted to have, and also provided a reasonable performance improvement over Java's own hash map implementations.

Instead of using Object2DoubleOpenHashMap directly, I extended in a ValueMap class, which allowed me to run some preemptive checks in the getter, like checking the type of the input against a HashSet of types that are contained in the map.

Parallelisation and chunks

With the same API that provides the block data, it is possible to load 16x16 columns of blocks, chunks, asynchronously, and then parse them individually. This was incredibly useful, as it meant I could process each chunk as it was loaded.

The API also provided chunk snapshots, which were thread-safe objects that gave some data about the chunk at the time of their creation. Critically, they provided a method to find the highest block in a specific block column inside the chunk, which served as an easy way to cut down on the blocks that I need to process.

Summary

I used:

  • A wrapper class for the block data, with a performant equals method that did what I wanted (although it forced me to use a terrible hash code)
  • A ValueMap class, which extended it.unimi.dsi.fastutil.objects.Object2DoubleOpenHashMap, allowing me to get the value for some block data quickly, with a customisable default value, and with preemptive checks for the type of the block.
  • Parallel computation and chunking provided by an external API, with an atomic double to count the total value of an area.

Upvotes: 1

Related Questions