CODEWITHSUNDEEP
javaarraysperformancebinary-databitset
Andre
Andre

Reputation: 219

Huge binary matrix (logical AND on array of bitsets) Java performance

We have a java service that computes some logical operations on a huge binary matrix (10 000 x 10 000). This matrix is array of bitsets. The most important operation is an intersection (logical AND) betwen a given bitset and each bitset in the array. We are using OpenBitset and it shows quite good results (at least better than java.util.BitSet). Data sparsity is moderate (could be many 0 or 1 in a row), bitset size is fixed.

The most important thing for us is fast response times (for now it's ~0.05 sec), so we would like to find ways for further improvements as the matrix and the quantity of requests are growing. There could be some algebraic methods or faster libraries for that.

We tried to use javaewah, but this library performed operations 10x times slower comparing to OpenBitset. There is comparision on the project's page, that shows that other bitset-compression libraries slower than Java BitSet.

Could you suggest some other methods or new ideas?

Upvotes: 2

Views: 952

Answers (2)

lisak
lisak

Reputation: 21961

If you don't mind using client-server solution, pilosa would be perfect for your use case.

  • bindings for java,python,go
  • groupBy support
  • time range support
  • huge matrix support
  • uses high-performance roaringbitmap
  • scales horizontally
  • helm chart https://github.com/pilosa/helm

Upvotes: 0

wkb
wkb

Reputation: 11

In my recent blog I discussed a "yet another" bitset implementation - with source code. Maybe you want to give it a try: http://www.censhare.com/en/aktuelles/censhare-labs/yet-another-compressed-bitset

Upvotes: 1

Related Questions