CODEWITHSUNDEEP
scalascalding
harshsinghal
harshsinghal

Reputation: 3780

How do I use HyperLogLogMonoid from Algebird to carry out arbitrary intersections and unions

I'd like to aggregate a bunch of values that belong to a particular category into an HLL data structure so I can carry out intersections and unions later and count resulting cardinality of such computations.

I was able to get to the point where I can estimate cardinality for each group using the com.twitter.algebird.HyperLogLogAggregator

I need help using com.twitter.algebird.HyperLogLogMonoid to store as HLL and then later utilize to compute intersections/unions.

val lines_parsed = lines.map { line => parseBlueKaiLogEntry(line) }
# (uuid, [category id array])

val lines_parsed_flat = lines_parsed.flatMap { 
  case(uuid, category_list) => category_list.toList.map {
       category_id => (category_id, uuid) 
     }
}
# (category_id, uuid)

# Group by category
val lines_parsed_grped = lines_parsed_flat.groupBy { 
        case (cat_id, uuid) => cat_id 
      }

# Define HLL aggregator
val hll_uniq = HyperLogLogAggregator.sizeAggregator(bits=12).composePrepare[(String, String)]{case(cat_id, uuid) => uuid.toString.getBytes("UTF-8")}

# Aggregate using hll count
lines_parsed_grped.aggregate(hll_uniq).dump
# (category_id, count) - expected output

Now, I try to use HLL Monoid

# I now want to store as HLL and this is where I'm not sure what to do
# Create HLL Monoid
val hll = new HyperLogLogMonoid(bits = 12)

val lines_grped_hll = lines_parsed_grped.mapValues { case(cat_id:String, uuid:String ) =>  uuid}.values.map {v:String => hll.create(v.getBytes("UTF-8"))}

# Calling dump results in a lot more lines that I expect to see
lines_grped_hll.dump

What am I doing wring here ?

Upvotes: 1

Views: 280

Answers (1)

FaigB
FaigB

Reputation: 2281

Use:

val result  =  hll.sum(lines_grped_hll) //or suitable method of hll for you

result.dump

Upvotes: 0

Related Questions