SmallWong
SmallWong

Reputation: 46

SparkOnHBase throws ERROR “had a not serializable result: org.apache.hadoop.hbase.client.Result”

It throws error when build spark like this

config("spark.serializer","org.apache.spark.serializer.JavaSerializer")

and the error is

ERROR TaskSetManager: Task 0.0 in stage 2.0 (TID 12) had a not serializable result: org.apache.hadoop.hbase.client.Result.Serialization stack:
    - object not serializable (class: org.apache.hadoop.hbase.client.Result

If i change JavaSerializer to KryoSerilizer,it works.

But in my application, it must use "JavaSerilizer", because of service.

Upvotes: 0

Views: 300

Answers (1)

Prashant Mittal
Prashant Mittal

Reputation: 21

You can't serialise HBase Result using JavaSerializer. You can convert result to (Array[Byte],java.util.List[(Array[Byte], Array[Byte], Array[Byte])]) with below code.

    val it = result.listCells().iterator()
    val list = new util.ArrayList[(Array[Byte], Array[Byte], Array[Byte])]()

    while (it.hasNext) {
      val kv: Cell = it.next()
      list.add((CellUtil.cloneFamily(kv), CellUtil.cloneQualifier(kv), CellUtil.cloneValue(kv)))
    }

    (result.getRow(),list)

Upvotes: 1

Related Questions