Reputation: 46
It throws error when build spark like this
config("spark.serializer","org.apache.spark.serializer.JavaSerializer")
and the error is
ERROR TaskSetManager: Task 0.0 in stage 2.0 (TID 12) had a not serializable result: org.apache.hadoop.hbase.client.Result.Serialization stack:
- object not serializable (class: org.apache.hadoop.hbase.client.Result
If i change JavaSerializer to KryoSerilizer,it works.
But in my application, it must use "JavaSerilizer", because of service.
Upvotes: 0
Views: 300
Reputation: 21
You can't serialise HBase Result using JavaSerializer. You can convert result to (Array[Byte],java.util.List[(Array[Byte], Array[Byte], Array[Byte])]) with below code.
val it = result.listCells().iterator()
val list = new util.ArrayList[(Array[Byte], Array[Byte], Array[Byte])]()
while (it.hasNext) {
val kv: Cell = it.next()
list.add((CellUtil.cloneFamily(kv), CellUtil.cloneQualifier(kv), CellUtil.cloneValue(kv)))
}
(result.getRow(),list)
Upvotes: 1