CODEWITHSUNDEEP
scalaapache-sparkapache-spark-mllibapache-zeppelin
jojo_Berlin
jojo_Berlin

Reputation: 693

Casting BigInt to Int in Spark

Hi I'm trying to cast a BigIntto an intin order to generate Rating classes. I only want to use instances that are small enough to fit into an in I use the following code:

val tup=rs.select("kunden_nr","product_list") 
val rdd=tup.rdd.map(row=>(row.getAs[BigInt](0),row.getAs[Seq[Int]](1)))
val fs=rdd.filter(el=>el._1.isValidInt)
fs.count()
rdd.count()

The fs count delivers the following exception in Zepplin:

java.lang.ClassCastException: java.lang.Long cannot be cast to scala.math.BigInt

Upvotes: 3

Views: 21280

Answers (3)

user7371662
user7371662

Reputation: 11

When I use below to cast , the value of number will be changed!

table.col("id").cast("long") //java

Upvotes: 1

user 923227
user 923227

Reputation: 2715

I noticed that the overflow is happening for me fairly often with .toInt so I tried Adding the remainder and the quotient.

((rec.getLong(0) % Int.MaxValue) + (rec.getLong(0) / Int.MaxValue)).toInt

This was better as I was getting 232 unique values in place of 234 but with toInt I was getting 6 unique values in place of 234.

Upvotes: -2

Pablo Francisco Pérez Hidalgo
Pablo Francisco Pérez Hidalgo

Reputation: 27692

Casting is like changing "the glasses" your code use to represent what is referenced by your value and not actually changing the referenced content nor changing the reference to point to a new BigInt instance.

That implies that you need to get your value with the type it really has and then build a BigInt instance from it:

BigInt(row.getAs[Long](0))

Following the same reasoning, you can create an Int instance from the Long as follows:

row.getAs[Long](0).toInt

But it might overflow the integer type representation range.

Upvotes: 3

Related Questions