CODEWITHSUNDEEP
scaladataframeapache-spark-sql
user2128702
user2128702

Reputation: 2121

Divide two BigInt values and cast the result to Float in Scala

I have a data frame which contains two columns of type BigInt. Then I have a user-defined function which performs an operation over these two columns and the final result is supposed to be of type Float.

def generateNewColumnValue(firstColumnValue: BigInt, secondColumnValue: BigInt): Float = {
   val calculated = scala.math.pow(firstColumnValue.toFloat / secondColumnValue, 1.0/3.0);
   return calculated;
}

val generateNewColumnValueUDF = udf[Float, BigInt, BigInt](generateNewColumnValue);

Inside of the body of the UDF I am doing some very simple calculations, as you can see. The problem is that I get the following error and I don't understand why it is not possible:

command-836521033094408:9: error: overloaded method value / with alternatives:
  (x: Double)Double <and>
  (x: Float)Float <and>
  (x: Long)Float <and>
  (x: Int)Float <and>
  (x: Char)Float <and>
  (x: Short)Float <and>
  (x: Byte)Float
 cannot be applied to (BigInt)
  val calculated = scala.math.pow(firstColumnValue.toFloat / secondColumnValue, 1.0/3.0);

The problem is that if I try to cast it to a lower range type (like Int) I might be losing some value after the decimal point.

Upvotes: 2

Views: 1226

Answers (1)

Alexey Romanov
Alexey Romanov

Reputation: 170723

The message just says you can divide a Float by Double, Float, etc. but not by BigInt. Call toFloat on both operands, not just one:

firstColumnValue.toFloat / secondColumnValue.toFloat

But math.pow takes Doubles, not Floats, so toDouble makes more sense. If the result has to be Float, call toFloat on the result of pow, not its arguments.

Or going through BigDecimal:

(BigDecimal(firstColumnValue) / BigDecimal(secondColumnValue)).toDouble

In most cases it should give approximately the same result as the first option, but slower; the problem is that BigInts can be so large that firstColumnValue.toDouble returns Double.PositiveInfinity/NegativeInfinity. You could check for that, and only use the second option in that case.

Upvotes: 3

Related Questions