CODEWITHSUNDEEP
scalaapache-spark
Khaned
Khaned

Reputation: 443

spark shell column multiplication and updating same dataframe

I have a dataframe result

  +-----+--------+----------+-----+------------------+
  |count|currency|      date|value|         converted|
  +-----+--------+----------+-----+------------------+
  |    3|     USD|2021-01-14|    4|2.9311893333333336|
  |  102|     USD|2021-01-14|    3|               3.0|
  |  234|     USD|2021-01-14|    5| 3.663986666666667|
  |   68|     USD|2021-01-14|    6| 4.933771999999999|
  |   20|     USD|2021-01-14|    1|0.7327973333333334|
  |   28|     USD|2021-01-14|    5| 3.663986666666667|
  +-----+--------+----------+-----+------------------+

I want to multiply converted * count and store in another column in result

Desired Output

  +-----+--------+----------+-----+------------------+----------------+
  |count|currency|      date|value|         converted| convertedValue |
  +-----+--------+----------+-----+------------------+----------------+
  |    3|     USD|2021-01-14|    4|2.9311893333333336| 8.793568       |
  |  102|     USD|2021-01-14|    3|               3.0| 306            |
  |  234|     USD|2021-01-14|    5| 3.663986666666667| 857.37288      |
  |   68|     USD|2021-01-14|    6| 4.933771999999999| 335.496496     |
  |   20|     USD|2021-01-14|    1|0.7327973333333334| 14.6559466667  |
  |   28|     USD|2021-01-14|    5| 3.663986666666667| 102.591626667  |
  +-----+--------+----------+-----+------------------+----------------+

My Attempt and Error

    scala> result.withColumn("convertedValue", result["count"]*result["converted"]).show()
<console>:1: error: identifier expected but string literal found.
       result.withColumn("convertedValue", result["count"]*result["converted"]).show()

Upvotes: 0

Views: 121

Answers (3)

Amardeep Flora
Amardeep Flora

Reputation: 1380

import spark.implicits._
result.withColumn("convertedValue", 'count * 'converted)

Upvotes: 1

koiralo
koiralo

Reputation: 23119

You can use the operators as below:

import spark.implicits._
result.withColumn("convertedValue", $"count" * $"converted")

Upvotes: 1

mck
mck

Reputation: 42422

Scala syntax for selecting column is different from Python. Try using parenthesis instead of square brackets:

result.withColumn("convertedValue", result("count")*result("converted")).show

Upvotes: 1

Related Questions