CODEWITHSUNDEEP
scalaapache-spark
koh-ding
koh-ding

Reputation: 135

How do update a single cell of Scala Dataframe?

Here is a sample dataframe:

    var df = Seq(("Bob","Male","20"),("Jenn","Female","25")).toDF("Name","Gender","Age")

enter image description here

I want to change the "Age" column of the row where "Name" is equal to "Bob" to 30.

    var name_to_change = "Bob"
    var new_age = 30

How would I go about this?

Upvotes: 3

Views: 384

Answers (2)

s.polam
s.polam

Reputation: 10382

You can also use Hive Conditional Functions and is similar to the IF statements in other programming languages

IF(boolean testCondition, T valueTrue, T valueFalseOrNull) inside expr(expr: String) function.

scala> df.show(false)
+----+------+---+
|Name|Gender|Age|
+----+------+---+
|Bob |Male  |20 |
|Jenn|Female|25 |
+----+------+---+


scala> df
       .withColumn("Age",expr("if((Name ='Bob'),30,Age)"))
       .show(false)

+----+------+---+
|Name|Gender|Age|
+----+------+---+
|Bob |Male  |30 |
|Jenn|Female|25 |
+----+------+---+

Upvotes: 1

Simon Delecourt
Simon Delecourt

Reputation: 1599

In spark you can't update a single cell. What you can do is update a column. But using condition you can do what you want.

df.withColumn("Age", when(col("Name") === name_to_change, lit(new_age)).otherwise(col("Age")))

Upvotes: 6

Related Questions