CODEWITHSUNDEEP
scalaintellij-ideaapache-spark-sql
Falcom
Falcom

Reputation: 67

Spark Dataframes , WithColumn

I tried the below code in Spark-shell and it works fine. 

val df2 = df3.withColumn("Escore", when($"col2" === $"col3",10).otherwise(0))orderBy(asc("col2"),desc("Escore"),desc("col5"))

But when i try the same command in IntelliJ's Scala with SBT, I am facing the below error.

in Scala IDE : 

val df2: DataFrame = df3.withColumn("Escore": String,when($"col2" === $"col3",10).otherwise(0))orderBy(asc("col2"),desc("Escore"),desc("col5"))

ERROR : Not able to resolve symbol "WHEN" .

Could anyone please provide a sample code or syntax of how to code a withColumn of dataframe in IntelliJ Scala.

My SBT is : 

name := "SparkSqlExample"

version := "1.0"

scalaVersion := "2.11.5"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"

Upvotes: 1

Views: 3651

Answers (1)

Akash Sethi
Akash Sethi

Reputation: 2294

I also tried on spark-shell this works fine but even on intelliJ it is working fine. i think you forget to import the sql functions.

import org.apache.spark.sql.functions._

try this. this will resolve the issue if not add the comment Thanks

Upvotes: 4

Related Questions