CODEWITHSUNDEEP
scalaapache-spark-sql
Edamame
Edamame

Reputation: 25366

scala: value not a member of a DataFrame

I read in a csv file to a DataFrame through scala. Then I got the following error when I tried to do filtering on a field:

val data = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("mode", "PERMISSIVE").option("inferSchema", "true").load("s3n://myPath/myData.csv.bz2")
val newData = data.filter(data.Name != null)

Then I got the error below:

error: value Name is not a member of org.apache.spark.sql.DataFrame
       val newData = data.filter(data.Name != null)

Did I do anything wrong here? Thanks!

Upvotes: 0

Views: 2024

Answers (2)

Himaprasoon
Himaprasoon

Reputation: 2659

You can use DataFrameNaFuctions for this

data.na.drop(Seq("Name"))

You can't access a column using the syntax dataframe.fieldname in scala spark.

Upvotes: 1

Dean Wampler
Dean Wampler

Reputation: 2151

Try data("Name"). DataFrames don't get named accessor methods added to them for your fields.

Upvotes: 0

Related Questions