CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Alessandroempire
Alessandroempire

Reputation: 1699

Spark sqlContext select all

I read data using Spark SQLContext and store it on a variable:

 val somevar = sqlContext.read.parquet(some_file.parquet)

Then I wish to select all the values using select, something like:

  somevar.select(*)

But this does not work.

The equivalent would be to do:

somevar.registerTempTable("sometable")

sqlContext.sql("SELECT * FROM sometable")

But I do not wish to do the previous.

Kind regards.

Upvotes: 1

Views: 3391

Answers (1)

eliasah
eliasah

Reputation: 40360

You have a syntax error. The following syntax is how you select all columns with spark-sql :

import sqlContext.implicit._

val df = Seq((1,2),(2,3)).toDF
// df: org.apache.spark.sql.DataFrame = [_1: int, _2: int]

// solution 1
df.select("*")
// res54: org.apache.spark.sql.DataFrame = [_1: int, _2: int]

// alternative 1
df.select($"*")
// res55: org.apache.spark.sql.DataFrame = [_1: int, _2: int]

// alternative 2
df.select('*)
// res56: org.apache.spark.sql.DataFrame = [_1: int, _2: int]

Upvotes: 6

Related Questions