CODEWITHSUNDEEP
javaapache-sparkapache-spark-sql
lte__
lte__

Reputation: 7583

Spark SQL - Select all AND computed columns?

This is a total noob question, sorry for that. In Spark, I can use select as:

df.select("*"); //to select everything
df.select(df.col("colname")[, df.col("colname")]); //to select one or more columns
df.select(df.col("colname"), df.col("colname").plus(1)) //to select a column and a calculated column

But. How can I select all the columns PLUS a calculated one? Obviously select("*", df.col("colname").plus(1)) doesn't work (compilation error). How can this be done under JAVA? Thank you!

Upvotes: 10

Views: 43114

Answers (3)

Saurabh
Saurabh

Reputation: 7833

The difference between .select() and .withColumn() methods is that .select() returns only the columns you specify, while .withColumn() returns all the columns of the DataFrame in addition to the one you defined.

You can directly use withColumn:

df.withColumn("ColName", col("colName").plus(1))

Upvotes: 1

Daniel Gallegos
Daniel Gallegos

Reputation: 41

You can use withColumn() method, this will create a new column to the DataFrame. 

df.select("*")
  .withColumn("ColName", col("colName").plus(1))

Upvotes: 4

Yuan JI
Yuan JI

Reputation: 2995

Just do:

df.select(df.col("*"), df.col("colName").plus(1));

Upvotes: 24

Related Questions