CODEWITHSUNDEEP
rsparkr
François M.
François M.

Reputation: 4278

Add column to DataFrame in sparkR

I would like to add a column filled with a character N in a DataFrame in SparkR. I would do it like that with non-SparkR code :

df$new_column <- "N"

But with SparkR, I get the following error : 

Error: class(value) == "Column" || is.null(value) is not TRUE

I've tried insane things to manage it, I was able to create a column using another (existing) one with df <- withColumn(df, "new_column", df$existing_column), but this simple thing, nope...

Any help ?

Thanks.

Upvotes: 13

Views: 5103

Answers (2)

Dmitriy Selivanov
Dmitriy Selivanov

Reputation: 4595

The straight solution will be to use SparkR::lit() function:

df_new = withColumn(df, "new_column_name", lit("N"))

Edit 7/17/2019

In newer Spark versions, the following also works:

df1$new_column <- "N"
df1[["new_column"]] <- "N"

Upvotes: 15

data princess
data princess

Reputation: 1158

There's an easier way to use SparkR::lit() that more closely mimics the syntax you tried first:

df$new_column <- lit("N")

Upvotes: 0

Related Questions