CODEWITHSUNDEEP
stringscaladataframeapache-spark
Haha
Haha

Reputation: 1009

Unable to extract String from df Column

I want to add a column to my DF. Content of the new Column is based on other columns. This is what I did: 

val dfr = DFRejID.withColumn("CAUSE_REJET", lit("Reg_ctrl_axe/pas de correspondance pour " + DFRejID.select("COD_ENTREP").as[String].collect()))

This is the result: 

Reg_ctrl_axe/pas de correspondance pour ID_ENTITE=[Ljava.lang.String;@9d1fe08

How can I decode that please. Thank you

Upvotes: 0

Views: 33

Answers (1)

Marsellus Wallace
Marsellus Wallace

Reputation: 18601

This should work!

val dfr = DFRejID.withColumn("CAUSE_REJET", concat(lit("Reg_ctrl_axe/pas de correspondance pour "), $"COD_ENTREP"))

You do not want to nest a .select within your first command. Also, calling .collect will create a collection on your master node (not the executors) and you want to avoid that as well.

Upvotes: 2

Related Questions