CODEWITHSUNDEEP
scalaapache-spark
Metadata
Metadata

Reputation: 2083

How to convert single row of a dataframe into a String in Scala?

My dataframe which was created by reading a RDBMS table, has one column and only one value in it:

val sourceCols = spark.read.format("jdbc").option("url", hiveMetaConURL)
                                               .option("dbtable", "(select source_columns from base.spe_tab where tablename='base.forecast') as sCols")
                                               .option("user", metaUserName)
                                               .option("password", metaPassword)
                                               .load()

I tried it convert it to a String in the below way:

val sourceColDataTypes = sourceCols.rdd.map(_.mkString(",")).collect.foreach(println)

When I try to print it as:

sourceColDataTypes.foreach(println)

I don't see the content, instead I see:

[Ljava.lang.String;@1e489957

Is there a way I can use yield of Scala to get the value. Could anyone let me know how can I convert a row in a DataFrame to a String ?

Upvotes: 0

Views: 6674

Answers (2)

Manoj Kumar Dhakad
Manoj Kumar Dhakad

Reputation: 1892

You can try this directly on dataframe, There is no need to covert to rdd at all like this

df.select("value").collect.mkString(",").replaceAll("[\\[\\]]","")

Or

df.map(row=>row.getAs("value").toString).collect.mkString(",")

Upvotes: 0

koiralo
koiralo

Reputation: 23119

To get the value you can use one of the following 

sourceCols.map(_.getString(0)).collect.foreach(println)
sourceCols.map(_.toSeq.mkString(",")).collect.foreach(println)

If you just want to see the data you can use 

sourceCols.show(false)

If you want a single string then you can get it as 

println(sourceCols.map(_.getString(0)).collect.mkString(","))

Hope this helps!

Upvotes: 1

Related Questions