CODEWITHSUNDEEP
apache-sparkapache-spark-sql
SimbaPK
SimbaPK

Reputation: 596

Multiline string to spark dataframe

i would like to convert a multiline string to a spark dataframe, what is the best way ? 

val s =
      """
        |col1,col2,col3
        |a,b,c
        |u,v,w
      """.stripMargin

My current method: I write the string to a csv file and i read the csv file with the sparkSession. This is obviously not the best way

Upvotes: 0

Views: 1995

Answers (2)

SimbaPK
SimbaPK

Reputation: 596

val u = s.split("\n").drop(1).toSeq.map(_.split(",")).map(x => (x(0),x(1),x(2))).toDF("col1","col2","col3")

u.show()

+----+----+----+
|col1|col2|col3|
+----+----+----+
|   a|   b|   c|
|   u|   v|   w|
+----+----+----+

Upvotes: 1

pawinder gupta
pawinder gupta

Reputation: 1265

That can be achieved by splitting the string into a sequence and then defining the dataframe on the sequence.

scala> """
     |         |col1,col2,col3
     |         |a,b,c
     |         |u,v,w
     |       """.stripMargin.split("\n").toSeq.toDF.show
+--------------+
|         value|
+--------------+
|              |
|col1,col2,col3|
|         a,b,c|
|         u,v,w|
|              |
+--------------+

Upvotes: 0

Related Questions