CODEWITHSUNDEEP
jsonscalaapache-spark
Weston Sankey
Weston Sankey

Reputation: 343

Convert DataFrame of JSON Strings

Is it possible to convert a DataFrame containing JSON strings to a DataFrame containing a typed representation of the JSON strings using Spark 2.4?

For example: given the definition below, I'd like to convert the single column in jsonDF using a schema that is inferred from the JSON string. 

val jsonDF = spark.sparkContext.parallelize(Seq("""{"a": 1, "b": 2}""")).toDF

Upvotes: 0

Views: 89

Answers (1)

ollik1
ollik1

Reputation: 4540

DataFrameReader can read JSON from string data sets. For example using toDS instead of toDF

val jsonDS = Seq("""{"a": 1, "b": 2}""").toDS
spark.read.json(jsonDS).show()

Output:

+---+---+
|  a|  b|
+---+---+
|  1|  2|
+---+---+

Upvotes: 1

Related Questions