CODEWITHSUNDEEP
jsonscalaapache-spark
DaehyunPark
DaehyunPark

Reputation: 77

How to parse JSON file with no SparkSQL?

I have the following JSON file.

{
  "reviewerID": "ABC1234",
  "productID": "ABCDEF",
  "reviewText": "GOOD!",
  "rating": 5.0,
},
{
  "reviewerID": "ABC5678",
  "productID": "GFMKDS",
  "reviewText": "Not bad!",
  "rating": 3.0,
}

I want to parse without SparkSQL and use a JSON parser.

The result of parse that i want is textfile.

ABC1234::ABCDEF::5.0
ABC5678::GFMKDS::3.0

How to parse the json file by using json parser in spark scala?

Upvotes: 0

Views: 248

Answers (1)

Jacek Laskowski
Jacek Laskowski

Reputation: 74769

tl;dr Spark SQL supports JSONs in the format of one JSON per file or per line. If you'd like to parse multi-line JSONs that can appear together in a single file, you'd have to write your own Spark support as it's not currently possible.

A possible solution is to ask the "writer" (the process that writes the files to be nicer and save one JSON per file) that would make your life much sweeter.

If that does not give you much, you'd have to use mapPartitions transformation with your parser and somehow do the parsing yourself.

val input: RDD[String] = // ... load your JSONs here
val jsons = jsonRDD.mapPartitions(json => // ... use your JSON parser here)

Upvotes: 1

Related Questions