CODEWITHSUNDEEP
jsonscalaapache-sparkcouchdb
Anubhav Dikshit
Anubhav Dikshit

Reputation: 1829

Parsing view results from CouchDB in Spark/Scala

Introduction:

I have a view in a couched running in my local (http://127.0.0.1:5984/sample/_design/view/_view/data)

Input:

The results of the view are so:

{"total_rows":3,"offset":0,"rows":[
{"id":"1","key":["2032","0"],"value":{"context":"2032","application_id":"2412"}},
{"id":"21","key":["214","0"],"value":{"context":"1312","application_id":"4242"}}
]}

Objective:

I wanted to parse this in spark, and thus I wrote the following code

Code:

// For implicit conversions like converting RDDs to DataFrames
import spark.implicits._

// SQL context
import org.apache.spark.sql.SparkSession

// Importing CouchDB Connector (made by my someone in our office)
import com.artoo.spark.connector.couch.CouchConnectorCore

// JSON parser
import org.json4s.native.JsonMethods._


val connector = new CouchConnectorCore("http://127.0.0.1:5984/sample/_design/view/_view/data?since=0&limit=10", "0", 10)
connector.start()    

// getting data
val json = connector.getData

// printing results
println(json)

// parse son
val parsed_json = parse(json)

Problem:

val parsed_json = parse(json)   

error: overloaded method value parse with alternatives:
  (in: org.json4s.JsonInput,useBigDecimalForDouble: Boolean,useBigIntForLong: Boolean)org.json4s.JValue <and>
  (in: org.json4s.JsonInput,useBigDecimalForDouble: Boolean)org.json4s.JValue
 cannot be applied to (String)
       val parsed_json = parse(json)

P.S:

Thing run fine till, "println(json)", its output is same as the "Input" shown above

Running on Spark 2.1.0 with hadoop 2.7, Scala version 2.11.8 on OSX

Upvotes: 1

Views: 587

Answers (2)

Julescs0
Julescs0

Reputation: 21

If you look at the source -

https://github.com/json4s/json4s/blob/master/native/src/main/scala/org/json4s/native/JsonMethods.scala

you can see the constructor -

def parse(in: JsonInput, useBigDecimalForDouble: Boolean = false, useBigIntForLong: Boolean = true): JValue

What's weird is that we are only proposed two options and the compiler doesn't recognise the possibility that excludes all defaults. I just tried this in my current setup with vsc/metals, scala 2.11 and latest json4s. The underlying reason I haven't checked right now, however importantly it is still good to know that there are two additional (defaulted) params here that can be set defining the parsing behaviour for Doubles/Longs. The default being false.

Conclusion: Setting it to false emulates the intended default behaviour so probably that is better than setting it to true unless you really need Big values.

Upvotes: 0

Paul Back
Paul Back

Reputation: 1319

I ran into this as well and resolved it by adding the second argument useBigDecimalForDouble.

val parsed_json = parse(json, true)

Upvotes: 3

Related Questions