Cannot created object walking though JSON string using Circe JSON parser

Question

So i am trying to create a custom decoder for a JSON string to be converted into a domain object. I am using Scala/Circe to walk through the JSON and create the object. I am unable to get this to run. I am sure i am not clear on how to walk through the JSON. Can someone advise please ?

This is the JSON in question ,

   {
  "args": {}, 
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3", 
    "Accept-Encoding": "gzip, deflate", 
    "Accept-Language": "en-US,en;q=0.9", 
    "Host": "httpbin.org", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36"
  }, 
  "origin": "61.16.136.118, 61.16.136.118", 
  "url": "https://httpbin.org/get"
}

Here is what my code looks like,

    import com.softwaremill.sttp._
import com.softwaremill.sttp.circe._
import io.circe._, io.circe.parser._

case class Headers(Accept: String, Accept_Encoding: String, Accept_Language: String, Host: String, Upgrade_Insecure_Requests: String, User_Agent: String)
case class RootInterface(args: String, headers: Headers, origin: String, url: String)

object SttpClientGetPost extends App {
  implicit val backend = HttpURLConnectionBackend()

  implicit val rootDecoder: Decoder[RootInterface] =
    (hCursor: HCursor) => {
      val tcursor = hCursor.downField("headers")
      val argsCursor = hCursor.downField("args")
        for{
          args <- for{
          testString <-  argsCursor.get[String]("args")
        }yield testString

        headers <- for {
          Accept <- tcursor.downField("Accept").as[String]
          Accept_Encoding <- tcursor.downField("Accept-Encoding").as[String]
          Accept_Language <- tcursor.downField("Accept-Language").as[String]
          Host <- tcursor.downField("Host").as[String]
          Upgrade_Insecure_Requests <- tcursor.downField("Upgrade-Insecure-Requests").as[String]
          User_Agent <- tcursor.downField("User-Agent").as[String]
        } yield Headers(Accept, Accept_Encoding, Accept_Language, Host, Upgrade_Insecure_Requests, User_Agent)
        origin <- hCursor.downField("Origin").as[String]
        url <- hCursor.downField("url").as[String]
      } yield RootInterface("", headers, origin, url)
    }
  val secondRequest = sttp //.headers(("userId", USER_ID),("password","testpassword"))
    .get(uri"http://httpbin.org/get")

  secondRequest.send().body match {
    case Left(fail) => System.out.println("The Get request was unsuccessful.   " + fail)
    case Right(rawJSONResponse) =>
      parser.decode(rawJSONResponse) match {
        case Left(fail) => System.out.println("The JSON response could not be parsed by Circe.  " + fail)
        case Right(rootInterfaceObject) =>
          System.out.println("Origin :" + rootInterfaceObject.origin)

      }
  }

}

Update : I am trying with this to no avail now,

case class Headers(Accept: String, Accept_Encoding: String, Accept_Language: String, Host: String, Upgrade_Insecure_Requests: String, User_Agent: String)
case class RootInterface(args: String, headers: Headers, origin: String, url: String)


val doc = """{
  "args": {}, 
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3", 
    "Accept-Encoding": "gzip, deflate", 
    "Accept-Language": "en-US,en;q=0.9", 
    "Host": "httpbin.org", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36"
  }, 
  "origin": "61.16.136.118, 61.16.136.118", 
  "url": "https://httpbin.org/get"
}"""


  implicit val decodeHeaders: Decoder[Headers] = Decoder.forProduct6(
  "Accept",
  "Accept-Encoding",
  "Accept-Language",
  "Host",
  "Upgrade-Insecure-Requests",
  "User-Agent"
 )(Headers(_, _, _, _, _, _))

implicit val decodeRootInterface: Decoder[RootInterface] = Decoder.forProduct4(
  "args",  
  "headers",
  "origin",
  "url"
)(RootInterface(_, _, _, _))

val t = decode[RootInterface](doc)

I am still getting a decoding failure. It looks like its because of the args field.

Answer 1

I'd strongly recommend building up your instances compositionally—e.g. instead of baking all of the Headers decoding logic into the RootInterface decoder, you can define a separate Decoder[Headers] instance and then use that in your Decoder[RootInterface].

I'd also recommend avoiding working with cursors directly unless you're doing something particularly complicated that you know requires that level of definition.

So given this document:

val doc = """{
  "args": {}, 
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3", 
    "Accept-Encoding": "gzip, deflate", 
    "Accept-Language": "en-US,en;q=0.9", 
    "Host": "httpbin.org", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36"
  }, 
  "origin": "61.16.136.118, 61.16.136.118", 
  "url": "https://httpbin.org/get"
}"""

…and these case classes:

case class Headers(Accept: String, Accept_Encoding: String, Accept_Language: String, Host: String, Upgrade_Insecure_Requests: String, User_Agent: String)
case class RootInterface(args: String, headers: Headers, origin: String, url: String)

…the following is a complete working example you can use in a REPL:

import io.circe.Decoder, io.circe.jawn.decode

implicit val decodeHeaders: Decoder[Headers] = Decoder.forProduct6(
  "Accept",
  "Accept-Encoding",
  "Accept-Language",
  "Host",
  "Upgrade-Insecure-Requests",
  "User-Agent"
 )(Headers(_, _, _, _, _, _))

implicit val decodeRootInterface: Decoder[RootInterface] = Decoder.forProduct3(
  "headers",
  "origin",
  "url"
)(RootInterface("", _, _, _))

…like this:

scala> decode[RootInterface](doc)
res0: Either[io.circe.Error,RootInterface] = Right(RootInterface(,Headers(text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3,gzip, deflate,en-US,en;q=0.9,httpbin.org,1,Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36),61.16.136.118, 61.16.136.118,https://httpbin.org/get))

(I'm not sure what the intention is with the args decoding, so I've just followed your implementation in using the empty string.)

If you control the Headers case class definition, though, I'd strongly suggest using idiomatic Scala member names (i.e. camel case, not upper snake). Combining that with circe-derivation makes a pretty clear solution in my view:

import io.circe.Decoder, io.circe.derivation.deriveDecoder

case class Headers(
  accept: String,
  acceptEncoding: String,
  acceptLanguage: String,
  host: String,
  upgradeInsecureRequests: String,
  userAgent: String
)

object Headers {
  implicit val decodeHeaders: Decoder[Headers] = deriveDecoder(
    _.replaceAll("([A-Z])", "-$1").capitalize
  )
}

case class RootInterface(
  args: Map[String, String],
  headers: Headers,
  origin: String,
  url: String
)

object RootInterface {
  implicit val decodeRootInterface: Decoder[RootInterface] = deriveDecoder
}

And then:

scala> io.circe.jawn.decode[RootInterface](doc)
res0: Either[io.circe.Error,RootInterface] = Right(RootInterface(Map(),Headers(text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3,gzip, deflate,en-US,en;q=0.9,httpbin.org,1,Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36),61.16.136.118, 61.16.136.118,https://httpbin.org/get))