play-json: JsNull is not equal to JsString(null), how to get around this issue?

Question

I'm working on some largeish JSON output. We've got a bunch of tests to check the output. We've created the tests by having a copy of the JSON on disk, performing a Json.parse() on the InputStream and comparing that to the JsObject we have build in memory.

This worked well until I started converting some of our Writes to using the functional syntax (i.e. instead of overriding the writes method in the trait, use the builders).

Suddenly, tests started failing: complaining about null fields that were not equal.

Apparently, when using functional syntax, an Option[String] will convert to a JsString(null) instead of JsNull. This is not noticeable In the stringified version.

Consider the following snippet, using

libraryDependencies += "com.typesafe.play" %% "play-json" % "2.7.4"

import play.api.libs.json._
import play.api.libs.functional.syntax._

object FooBar {

  case class Foo(option: Option[String])

  def main(args: Array[String]): Unit = {

    val classic: OWrites[Foo] = new OWrites[Foo] {
      override def writes(f: Foo): JsObject = Json.obj("foo" -> f.option)
    }

    val dsl: OWrites[Foo] = (__ \ "foo").write[String].contramap(foo => foo.option.orNull)

    val json_classic = Json.toJsObject(Foo(None))(classic)
    val json_dsl = Json.toJsObject(Foo(None))(dsl)


    val json_parse = Json.parse("""{"foo":null}""")

    val string_classic = Json.prettyPrint(json_classic)
    val string_dsl = Json.prettyPrint(json_dsl)

    println(
      s"""Result is:
         |json_dsl       == json_classic : ${json_dsl == json_classic} // (expect true)
         |json_dsl       == json_parse   : ${json_dsl == json_parse} // (expect true)
         |json_classic   == json_parse   : ${json_classic == json_parse} // (expect true)
         |string_classic == string_dsl   : ${string_classic == string_dsl} // (expect true)
         |""".stripMargin)

    println(s"classic:
$string_classic")
    println(s"dsl:
$string_dsl")

  }
}

Actual output is

Result is:
json_dsl       == json_classic : false // (expect true)
json_dsl       == json_parse   : false // (expect true)
json_classic   == json_parse   : true // (expect true)
string_classic == string_dsl   : true // (expect true)

classic:
{
  "foo" : null
}
dsl:
{
  "foo" : null
}

When debugging, you'll see that the classic create a wrapper object with a Tuple ("foo", JsNull), whereas the dsl creates a wrapper with a Tuple ("foo", JsString(null)).

It seems that the intended way of the dsl is to use writeNullable in this case, but it feels odd that it works this way.

I'd either expect JsString(null) == JsNull to be true, or that the dsl would catch the null value and prevent a JsString from being created.

Am I doing something totally misguided?

I would just rewrite to .writeNullable[String], which will remove the field from the JSON, but we have a schema in place that requires the field to be present:

...
"properties": {
  ...
  "foo": {
    "oneOf": [
      {"type": "string"},
      {"type": "null"}
    ]
  },
  ...
}
...
"required": [ "foo" ],

This is part of an API, so changing it will take time.

To clarify: String representation is correct in all cases. I'm only interested in the in-memory representation of the JsValue so I can use its equality during testing.

James Roper · Accepted Answer

I think what you want is this:

val dsl: OWrites[Foo] = (__ \ "foo").writeOptionWithNull[String]
  .contramap(_.option)

Or, if using an older play-json version that doesn't have writeOptionWithNull:

val dsl: OWrites[Foo] = (__ \ "foo").write[JsValue]
  .contramap(_.option match {
    case None => JsNull
    case Some(string) => JsString(string)
  })

Note, play.api.libs.json.JsNull, and null, are two completely different and unrelated concepts, and should never be confused or mixed. The convention across the whole Scala ecosystem in Scala-only APIs is to never, ever use null, to the extent where most libraries simply pretend that it doesn't even exist. So, you won't find many Scala libraries that do null checks, they just assume everything is non null, and the moment you start using null, you're in the wild west of untested and unsupported edge cases. The only time you use or deal with nulls is if working with a Java API, because they use nulls, and the convention there is to wrap anything that can produce a null in Option as early as possible, and unwrap Option using orNull as late as possible, so that at least internally in the Scala code that doesn't directly interface to Java code, everything is using Option, rather than null. But play-json is designed for Scala use only, so like the rest of the Scala-only ecosystem, it just assumes null doesn't exist.

Of course, in json, null does exist, and there may be valid reasons to use it (especially when integrating with other systems that require it). So play-json does model it, but it models it in a very strongly typed way - nothing will ever automatically go from being null in Scala to being null in JSON, it's always very explicit, this is in line with Scala being strongly typed. Also, use of null in JSON doesn't tend to be that common, so most of the default methods (ie, the writeNullable method) map Option to a non-existent value, so you have to be a little more explicit when you want to write a JSON null, as shown above.

play-json: JsNull is not equal to JsString(null), how to get around this issue?

Answers (2)

Related Questions