CODEWITHSUNDEEP
scalaserializationpersistenceclassloaderdeserialization
Daniel Mahler
Daniel Mahler

Reputation: 8193

Deserialization throws 'ClassNotFoundException: JavaConversions$SeqWrapper' in Scala 2.10

I have a fairly complex object graph serialized out from Scala-2.9 and I need to read it into Scala-2.10. However, somewhere deep in the object graph Scala-2.10 throws:

! java.lang.ClassNotFoundException: scala.collection.JavaConversions$SeqWrapper
! at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_21]
! at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_21]
! at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_21]
! at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_21]
! at java.lang.ClassLoader.loadClass(ClassLoader.java:423) ~[na:1.7.0_21]
! at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_21]
! at java.lang.ClassLoader.loadClass(ClassLoader.java:356) ~[na:1.7.0_21]
! at java.lang.Class.forName0(Native Method) ~[na:1.7.0_21]
! at java.lang.Class.forName(Class.java:266) ~[na:1.7.0_21]
! at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:623) ~[na:1.7.0_21]
...

What is the simplest way to load this serialized object into Scala-2.10? The object deserializes correctly with Scala-2.9, but looks like things have moved around in the standard library. Most of the members of scala.collection.JavaConversions are now in scala.collection.convert.Wrappers

Going forward, I am also interested in more robust ways of persisting large complex object graphs without having to explicitly specify the serialization for every class involved.

Upvotes: 1

Views: 1811

Answers (4)

Daniel Mahler
Daniel Mahler

Reputation: 8193

So here is what ended up working for me, thanks to @som-snytt for pointing me in the right direction:

object MyWrappers {
  import java.{ lang => jl, util => ju }, java.util.{ concurrent => juc }
  import scala.collection.convert._
  import WrapAsScala._
  import WrapAsJava._
  import Wrapper._

  @SerialVersionUID(3200663006510408715L)
  case class SeqWrapper[A](underlying: Seq[A]) extends ju.AbstractList[A] with Wrappers.IterableWrapperTrait[A] {
    def get(i: Int) = underlying(i)
  }
}

import org.apache.commons.io.input.ClassLoaderObjectInputStream
object Loader extends ClassLoader {
  override def loadClass(name: String) : Class[_] = {
    import javassist._
    try super.loadClass(name)
    catch {
      case e: ClassNotFoundException if name.startsWith("scala.collection.JavaConversions") => {
            val name2 = name.replaceFirst("scala.collection.JavaConversions",
                                          "MyWrappers")
            val cls = ClassPool.getDefault().getAndRename(name2, name)
            cls.toClass()
          }
    }
  }
}

val objectStream = new ClassLoaderObjectInputStream(Loader, stream)
objectStream.readObject()

This allows me to read in my original 2.9 serialized files directly into 2.10 without re-serialization. It depends on Javassist to do the Class indirection and uses ClassLoaderObjectStream from Apache Commons, even though that would be simple to roll your own. I am not that happy that I had to make my own copy of SeqWrapper (this turned out to be the only offending class in my file), but the wrapper classes in scala-2.10's scala.collection.convert.Wrappers have different SerialVersionUIDs than the corresponding classes in 2.9's scala.collection.JavaConversions even though the sources are textually identical. I originally tried to just redirect to scala.collection.convert.Wrappers and set the SerialVersionUID with Javassist:

object Loader extends ClassLoader {
  override def loadClass(name: String) : Class[_] = {
    import javassist._
    try super.loadClass(name)
    catch {
      case e: ClassNotFoundException if name.startsWith("scala.collection.JavaConversions") => {
            val name2 = name.replaceFirst("JavaConversions", "convert.Wrappers")
            val cls = ClassPool.getDefault().getAndRename(name2, name)
            cls.addField(CtField.make("private static final long serialVersionUID = 3200663006510408715L;", cls))
            cls.toClass()
          }
    }
  }
}

That allowed me to read the serialized file without exception, but the object that was read in that way was incomplete. (If this approach worked and there was more than one problem class in the file, I would really need a lookup table for the SerialVersionUIDs, but that is beside the point). If anybody knows a way set the SerialVersionUID on the Javassist generated class without breaking anything else something else I would like to hear it.

Upvotes: 0

som-snytt
som-snytt

Reputation: 39577

Please don't downvote as a knee-jerk, but my thought was to deserialize in one class loader (on scala 2.9), convert to a java collection, then in a second class loader (with 2.10 on the class path) convert from java back to scala.

In other words, java is the common format (the java runtime is common to both class loaders).

(Alternatively, instead of two classloaders, try serializing the java forms, then scarfing it back.)

I think this goes to Richard's #6. It doesn't have to be entirely java core, only what is compatible.

I'll try to come up with an example during a coffee break, but of course the caveat would be that the incompatibility rests with the collection and not with what is collected.

Upvotes: 1

Reuben
Reuben

Reputation: 165

This will probably blow up in your face, but you could try putting the 2.9 scala-library.jar on your classpath after the 2.10 scala-library.jar. The class loader should then find

 scala.collection.JavaConversions$SeqWrapper

, but I'd be surprised if the serialization goes all the way through ...

You should consider just serializing to json with gson or jackson or lift-json or whatever. It's tedious, but you do it once, and it's done, and it will work with other languages too - your "quick" solution will mean recurring pain going forward ...

Good luck!

Reuben

Upvotes: 0

Richard Sitze
Richard Sitze

Reputation: 8463

Thoughts, of no real help:

  1. You've bumped up against a change in underlying implementation for Scala collections that is reflected in the serialization of same. You're unable to "just load it" in to 2.10, so you need some common ground.

  2. You're likely to run into this with every version of Scala, as the Collections haven't fully settled.

  3. I presume your intent is to load your graph using 2.9 based code, convert to some new format, and dump in a new "common" format.

  4. In the java world, I'd reach for JAXB or SDO; perhaps EclipseLink MOXy. I doubt MOXy will be aware of Scala collection types.

  5. I presume you've already seen this.

  6. Can your object graph be converted to something based entirely on core Java data types?

Upvotes: 2

Related Questions