CODEWITHSUNDEEP
javaserializationkryo
vladimir vyatkin
vladimir vyatkin

Reputation: 79

deserialize with kryo without dto class in classpath

Is it possible in kryo to serialize an object along with the data schema, or to get the schema from the data serialized in the standard way ? I Need to make sure that the client side doesn't need a class in classpath. to load it from serialized data and then use reflection to subtract its fields, or deserialize all data in Maps, Lists, primitive types etc same as JSON or XML

Upvotes: 0

Views: 203

Answers (1)

QuickSilver
QuickSilver

Reputation: 4045

Saves SampleBean as JSON string 

val conf = new SparkConf()
      .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .set("spark.kryo.registrationRequired", "true")
      .registerKryoClasses(Array(classOf[SampleBean], classOf[InternalRow]
        , classOf[Array[InternalRow]]
        , classOf[WriteTaskResult]
        , classOf[FileCommitProtocol.TaskCommitMessage]
        , classOf[ExecutedWriteSummary],
        classOf[BasicWriteTaskStats]))

    val spark = SparkSession.builder.master("local[*]")
      .config(conf)
      .getOrCreate

    import spark.implicits._
    val df = List(SampleBean("A", "B")).toDF()
    df.write.mode(SaveMode.Overwrite).json("src/main/resources/kryoTest")
    df.printSchema()

reads the data simple JSON

val sparkNew = Constant.getSparkSess
    val dfNew = sparkNew.read.json("src/main/resources/serialisedJavaObj.json").toDF()
    dfNew.printSchema()

Upvotes: 0

Related Questions