CODEWITHSUNDEEP
scalaapache-flink
Bhavin C
Bhavin C

Reputation: 1

Flink: scala case class being serialized as GenericType via Kryo

I have following case classes defined in my flink application (Flink 1.10.1)

case class FilterDefinition(filterDefId: String, filter: TileFilter)

case class TileFilter(tiles: Seq[Long], zoomLevel: Int)

During runtime, I noticed the log saying

FilterDefinition cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.

If I interpreted Flink documentation correctly, the flink should be able to serialize the scala case classes and not need Kryo for it. However, it looks like for me, the above case class fallbacks on Kryo serializer.

Did I miss interpret how case classes are handled by flink?

Upvotes: 0

Views: 1355

Answers (1)

David Anderson
David Anderson

Reputation: 43707

Excerpting here from the documentation:

Java and Scala classes are treated by Flink as a special POJO data type if they fulfill the following requirements:

  • The class must be public.

  • It must have a public constructor without arguments (default constructor).

  • All fields are either public or must be accessible through getter and setter functions. For a field called foo the getter and setter methods must be named getFoo() and setFoo().

  • The type of a field must be supported by a registered serializer.

In this case Flink it appears that Flink doesn't know how to serialize TileFilter (or more specifically, Seq[Long]).

Upvotes: 0

Related Questions