CODEWITHSUNDEEP
javaserializationapache-stormkryo
Gal
Gal

Reputation: 11

How to serialize a field of a non-primitive type (your own class) in a Storm topology?

The following exception is thrown while I run my Storm project:

java.lang.RuntimeException: java.io.NotSerializableException: com.youtab.dataType.id.GUID
at backtype.storm.serialization.DefaultSerializationDelegate.serialize(DefaultSerializationDelegate.java:43)
at backtype.storm.utils.Utils.serialize(Utils.java:85)
at backtype.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:111)

As part of my Storm project, I am trying to transmit an Object type Event from the first Spout to the first Bolt and then to use it. Unfortunately, after implementing all needed changes and commits in my configuration variable - as described in the Storm documentations, it is still failing to deserialize one private field of type "GUID" which is one field from my own private class Event.

I have created the following serializing class:

public class GUIDSerializer extends Serializer<GUID> {
    @Override
    public void write(Kryo kryo, Output output, GUID guid) {
        output.write(guid.toString().getBytes());
    }

    @Override
    public GUID read(Kryo kryo, Input input, Class<GUID> aClass) {
        return GUID.of(input.readString());
    }
}

And I registered the serialize as needed:

Config conf = new Config();
conf.registerSerialization(GUID.class, GUIDSerializer.class);

Upvotes: 0

Views: 1053

Answers (2)

f_puras
f_puras

Reputation: 2505

I do not know the Serializer class, but judging from the exception, you need to make your GUID class implement the Interface java.io.Serializable like:

public class GUID implements Serializable {
//...

When in doubt, please post your current code for GUID.

Upvotes: 1

Matthias J. Sax
Matthias J. Sax

Reputation: 62310

All classes used as data type for attributes/fields must implement Java's Serializable interface. In your case, this applies to your Event class as well as all used members of Event like GUID. Of course, this applied recursively, ie, if GUID contains custom types, those must implement Serializable, too.

Providing a custom Kryo serializer is actually not required. Storm can use Java's default serializer. For performance reasons however, it is highly recommended to register custom types. In most case, it is sufficient to simple register the user type classes via

conf.registerSerialization(MyUserType.class);

In your case

conf.registerSerialization(Event.class);
conf.registerSerialization(GUID.class);

Registering a class allows Storm to use a more efficient (general) Kryo serializer instead of Java's default serializer.

If this general Kryo serializer is still not efficient enough, you can provide an own Kryo serializer (as you mentioned in your question). However, the class must still implement Java's Serializable interface!

Upvotes: 3

Related Questions