Akka Remoting: Dead Letters after enabling Serialization

Question

I am working on a Akka-Remoting example. It was running fine before configuring serialization support into the application.conf file. But, after enabling Serialization messages are not being delivered properly.

I can see the log for DeadLetter on both remote and client application upon receive and send of messages. I think messages are being exchanged, but not being delivered to the right actor.

Here is the conf file of remote calculator which runs in akka-mircokernel.

remoteCalculator {
  akka {
    actor {
      #serialize-messages = on

  #No need to define serializers: java and protobuf are supported by Akka.
  serializers {
    java = "akka.serialization.JavaSerializer"
    #akka-containers = "akka.remote.serialization.MessageContainerSerializer"
    proto = "akka.remote.serialization.ProtobufSerializer"
    #daemon-create = "akka.remote.serialization.DaemonMsgCreateSerializer"
 }

serialization-bindings {
  #"akka.actor.ActorSelectionMessage" = akka-containers
  #"com.google.protobuf.GeneratedMessage" = proto
  #"akka.remote.DaemonMsgCreate" = daemon-create
  "java.lang.Integer"       = proto
  "scala.Int"               = proto
  "akka.example.MathOp" = proto
  #"akka.example.Add"       = proto
  #"akka.example.Subtract" = proto
  #"akka.example.Multiply" = proto
  #"akka.example.Divide"    = proto
  "akka.example.MathResult"         = proto
  #"akka.example.AddResult"             = proto
  #"akka.example.SubtractResult"        = proto
  #"akka.example.MultiplicationResult" = proto
  #"akka.example.DivisionResult"        = proto 
}
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
  log-received-messages = on
  log-sent-messages = on
  log-remote-lifecycle-events = on
  enabled-transports = ["akka.remote.netty.tcp"]
  netty.tcp {
    hostname = "IGLR000PBG3K8Y-ubuntu"
    port = 2555
    log-received-messages = on
    log-sent-messages = on
    log-remote-lifecycle-events = on
  }
 }
}
}

and the conf file for the calculator-client application

LocalSys {
  akka {
    actor {
      #serialize-messages = on

  #No need to define serializers: java and protobuf are supported by Akka.
  serializers {
    java = "akka.serialization.JavaSerializer"
    #akka-containers = "akka.remote.serialization.MessageContainerSerializer"
    proto = "akka.remote.serialization.ProtobufSerializer"
    #daemon-create = "akka.remote.serialization.DaemonMsgCreateSerializer"
 }

serialization-bindings {
  #"akka.actor.ActorSelectionMessage" = akka-containers
  #"com.google.protobuf.GeneratedMessage" = proto
  #"akka.remote.DaemonMsgCreate" = daemon-create
  "java.lang.Integer"       = proto
  "scala.Int"               = proto
  "akka.example.MathOp" = proto
  #"akka.example.Add"       = proto
  #"akka.example.Subtract" = proto
  #"akka.example.Multiply" = proto
  #"akka.example.Divide"    = proto
  "akka.example.MathResult"         = proto
  #"akka.example.AddResult"             = proto
  #"akka.example.SubtractResult"        = proto
  #"akka.example.MultiplicationResult" = proto
  #"akka.example.DivisionResult"        = proto 
}
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
  log-received-messages = on
  log-sent-messages = on
  log-remote-lifecycle-events = on
  enabled-transports = ["akka.remote.netty.tcp"]
  netty.tcp {
    hostname = "IGLR000PBG3K8Y-ubuntu"
    port = 2556
    log-received-messages = on
    log-sent-messages = on
    log-remote-lifecycle-events = on
  }
 }
}

}

Here is what I can see on the remote calculator console

[INFO] [11/27/2013 18:40:00.425] [CalculatorApplication-akka.actor.default-dispatcher-6] [akka://CalculatorApplication/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FCalculatorApplication%4010.147.137.227%3A60905-1] Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://CalculatorApplication/deadLetters] to Actor[akka://CalculatorApplication/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FCalculatorApplication%4010.147.137.227%3A60905-1#-1343833615] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [11/27/2013 18:40:00.429] [CalculatorApplication-akka.actor.default-dispatcher-5] [akka://CalculatorApplication/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FCalculatorApplication%4010.147.137.227%3A60905-1] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://CalculatorApplication/deadLetters] to Actor[akka://CalculatorApplication/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FCalculatorApplication%4010.147.137.227%3A60905-1#-1343833615] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

and on the calculator client console

[INFO] [11/27/2013 18:40:00.343] [clientkernel-akka.actor.default-dispatcher-2] [akka://clientkernel/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FCalculatorApplication%40IGLR000PBG3K8Y-ubuntu.ampf.com%3A2555-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FCalculatorApplication%40IGLR000PBG3K8Y-ubuntu%3A2555-0] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://clientkernel/deadLetters] to Actor[akka://clientkernel/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FCalculatorApplication%40IGLR000PBG3K8Y-ubuntu.ampf.com%3A2555-0/endpointWriter/endpointReader-akka.tcp%3A%2F%2FCalculatorApplication%40IGLR000PBG3K8Y-ubuntu%3A2555-0#-773728878] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [11/27/2013 18:40:00.361] [clientkernel-akka.actor.default-dispatcher-3] [akka://clientkernel/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FCalculatorApplication%40IGLR000PBG3K8Y-ubuntu%3A2555-1] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://clientkernel/deadLetters] to Actor[akka://clientkernel/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FCalculatorApplication%40IGLR000PBG3K8Y-ubuntu%3A2555-1#-249262169] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

Before enabling serialization everything was working fine. I am not able to figure out what configuration options I am missing or doing it wrong. Any help appreciated.

Edit: Do I need to extend any class in my pojos for the serialization to work correctly?

Answer 1

If changing the serialization configuration causes messages to be lost then obviously serialization must fail (I would not call it “enabling” since serialization must always happen). The classes which you configure to use protobuf are not suitable for that serializer, which only knows how to treat com.google.protobuf.Message. You will have to define a protobuf protocol definition, compile it using protoc and then either use the generated classes as messages or write your own serializer which converts from your message classes—like MathOp—to their protobuf equivalent.

Protobuf is not automagic; you can also look into kryo, for which there is an externally maintained akka module.