Akka: - why is it that messages are not guaranteed to arrive (after being send)? What is the failure rate for messages?

Question

It says in the documentation that akka has "at-most once delivery", and that messages are not guaranteed to arrive at destination.

What is the reason for this behavior? What happens to the messages which are not delivered? Are they considered lost?

Edit: I forgot the most important part. Is there a rate of loss I can refer to? Do you know how pessimistic I have to be as regards message delivery guarantee (like is there % rate of failures?).. Just because I have a cluster of actors, and they will be running on the same web-server, and I don't know If I should be thinking if message failures will be 1 in 5, or 1 in 100.

Answer 1

So, Akka actually offers both options of "at-most once delivery" and guaranteed delivery aka "at least once delivery". See the link user1234 posted in the comments for a good introduction into the concepts of "at least once" vs "at most once".

So Akka can do either option. Why is "at most once" the default?

The short answer to your question is that switching to guaranteed delivery has at least five very significant costs:

• Guaranteed delivery in a distributed system requires persistence (otherwise you can't guarantee delivery in case of node failure). This introduces an enormous performance overhead, however.

• Guaranteed delivery in a distributed system introduces the chance of duplicate deliveries. (If you can't confirm delivery, fundamentally the only thing you can do is retry, introducing potential duplicates). For some applications this is problematic. (You can resolve this with de-duping, but this introduces additional overhead and is generally easier to do in application code.) Thus the term "at least once".

• Guaranteed delivery can also affect message order. e.g. you can either wait for each acknowledgement before attempting the next (which would be insanely slow) or you can deliver retries out of order. Which, for many applications, can be problematic.

• Guaranteed delivery just has a lot more overhead. This includes both the bookkeeping necessary, the memory to basically keep messages around longer, and also the network chatter of acknowledgements.

• Guaranteed delivery also requires persistence, as noted above. But not only does this add overhead as already mentioned, but this just also adds complexity. Especially since this generally means centralized storage if you want to handle failed nodes well and/or expanding/contracting clusters.

So, Akka gives you the choice. But it also cautions you (in that doc) that for most applications it is much easier to handle message delivery failures via application code than by relying on guaranteed delivery. (Both for performance reasons and because guaranteed delivery introduces its own problems like duplicates and out of order messages.)

EDIT: In response to the follow up on how reliable "at most once" messaging is, it's better to think in terms of resiliency, not "reliability". In other words, under normal circumstances both "at most once" and "at least once" will deliver a message exactly once.

What distinguishes "at most once" and "at least once" is how they handle problems. For example, imagine a sample application that sends a million messages from actor A on node A to actor B on node B. Under normal circumstances, both "at most once" and "at least once" will have a 100% success rate. But if I shutdown node B in the middle of the test run, the "at most once" application will start having network timeout errors and some messages will fail. Whereas, the "at least once" version will save all messages to persistence once sent. I could actually shutdown both node A and node B, wait a month, turn them back on and the system would recover and all messages would have been delivered.

The key thing to understand is that it's not as if the messaging protocol is inherently unreliable: there's no "how many messages will ordinarily fail" because messages don't fail under ordinary circumstances even in "at most once". If your hardware is 100% reliable and your network is 100% reliable and your software is 100% reliable, then your messages will be delivered 100% of the time with either approach. But if your network is down then you will lose 100% of your messages with "at most once" and you will lose 0% with "at least once". If your destination server crashed due to an NPE, you will lose 100% your messages with "at most once" and 0% with "at least once".

Under normal circumstances all messages will arrive. The only question is how often you experience abnormal circumstances.