CODEWITHSUNDEEP
messagingaeron
Welly Tambunan
Welly Tambunan

Reputation: 160

Aeron Messaging Pattern for Pub/Sub, One Way and Competing Consumer

Currently i'm in the process of assessing Aeron for our High-frequency trading use case. As i see from the documentationm Aeron is basically the transport. Is that some higher level implementation like competing consumer, fair dispatch or pattern similiar with RabbitMQ ?

Thanks

Upvotes: 1

Views: 1795

Answers (2)

mjlee
mjlee

Reputation: 3434

No. Competing consumer is not applicable to trading applications. Actually, I have seen many trading applications trying to adapt similar patterns (i.e. SEDA framework), it does not scale and worse riddle with concurrency problems.

In order for competing consumer to work, tasks are independent and can run in parallel. In trading application, you cannot do this since an order can only process one event at single point in time. We usually invert the problem - having thread affinity to specific orders and making sure each event is processed by the same thread - so you achieve sequential processing.

IMO - Aeron, 24West LBM gives is a fast and sequenced/ordered event delivery. You want to run on reliable mode ( LBM - UME and Aeron cluster ) so you won't lose any trades. Multicast is really efficient - write once and consume everywhere. This pattern is applicable in trading system since you can offload non-critical processes. You can achieve this with 'shared-memory' but you are bounded by a single physical server. You can think 'aeron' has replication of shared-memory in real-time.

Furthermore, you will run into the physical boundary - so often you have to use 'load-balacer' to direct your orders to specific severs - the same pattern as thread-affinity within the server. For a concrete example, options market data volume is very high - so you want to limit market data process/consumption. So you want the orders of related symbols to be processed by the same physical server.

Upvotes: 1

Michael Barker
Michael Barker

Reputation: 14378

The short answer is no.

The longer and more complicated answer is that you could build something on top of Aeron to implement competing consumer/fair dispatch. The primary challenge is that you need some mechanism to manage co-ordination between consumers. This gets more complex the more widely distributed the consumers are.

Single Process (multiple threads)

A single producer, multiple consumer queue would probably be sufficient for this case. With one thread reading from Aeron, copying and en-queuing the message and the consumers reading from the queue. The Agrona library (that ships with Aeron) has some fast queue implementations.

Single host

Similar to single process, you would need something similar to a queue, but implemented using shared memory or similar. There isn't anything out of the box that I'm aware of to do this.

Multiple hosts

To manage this there needs to be some sort of central broker to manage work distribution, acknowledgements, re-delivery, etc. This service would need to be redundant in order to ensure that processing can continue if the broker itself failed. You could conceivably build something like this on top of Aeron Archive and Cluster, but I wouldn't recommend it. There are other solutions that require less co-ordination overhead, e.g. partitioning by business key.

Upvotes: 3

Related Questions