RabbitMQ tracking message processing

Question

Background

I using RabbitMQ as a distributing queue. This takes place part of Micro-services architecture.

My software architecture looks like this:

The communication between all services (1 to 3) is via messages in queues (rabbit-MQ). Each red arrow is a message enqueued into RabbitMQ.

The problem:

I want to calculate the executing time of each service. It should create a result like this:

As you can see, although is the same service, the executing of the queue message is different from time to time (depends on the data).

I want to create a visual way to show those chains, in order to understand better what is my most critical supply chain?

Ideas to solution

Each message on the queue has a unique message ID. Follow:

• Add the message another property - when it was enqueued to queue.
• Insert the message to a queue and store it in the database. In the database, we will show the message-id, target service, and the parent queue message-id (if has).
• The target service, who dequeue the message, track it's processing time (how much took) at the end - update the data.

As you can see, the target goal is to make this solution generic.

Questions:

1. Do you have tips to make it even more generic?
2. Maybe RabbitMQ can provide this summary itself?
3. Any other tip will be welcomed.

Answer 1

What you are looking for is a distributed tracing tool. There are at least two tools which can help you to achieve your goals: Zipkin and Jaeger. Both are compatible with OpenTracing standard. The overall architecture of both tools is similar.

About OpenTracing:

OpenTracing is a new, open distributed tracing standard for applications and OSS packages.

At the end you would get something like this (zipkin):

Unfortunately, they do not have an out-of-the-box solution for amqp. It would be necessary to integrate tracing yourself for amqp messages.

As a start, I would suggest reading about zipkin. Then dig into how zipkin works with consumers and producers, which is a relatively new feature. Also to look at implementation for kafka may help to clarify a lot.

Trace, which includes amqp messages and http requests, might look like this:

So, your questions:

Do you have tips to make it even more generic?

There is already a standard for it

Maybe RabbitMQ can provide this summary itself?

Unfortunately no