CODEWITHSUNDEEP
c#rabbitmqqueuemessage-queueproducer-consumer
Felix Arnold
Felix Arnold

Reputation: 901

C# BigData Queue

We´ve got an REST server which runs on a seperate machine than the main application server. Now we want to shift the data amongst other things from the REST server to the main-application server, also we want to send some messages from main-server to the REST server. Therefor we evaluated MQRabbit, for the message porpose it seems very suitable. But we now wondering whether MQRabbit can proceed about 1~2 GB of data in its queues.

I´ve followed the RabbitMQ tutorials

And now we have the following code:

public class QueueController<T> : IDisposable
{
    private IModel channel;
    private IConnection connection;
    private ConnectionFactory factory = new ConnectionFactory() { HostName = "localhost" };

    public string Topic { get; private set; }

    public string LastMessage { get; private set; }

    public QueueController() 
    {
        connection = factory.CreateConnection();
        channel = connection.CreateModel();
        
        Topic = nameof(T);
    }

    public void Publish(List<T> data)
    {
        var body = Encoding.UTF8.GetBytes(LastMessage = data.SerializeJson());

        var properties = channel.CreateBasicProperties();
        properties.Persistent = true;

        channel.BasicPublish(exchange: "",
                             routingKey: $"{Topic}_queue",
                             basicProperties: properties,
                             body: body);
    }

    public void Dispose()
    {
        channel.Dispose();
        connection.Dispose();
    }
}

Als MQRabbit´s tutorials show one producer and many consumer but our case is the other way around. Many producer and one consumer. Are there some best practices for those cases?

Upvotes: 1

Views: 64

Answers (1)

Mehdi Daustany
Mehdi Daustany

Reputation: 1226

Let's first consider what a message queue does: sending messages -- small bits of data which communicate something to another computer system. The operative word here is small. Messages typically contain one of three things: 1. commands (go do something), 2. events (something happened), 3. requests (give me some data), and 4. responses (here is your data). A full discussion on these is beyond the scope, but suffice it to say that each of these can generally be composed of a small message less than 100kB.

Indeed, the AMQP protocol, which underlies RabbitMQ, is a fairly chatty protocol. It requires large messages be divided into multiple segments of no more than 131kB. This can add a significant amount of overhead to a large file transfer, especially when compared to other file transfer mechanisms (FTP, for instance). Secondly, the message has to be fully processed by the broker before it is made available in a queue, and it ties up valuable resources on the broker while this is being done. For one, the whole message must fit into RAM on the broker due to its architecture. This solution may work for one client and one broker, but it will break quickly when scaling out is attempted.

Upvotes: 1

Related Questions