CODEWITHSUNDEEP
flumeflume-ngfiware-cygnus
hiro
hiro

Reputation: 59

How can I calculate the appropriate amount of channel capacity?

I am looking for a solution because the sth-channel is full. I am troubled with calculating the appropriate capacity of channel capacity.

This document has the following description.

In order to calculate the appropriate capacity, just have in consideration the following parameters:
・The amount of events to be put into the channel by the sources per unit time (let's say 1 minute).
・The amount of events to be gotten from the channel by the sinks per unit time.
・An estimation of the amount of events that could not be processed per unit time, and thus to be reinjected into the channel (see next section).

How can I check the values of these parameters?

Upvotes: 2

Views: 178

Answers (1)

Alexis Wilke
Alexis Wilke

Reputation: 20798

How can I check the values of these parameters?

You can't just check these parameters. They depend on your application.

What they are saying is that you should have a size which is large enough so the generator doesn't get stuck. This may not be possible in your application.

Say your generator receives one event per second and it takes 2 seconds for a receiver to manage that event. Now lets assume you have 3 receivers. In 1 second, you can manage to process 0.5 events per receiver. You have 3 receivers, so your receivers, together, are capable of processing 0.5 × 3 = 1.5 events, which is more than what you get as input. Your capacity can be 1 or 2, using 2 will greatly increase your chances that you do not get blocked.

Let's review another example:

  • Your generator wants to pushes 1,000 events per second
  • Your receivers take 3 seconds to process one event
  • You would need 1,000 x 3 = 3,000 receivers (3,000 goroutines that can run at full speed in parallel...)

In this example, the total number of receivers is so large that you have to either break up your code to work on multiple computers or optimize your receiver code so it can process the data in an amount of time that makes sense. Say you have 50 processors, your receivers will get 1,000 events per second, all 50 can run at full speed, you need one receiver to do its work in:

50 / 1000 = 0.05 seconds

Now let's assume that in most cases your goroutines take 0.02 but once in a while one will take 1 second. That means your goroutines can get a little behind. In that case your capacity (so the generator doesn't get blocked) should be a little over 1,000. Again, it will depend on how many of the routines get slowed down, etc. In this last example, a run is 0.02 seconds so to process 1,000 events it usually takes 0.02 seconds. If you can send those 1,000 event over the 1 second period, you may not even need the 50 goroutines and could have a smaller capacity. On the other hand, if you have big bursts where you may end up sending many (say 500) events all at ones, then more goroutines and a larger capacity is important to not get blocked.

Upvotes: 0

Related Questions