CODEWITHSUNDEEP
erlang
akonsu
akonsu

Reputation: 29576

Terminate process if it is slow at receiving messages

I have a data source process that sends messages to worker processes. To keep memory consumption under control, I need to terminate the workers that are slow retrieving their messages from their mailboxes.

I am new to Erlang, I would appreciate any pointers. If this is difficult to accomplish with Erlang messages, maybe I can use sockets? If so, are there examples?

EDIT:

I have a registered process that reads from the web and generates a lot of data. It sends these data to all the "subscribed" processes using Erlang messages. For each particular piece of data, it sends the same message to all subscribers.

I also have a web server that streams the data that the registered process reads. So, when a http client connects, the web server creates a process and this process subscribes to the registered process and starts receiving its messages.

The registered process uses monitors to monitor subscribers. The subscribers are controlled by the web server, and when a connection is closed, the process that were serving this connection, dies.

There is no acknowledgement, that is, subscribers do not respond when a message is sent to them. Although I can program them this way, but I think it is too much traffic.

Basically I want to close the connection if a http client is too slow.

Upvotes: 1

Views: 297

Answers (3)

akonsu
akonsu

Reputation: 29576

I ended up using a separate process that controls the number of messages received by the target process. For that I use https://github.com/ferd/pobox

Upvotes: 0

Vinod
Vinod

Reputation: 2243

I would suggest you to use ets to store the data which is being read by the registered process and send only the key to the subscriber processes. All the subscribers can now read from the ets table whenever then can. In this way large amount of data in multiple messages will not reside in the process message queue thus reducing the memory consumption.

Even though copying of data to each process still exist when you read from the ets table, only that message which is being processed by the subscriber is copied.

You can keep the message reference in the subscriber process state and based on the the number of messages pending you can decide whether the subscriber is running slowly (slow http client) and you can remove the connection.

Now you can send acknowledgement back to the registered process (or any other process whose job is to accumulate the acknowledgements and remove message from the queue) so that it can remove the message from the ets (ets is not garbage collected so needs to be delete from ets).

Upvotes: 3

sitifensys
sitifensys

Reputation: 2034

Messages passing in Erlang is great, but it can also become one of its weak points. You can quickly overwhelm a process if you send more messages than it can cope with.

A solution would be to inspect the mailbox size of the controlling process using erlang:process_info/2 :

process_info(self(),message_queue_len).

This can help you decide if your process is too slow, and kill it if you so desire.

Of course this is not the only problem, and I don't know exactly how your server is designed, so I'll just add some suggestions that may or may not be useful to you.

  1. The erlang VM will copy each message in the receiver's mailbox, only binary references are not, so to keep memory as low as possible, make sure you understand and make good use of binaries.
  2. Don't forget that the tcp stack is also implemented using processes, so the real bottleneck can be in the underlying libraries.
  3. Make sure to profile your code (that's especially true for Erlang), only profiling can tell you where the real optimizations are needed.

Upvotes: 3

Related Questions