CODEWITHSUNDEEP
memory-managementerlanggen-server
Chathuri Gunawardhana
Chathuri Gunawardhana

Reputation: 171

A single erlang process eating up more memory and firing process memory high alarms

I have an erlang gen_server which receives messages from clients asynchronously(using cast). On each message handling, the server inserts them into a ordered ETS table and deletes some of them based on conditions (Because of the conditional delete, the number of records in the ETS table is in most of the time around 5000. But there are rare scenarios which can increase this size up to 200k, introducing overhead for both inserts and deletes).

This works fine until messages per second is 100k but above this it gives me the process memory high watermark and starts consuming a lot of memory. When I comment out the insert and delete part to the ordered ets, it can process more than 100k per second. Can you please give me some hints on how to solve this?

Is there any maximum limit of memory that a process can allocate? I'm using 35 GB of Memory and I set the watermark threshold by memsup:set_procmem_high_watermark(0.6). I also tried gbtree and ordered dictionary but they didn't solve the memory error.

Upvotes: 0

Views: 1692

Answers (2)

mentels
mentels

Reputation: 196

Process memory limit

There's no such option in Erlang. memsup:set_procmem_high_watermark/1 simply sets the threshold for process memory allocation. If the threshold is passed, the alarm will be raised.

What may be the cause of the problem?

At the beginning you have to find out why your process slows down. As already stated here, check characteristics of your message queue: is it growing over time? If that's the case, your process isn't able to handle the messages at the rate they are arriving. erlang:process_info/2 will be helpful.

Another thing to keep in mind is when you insert/look up data in ETS, the objects are copied around. So, if the conditions that have to be met when deleting the objects involve retrieving the data from your ETS, that also might be possible cause of your problem.

How to tackle this?

There're several ways to approach this problem. The obvious one is to simply use back pressure mechanism by, for example, making the clients sending the messages synchronously.

Altenatively you could try to split what your one process is doing into several of them. To achieve that you could have one process distributing the messages to others based on some information contained within the messages.

Finally, you could try to apply some optimisation regarding your ETS. For instance, if you're storing big objects, you could check whether you're not retrieving them back unnecessarily (by using ets:lookup/X instead of ets:match/X or ets:match_object/X to get exactly the bits you want from the stored message).

Upvotes: 0

nikit
nikit

Reputation: 153

I have a suggestion. When it comes more than 100k/s, gen_server can not process in time message queue, so it starts to grow => gen_server gets slower and memory leak . Try to monitor erlang:process_info(Pid, message_queue_len) if it grows.

Upvotes: 1

Related Questions