CODEWITHSUNDEEP
c++centos7zeromq
Olkan
Olkan

Reputation: 31

ZeroMq: Too many open files.. Number of fd usage growing continuosly on the same object

Through the same class object which includes 2 zeromq subscriber and 1 zeromq request socket, I create objects in different threads. I use inproc zeromq sockets and that belong to same ZContext.

Each time I create the object the number of open files (lsof | wc -l) in the server (operating Centos 7) system increases incrementally. After creating the first object the open file # increases by amount of 300 and the second one increases the open file number by 304 and continuously growing.

As my programme can use many of these objects during runtime this can result in too many open files error for zeromq even though I set the limit to 524288 (ulimit -n). As the # of objects getting higher each object consumes the open file limit much more as some of them around 1500.

During runtime my programme crashes with the too many open files error at the times of many objects created and threads doing their work (sending messages to another server or clients) on the objects.

How can I overcome this through?

example code:

void Agent::run(void *ctx) {
    zmq::context_t *_context = (zmq::context_t *) ctx;
    zmq::socket_t dataSocket(*(_context),ZMQ_SUB);
    zmq::socket_t orderRequestSocket(*(_context),ZMQ_REQ);//REQ

    std::string bbpFilter = "obprice.1;
    std::string bapFilter = "obprice.2"
    std::string orderFilter = "order";
    dataSocket.connect("inproc://ordertrade_publisher");   
    dataSocket.connect("inproc://orderbook_prices_pub");      
    orderRequestSocket.connect("inproc://frontend_oman_agent");    
    int rc;
    try {         
        zmq::message_t filterMessage;
        zmq::message_t orderMessage;
        rc = dataSocket.recv(&filterMessage);
        dataSocket.recv(&orderMessage); 
        //CALCULATION AND SEND ORDER
        // end:
        return;
    }
    catch(std::exception& e) {
        std::cerr<< "Exception:" << e.what() << std::endl;        
        Order.cancel_order(orderRequestSocket);
        return;
    }
}

Upvotes: 3

Views: 2140

Answers (1)

jwm
jwm

Reputation: 1844

I'm running into this as well. I'm not sure I have a solution, but I see that a context (zmq::context_t) has a maximum number of sockets. See zmq_ctx_set for more detail. This limit defaults to ZMQ_MAX_SOCKETS_DFLT which appears to be 1024.

You might just need to increase the number of sockets your context can have, although I suspect there might be some leaking going on (at least in my case).

UPDATE:

I was able to fix my leak through a combination of socket options:

  • ZMQ_RCVTIMEO - I was already using this to avoid waiting forever if the other end wasn't there. My system handles this by only making one request on a socket, then closing it.
  • ZMQ_LINGER - set to 0 so the socket doesn't wait around trying to send the failed message. The default behavior is infinite linger. This is probably the key to your problem
  • ZMQ_IMMEDIATE - this option restricts the queueing of messages to only completed connections. Without a queue, there's no need for the socket to linger.

I can't say for sure if I need both linger and immediate, but they both seemed appropriate to my use case; they might help yours. With these options set, my number of open files does not grow infinitely.

Upvotes: 2

Related Questions