CODEWITHSUNDEEP
cmultithreadingcurl
sudo
sudo

Reputation: 5794

cURL Returns Error Due to Too Many Threads (or Sockets?) [C]

I'm writing a server that uses a thread pool. Each thread connects to an SQL database then calls accept on the same server socket once it starts and waits for a client to connect. I've found that if and only if I spawn too many threads, the curl_easy_perform function fails the very first time it is called, returning error code 56 (CURL_RECV_ERROR). The limit is somewhere between 950 and 970 threads, but my computer allows me to create up to 2047 threads for the program, so this limit seems arbitrary to me. I also am not low on RAM.

Just one thread calls the cURL function once before it gives me the error, so I'm not doing multiple cURL requests simultaneously in this test. Am I doing anything wrong, or do I have to accept a thread limit for cURL to work?

These are my functions for sending HTTPS requests and getting responses with cURL:

size_t write_data(void *ptr, size_t size, size_t nmemb, struct url_data *s){ // callback function I have to use with Curl
    size_t new_len = s->len + size*nmemb;
    s->ptr = erealloc(s->ptr, new_len+1);
    memcpy(s->ptr+s->len, ptr, size*nmemb);
    s->ptr[new_len] = '\0';
    s->len = new_len;
    return size*nmemb;
}

char* curlHTTPS(char* url){ // returns the response as a string or NULL upon error
    url = strdup(url);
    CURL* curl = curl_easy_init();
    struct url_data* response = emalloc(sizeof(struct url_data));
    response->len = 0;
    response->ptr = emalloc(4096*sizeof(char));
    curl_easy_setopt(curl, CURLOPT_URL, url); // Define target site
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, true);
    curl_easy_setopt(curl, CURLOPT_USERAGENT, userAgent);
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 2);
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
    curl_easy_setopt(curl, CURLOPT_HEADER, false);
    curl_easy_setopt(curl, CURLOPT_HTTPGET, 1);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, response);
    CURLcode curlCode = curl_easy_perform(curl);
    if (curlCode!=CURLE_OK){
        if (logLevel >= LOG_ERROR) printf("Curl failed! Error: %d\n", curlCode);
        curl_easy_cleanup(curl);
        efree(response->ptr);
        efree(response);
        return NULL;
    }
    char* ret = strdup(response->ptr);
    curl_easy_cleanup(curl);
    efree(response);
    return ret;
}

Upvotes: 0

Views: 542

Answers (1)

dohashi
dohashi

Reputation: 1841

Sounds like you are opening lots of file handles. Have you checked your open file limit?

Socket accept - "Too many open files"

Upvotes: 1

Related Questions