CODEWITHSUNDEEP
multithreadingc++11urllibcurl
ASamboy2
ASamboy2

Reputation: 11

Which is more efficient: curl_easy_perform() in a multi-threaded program or curl_multi_perform() in a single threaded program?

I am working on a program where I am required to download a large amount of JSON files from different URLs.

Currently, my program creates multiple threads, and in each thread, it calls the LibCurl easy_perform() function but I am running into issues where the program fails occasionally with an error of "double free". It seems to be some sort of Heisenbug but I have been able to catch it in GDB which confirms the error originates in LibCurl (backtraced).

While I would love suggestions on the issue I am having, my actual question is this: Would it be better if I were to change the structure of my code to use the LibCurl Multi Interface on one thread instead of calling the single interface across multiple threads? What are the trade offs of using one over the other?

Note: By "better", I mean is it faster and less taxing on my CPU? Is it more reliable as the multi interface was designed for this?

EDIT:

The three options I have as I understand it are these:

1) Reuse the same easy_handle in a single thread. The connections wont need to be reestablished making it faster.

2) Call curl_easy_perform() in each individual thread. They all run in parallel, again, making it faster.

3) Call curl_multi_perform() in a single thread. This is non-blocking so I imagine all of the files are downloaded in parallel making it faster?

Which of these options is the most time efficient?

Upvotes: 1

Views: 2343

Answers (1)

Anty
Anty

Reputation: 1516

curl_easy_perform is blocking operation. That means if you run in in one thread you have to download files sequentially. In multithreaded application you can run many operations in parallel - this usually means faster download time (if speed is not limited by network or destination server).

But there is non-blocking variant that may work better for you if you want to go single threaded way - curl_multi_perform

From curl man

You can do any amount of calls to curl_easy_perform while using the same easy_handle. If you intend to transfer more than one file, you are even encouraged to do so. libcurl will then attempt to re-use the same connection for the following transfers, thus making the operations faster, less CPU intense and using less network resources. Just note that you will have to use curl_easy_setopt between the invokes to set options for the following curl_easy_perform.

In short - it will give few benefits you want vs curl_easy_perform.

Upvotes: 1

Related Questions