CODEWITHSUNDEEP
c++curllibcurl
Graeme
Graeme

Reputation: 4592

Handling HTTP responses from libcurl over multiple callbacks

I'm using libcurl to send HTTP post requests and subsequently process the responses. Upon a response being received the library calls back to my program using the function given to the CURLOPT_WRITEFUNCTION property via curl_easy_setopt. 

// callback function
std::size_t on_data(const char* buffer, const std::size_t size, const std::size_t nmemb, void* context);

It's unclear from the documentation wither you can leave data on the buffer (by returning 0 from the callback) or wither you have to copy the data to a local buffer and then continue appending to this on each subsequent callback until you have received the entire message.

My questions are:

  1. What is the best way to get the entire message length; is it from the Content-Length header via CURLOPT_WRITEHEADER?
  2. Is there a way to avoid copying partial responses, instead waiting for the final message to be received and then processing this in it's entirety?

Upvotes: 2

Views: 2912

Answers (1)

deltheil
deltheil

Reputation: 16121

As far as the CURLOPT_WRITEFUNCTION option is concerned the documentation explicitly states that this function must:

[r]eturn the number of bytes actually taken care of. If that amount differs from the amount passed to your function, it'll signal an error to the library. This will abort the transfer and return CURLE_WRITE_ERROR.

So unless you want to explicitly abort the transfer you should always return the real size, i.e. size * nmemb. Also if you choose to work with a local buffer, then yes, you have to take care to copy the incoming data into it (and take care of reallocation) as illustrated by the docs/examples/getinmemory.c sample code.

Please note that if you do not want to use an in-memory buffer you can also use a file via the CURLOPT_WRITEDATA option, as illustrated e.g. by docs/examples/url2file.c or docs/examples/fopen.c examples.

Otherwise:

  1. I would say that to check a priori what's the size of the resource you're willing to fetch, then you should perform a HEAD request (via CURLOPT_NOBODY set to 1 and CURLOPT_HEADER set to 1 too) asking for the HTTP headers to be written via CURLOPT_WRITEHEADER and CURLOPT_WRITEFUNCTION and at last parse the Content-Length value. But: this is definitely not the way to go with a POST request as it is not idempotent!

  2. Here again if you are not comfortable with working with a growing in-memory buffer (e.g. maybe because you expect very large responses), then you should use the ability to work with files with CURLOPT_WRITEDATA so that the response is progressively written on-disk. Once done, then use your file - that contains the entire message, the way you want.

Upvotes: 1

Related Questions