Issue with CURLOPT_WRITEDATA

Question

I am using libcurl to fetch json data using GET request from a webserver.

This is my sample code:

char *DownloadedResponse;

static int writer(char *data, size_t size, size_t nmemb, char *buffer_in)
{
    if (buffer_in != NULL)  
    {
        buffer_in = new char[size*nmemb];
        strcpy(buffer_in,data);
        DownloadedResponse = buffer_in;

        return size * nmemb;  

    }

    return 0;

} 


char * DownloadJSON(string URL)
{   
    CURL *curl;
    CURLcode res;
    struct curl_slist *headers=NULL;

    curl_slist_append(headers, "Accept: application/json");  
    curl_slist_append( headers, "Content-Type: application/json");
    curl_slist_append( headers, "charsets: utf-8"); 
    curl = curl_easy_init();

    if (curl) 
    {
        curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
        curl_easy_setopt(curl, CURLOPT_URL, URL.c_str());
        curl_easy_setopt(curl, CURLOPT_HTTPGET,1); 
        curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers); 
        curl_easy_setopt(curl,CURLOPT_WRITEFUNCTION,writer);

        res = curl_easy_perform(curl);

        if (CURLE_OK == res) 
        { 

            char *ct;         
            res = curl_easy_getinfo(curl, CURLINFO_CONTENT_TYPE, &ct);

            if((CURLE_OK == res) && ct)
            {
                cout<<"\nresponse received: "<<DownloadedResponse;

            }
            else
            {
                curl_slist_free_all(headers);
                curl_easy_cleanup(curl);
                curl = NULL;
                return NULL;
            }

        }
    }

    curl_slist_free_all(headers);
    curl_easy_cleanup(curl);
        curl = NULL;


}

Here I am able to get json data in DownloadedResponse in callback "writer" of CURLOPT_WRITEFUNCTION.

But if I print using custom pointer of CURLOPT_WRITEDATA,

char *dataPointer = NULL;
CURLcode curl_easy_setopt(curl, CURLOPT_WRITEDATA, dataPointer);
cout<<dataPointer;

Output of dataPointer is empty. What is the issue here since i able to print json data in callback of CURLOPT_WRITEFUNCTION but not in the pointer of CURLOPT_WRITEDATA

Answer 1

You write a function that takes data read from the network, and writes it to where you want it.

static int writer(char *data, size_t size, size_t nmemb, char *buffer_in){
    if (buffer_in != NULL) { 
        // very bad code which is never executed
    }
    return 0;
} 

In order for that function to write the data, it has to know where to write it, so you tell it to write to NULL

char *dataPointer = NULL;
CURLcode curl_easy_setopt(curl, CURLOPT_WRITEDATA, dataPointer);

What value do you tell it to use as buffer_in? You pass it dataPointer, which is NULL, so you just told it buffer_in = NULL. I think instead you meant to say "the address of dataPointer", which would be &dataPointer.

Technically, I have answered your question now. You passed it NULL for the buffer, so the write function exited immediately. But there's more. Now you get to execute that really bad code in writer().

if (buffer_in != NULL)  
{
    // if buffer_in already has allocated memory then leak it immediately
    // create a new buffer of memory to leak later
    buffer_in = new char[size*nmemb];

    // store the data in buffer_in
    // assume it is null terminated (it is not)
    // rather than using the length we already know
    strcpy(buffer_in,data);

    // remember buffer_in? We don't use it so assign that data pointer to a global variable. 
    DownloadedResponse = buffer_in;

    // return size of this particular chunk of data
    return size * nmemb;  
}

This function MUST use the length of the data, and not assume data is null terminated (see https://curl.haxx.se/libcurl/c/CURLOPT_WRITEFUNCTION.html).

This function MUST be able to handle the data in multiple small pieces by adding them to what it has already read. You can't call new and then discard the new memory. And you can't do that anyway because you just leaked that memory -- every new must be matched with exactly one delete. In fact, you would be very well advised not to use new or delete at all, now that we have the standard library.

This function should use the buffer_in argument you give it rather than a global variable, but you can use a global variable if you want, it's just error prone. It's not literally an error like the other stuff.

The whole point of buffer_in is to give you a persistent data structure where you can accumulate the answers. It probably should be in local scope around the curl_easy_execute, so you can then just return the content from that data structure if you got CURLE_OK. I strongly recommend you write the data to std::vector, so you don't have to keep track of memory allocation. You have trouble with it, but you don't need to do it at all. Modern style says everybody has trouble with it, so just let the standard library handle it.

You claim to follow the example in the docs, which links to https://curl.haxx.se/libcurl/c/getinmemory.html If you look again, you will see what they are doing, and how your code doesn't match. In particular, they pass &chunk (the address of chunk) and then write data into chunk so they keep what was there before.

struct MemoryStruct {
  char *memory;
  size_t size;
};

static size_t
WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
    // here is where they get access to the buffer
    struct MemoryStruct *mem = (struct MemoryStruct *)userp;

In the call to curl, you will find the struct locally defined, then the remote call:

struct MemoryStruct chunk;
curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void *)&chunk);
res = curl_easy_perform(curl_handle);
if (stuff)
    printf("%lu bytes retrieved\n", (long)chunk.size);