Curl download through Curl API is very slow

Question

I am using CURL library to download a file. We are using https:// with TLS 1.2 . Curl version used is 7.48.0. We are observing a strange problem. Curl download works very slow between a server and client for a file size of 224 MB. If we use commandline tool of curl its very fast, but same is not the case if we call "curl_easy_perform" from the application side.

Also, we used --libcurl option to check the difference between command line and our code, and there is none. We are using the same options as the Command line tool, still the download is very slow when done through direct call to curl_easy_perform.

If we do the download from some other server and same client it works fine. However, only with a particular server, we get into a download time issue. Debugging further, we found that the netstat output shows that the receive queue for tcp socket is very high. However, its not clear why this would be an issue only through our program and not through command line, even though the same options are set.

Answer 1

It turned out that the issue was because of the write call back, which was taking time with this server. In the write call back at application layer, we had done malloc, memcpy and free to copy the new data received. This was working ok with all servers except one. The difference with the connection with this server was , Curl was giving 16378 followed by 6 bytes of data in Write call back, while with other servers where it works fine, it was giving 16384 bytes of data always. This is same as rmem_max size set for the socket buffer by default on the device. Not sure why with a particular server (Wildfly) , it breaks up into 16378 bytes followed by 6 bytes. This results into continuous malloc, memcpy and free cycles for 224 MB file size download. As a result, reading from socket gets very slow resulting in slow download. In Curl commandline tool, write call back is written differently, where a file is opened and the data is written to that file by calling fwrite regularly as the data is received, followed by fclose after all the data is received. This is very fast operation compared to malloc, memcpy and free operation. So curl commandline tool works fine for the download of same file. We changed our write call back to align the same way with what curl commandline does and it solves the issue. However, the reason of dividing the data into 16378 + 6 bytes by Curl for this particular server is not clear and I am going to investigate that further.