C libcurl - Is it possible to get request data in raw format

Question

In Libcurl C, using CURLOPT_DEBUGFUNCTION I can set a callback function. Libcurl calls callback and gives data for

1. CURLINFO_HEADER_IN
2. CURLINFO_HEADER_OUT
3. CURLINFO_DATA_IN
4. CURLINFO_DATA_OUT
5. CURLINFO_SSL_DATA_IN
6. CURLINFO_SSL_DATA_OUT

The tx data that is given for CURLINFO_SSL_DATA_OUT is in binary format (encrypted). I need corresponding raw data (unencrypted).

Is it possible?

Answer 1

If you want to see what is actually transmitted, I would recommend using a man-in-the-middle HTTPS proxy.

The proxy is placed between your client and the server.

When a client normally communicates with a server via HTTPS, the client checks the server's certificate chain to see if it is signed by a root certificate trusted by the system.

Thus any proxy between client and server could only forward the data without the possibility to have a look into the encrypted data stream.

So how does it work, then?

Man-in-the-middle proxies intercept traffic and create a server certificate on-the-fly. So it looks to the client as if it is communicating with the server and vice versa.

However, if the client then evaluates the certificate chain, it encounters the root certificate of the proxy that it does not know and therefore does not trust, and terminates server communication immediately (if the client is configured correctly).

So that the proxy can take a look at the HTTPS data, you must include the proxy root certificate in your list of trusted root certificates on your client system.

The proxy can then show all data including the headers.

Some typical representatives of such proxies are (incomplete list):

• mitmproxy (https://mitmproxy.org)
• Charles (https://www.charlesproxy.com)
• Fiddler (https://www.telerik.com/fiddler)

Different methods have different advantages and disadvantages. HTTPS proxies are especially useful when you want to analyze request and response including header quickly and efficiently.

Demo of mitmproxy with a libcurl program

For a quick demo one can use the example program from the libcurl site: https://curl.se/libcurl/c/simple.html:

gcc -o simple simple.c -lcurl
export https_proxy=localhost:8080

In another terminal window one can start the mitmproxy. When the program is started, the request/response appears in mitmproxy and can be analyzed.

It looks like this then:

As mentioned above, you need to configure the appropriate root certificate as described on mitmproxy's website for different operating systems.

Answer 2

I would strongly recommend using the SSLKEYLOGFILE approach which allows you to monitor and capture all curl traffic (TLS and non-TLS) using an external tool like Wireshark.

No need to install any MITM software or extra certificates anywhere. Also works with applications linked to use libcurl, not only curl itself.