Curl doesn't send entire form-data in HTTP POST request

Question

Edit:

Problem: 2 and Problem: 3 solved by following @melpomene comment i.e., by using number of bytes read to print the buffer.

But still struck on Problem: 1.

---

I have written a TCP server-client program. Later out of curiosity, I want to know about HTTP server.

My previous question: Simple TCP server can't output to web browser

Now I'm just seeing what and how the data is transferred to the server by using GET and POST( form-data and x-www-form-urlencoded for now).

I'm following How to cURL POST from the Command Line to send POST requests.

When I send x-www-form-urlencoded as:

curl -d "data=example1&data2=example2" localhost:8080

Output on Server:

POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.54.0
Accept: */*
Content-Length: 28
Content-Type: application/x-www-form-urlencoded

data=example1&data2=example2

This is as expected.

---

Problem: 1

Now comes the problem. When I try to send form-data, the output is not expected.

When I send form-data as:

curl -X POST -F "name=user" -F "password=test" localhost:8080

Output on server:

POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.54.0
Accept: */*
Content-Length: 244
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------78b7f8917ad1992c

I'm getting the boundary but I'm not getting the next part like the data I'm sending.

Problem: 2

One more odd thing is when I try to send x-www-form-urlencoded after sending form-data.

When I send x-www-form-urlencoded after form-data as:

curl -d "data=example1&data2=example2" localhost:8080

Output on server:

POST / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.54.0
Accept: */*
Content-Length: 28
Content-Type: application/x-www-form-urlencoded

data=example1&data2=example2------------78b7f8917ad1992c

Why am I getting boundary here?

Problem: 3

And also while sending GET as:

curl localhost:8080

Output on server:

GET / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.54.0
Accept: */*

ontent-Length: 28
Content-Type: application/x-www-form-urlencoded

data=example1&data2=example2------------78b7f8917ad1992c

I'm getting Content-Type and x-www-form-urlencoded data along with boundary.

What am I doing wrong? Is something wrong with my code or with my understanding?

Server.c:

// Server side C program to demonstrate Socket programming
#include <stdio.h>
#include <sys/socket.h>
#include <unistd.h>
#include <stdlib.h>
#include <netinet/in.h>
#include <string.h>

#define PORT 8080
int main(int argc, char const *argv[])
{
    int server_fd, new_socket; long valread;
    struct sockaddr_in address;
    int addrlen = sizeof(address);
    char buffer[1024] = {0};
    char *hello = "HTTP/1.1 200 OK\nContent-Type: text/plain\nContent-Length: 12\n\nHello world!";

    // Creating socket file descriptor
    if ((server_fd = socket(AF_INET, SOCK_STREAM, 0)) == 0)
    {
        perror("In socket");
        exit(EXIT_FAILURE);
    }


    address.sin_family = AF_INET;
    address.sin_addr.s_addr = INADDR_ANY;
    address.sin_port = htons( PORT );


    if (bind(server_fd, (struct sockaddr *)&address, sizeof(address))<0)
    {
        perror("In bind");
        exit(EXIT_FAILURE);
    }
    if (listen(server_fd, 10) < 0)
    {
        perror("In listen");
        exit(EXIT_FAILURE);
    }
    while(1)
    {
        printf("\n+++++++ Waiting for new connection ++++++++\n\n");
        if ((new_socket = accept(server_fd, (struct sockaddr *)&address, (socklen_t*)&addrlen))<0)
        {
            perror("In accept");
            exit(EXIT_FAILURE);
        }
        valread = read( new_socket , buffer, 1024);
        printf("%s\n",buffer );
        write(new_socket , hello , strlen(hello));
        printf("------------------Hello message sent-------------------\n");
        close(new_socket);
    }
    return 0;
}

Answer 1

Problem 1

I have checked the request headers and found Expect: 100-continue header. This is the first time I have seen this header.

A simple search in Google shows this is causing the problem.

Expect: 100-Continue' Issues and Risks (I'm just gonna paste everything to avoid dead link)

How the Expect: 100-Continue Header Works

When Expect: 100-Continue is NOT present, HTTP follows approximately the following flow (from the client's point of view):

1. The request initiates a TCP connection to the server.

2. When the connection to the server is established, the full request--which includes both the request headers and the request body--is transmitted to the server.

3. The client waits for a response from the server (comprised of response headers and a response body).

4. If HTTP keep-alives are supported, the request is optionally repeated from step 2.

When the client is using the Expect: 100-Continue feature, the following events occur:

1. The request initiates a TCP connection to the server.

2. When the connection to the server is established, the request--including the headers, the Expect: 100-Continue header, without the request body--is then transmitted to the server.

3. The client then waits for a response from the server. If the status code is a final status code, using the prior steps above the client retries the request without Expect: 100-Continue header. If the status code is 100-Continue, the request body is sent to the server.

4. The client will then wait for a response from the server (comprised of response headers and a response body).

5. If HTTP keep-alives are supported, the request is optionally repeated from step 2.

Why use Expect: 100-Continue?

API POST requests that include the Expect: 100-Continue header save bandwidth between the client and the server, because the server can reject the API request before the request body is even transmitted. For API POST requests with very large request bodies (such as file uploads), the server can, for example, check for invalid authentication and reject the request before the push body was sent, resulting in significant bandwidth savings.

Without Expect: 100-Continue:

Without the Expect: 100-Continue feature, the entire API request, including the (potentially large) push body would have to be transmitted before the server could even determine if the syntax or authentication is valid. However, since the majority of our API requests have small POST bodies, the benefits of separating the request header from the request body is negligible.

Problems when the request header and body are sent separately

Because of the high volume of requests that Urban Airship handles, many levels of complexity exist between our clients and the servers responsible for responding to API requests. This is not an abnormal phenomenon for most server configurations and strategies, but it does introduce a risk of elevated request failures to any API POST requests using the Expect: 100-Continue header. This is due to the fact that the request header and the request body are sent separately from one another, and must travel through the same connection throughout the entire API server infrastructure.

With the number of proxies, load-balancing servers, and back-end request processing servers that are implemented, requests with the Expect: 100-Continue header have an increased probability of becoming separated from one another, and hence returning with an error.

What To Expect:

We've always attempted to support Expect: 100-Continue. However, we have determined that our customers that use Expect: 100-Continue are receiving a sub-optimal quality of service due to elevated request failures.

Additionally, the majority of our API requests have small POST bodies, and as a result the benefits of separating the request header from the request body are negligible. These reasons have motivated us to disable support for Expect: 100-Continue service-wide.

Our Recommendations:

We recommend against the use of Expect: 100-Continue. If you receive an HTTP Error 417 (Expectation failed), retry the request without Expect: 100-Continue.

So, to prevent Expect: 100-continue header in POST form-data, include -H 'Expect:' in your `curl

curl -X POST -F "name=user" -F "password=test" localhost:8080 -H 'Expect:'

Now you can receive your entire data in one go(just like Postman) as you said in comments.

---

Problem 2 & 3

As @melpomene said in comments, read() doesn't put \0 after reading. That's why you are seeing data from previous requests.

So, just use valread to iterate over string to print or just declare variable in your while loop as I said in the comments.

Code:

while(1)
    {
        printf("\n+++++++ Waiting for new connection ++++++++\n\n");
        if ((new_socket = accept(server_fd, (struct sockaddr *)&address, (socklen_t*)&addrlen))<0)
        {
            perror("In accept");
            exit(EXIT_FAILURE);
        }

        char buffer[30000] = {0};  // This way you get new variable everytime. So, there is no need to iterate over the string using valread value.
        valread = read( new_socket , buffer, 30000);
        printf("%s\n",buffer );
        write(new_socket , hello , strlen(hello));
        printf("------------------Hello message sent-------------------%lu\n", valread);
        close(new_socket);
    }