CODEWITHSUNDEEP
pythonsocketshttpwebservermicropython
joshuai96
joshuai96

Reputation: 11

MicroPython and ESP8266 http server incomplete request

I am trying to implement a simple http server based on the basic socket module of MicroPython which serve a static html and can receive and handle simple http GET and POST requests to save some data on the ESP.

I followed this tutorial https://randomnerdtutorials.com/esp32-esp8266-micropython-web-server/ and changed some parts.

webserver.py

import logging
from socket import socket, getaddrinfo, AF_INET, SOCK_STREAM, SOL_SOCKET, SO_REUSEADDR

from request import Request
from request.errors import PayloadError

log = logging.getLogger(__name__)

def __read_static_html(path: str) -> bytes:
  with open(path, "rb") as f:
    static_html = f.read()

  return static_html

def __create_socket(address: str = "0.0.0.0", port: int = 8080) -> socket:
  log.info("creating socket...")
  s = socket(AF_INET, SOCK_STREAM)
  s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)

  addr = getaddrinfo(address, port)[0][-1]
  s.bind(addr)

  log.info("socket bound on {}  ".format(addr))

  return s

def __read_from_connection(conn: socket) -> Request:
  log.info("read from connection...")
  raw_request = conn.recv(4096)

  return Request(raw_request.decode("utf-8"))

def listen_and_serve(webfile: str, address: str, port: int):
  server_socket = __create_socket(address, port)

  log.info("listen on server socket")
  server_socket.listen(5)

  while True:
    # accept connections
    client_server_connection, client_address = server_socket.accept()
    log.info("connection from {}".format(client_address))

    req = __read_from_connection(client_server_connection)

    log.info("got request: {}".format(req.get_method()))

    path = req.get_path()
    if path != '/':
      log.info("invalid path: {}".format(path))
      client_server_connection.send(b"HTTP/1.1 404 Not Found\n")
      client_server_connection.send(b"Connection: close\n\n")
      client_server_connection.close()
      continue

    if req.get_method() == "POST":
      log.info("handle post request")
      try:
        pl = req.get_payload()
        log.debug(pl)
      except PayloadError as e:
        log.warning("error: {}".format(e))
        client_server_connection.send(b"HTTP/1.1 400 Bad Request\n")
        client_server_connection.send(b"Connection: close\n\n")
        client_server_connection.close()
        continue

    log.info("read static html...")
    static_html = __read_static_html(webfile)

    log.info("send header...")
    client_server_connection.send(b"HTTP/1.1 200 OK\n")
    client_server_connection.send(b"Connection: close\n\n")

    log.info("send html...")
    client_server_connection.sendall(static_html)

    log.info("closing client server connection")
    client_server_connection.close()

The request module is my self written http request parser with minimal support for what I need.

The served html:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>ESP Basic Configuration</title>
  </head>
  <body>
    <h1>ESP Basic Configuration</h1>
    <form action="/" method="post" enctype="application/x-www-form-urlencoded">
      <h3>Network</h3>
      <div>
        <label for="network.ssid">SSID</label>
        <input id="network.ssid" name="network.ssid" type="text" />
      </div>
      <div>
        <label for="network.password">Password</label>
        <input id="network.password" name="network.password" type="password" />
      </div>
      <button type="submit">Save</button>
    </form>
  </body>
</html>

When I run the code on my system with normale Python3.9 everything seems to work.
Running the code on my ESP8266 the length of the raw_request is truncated to 536 bytes. So some request are incomplete and the payload can not be read.

I have read the socket is non-blocking by default and a short read can occure. I've tried using blocking sockets with timeout. But I got timeouts all the time, when I think there should not be one.

I have tried using the file-like socket object like shown here:
https://docs.micropython.org/en/latest/esp8266/tutorial/network_tcp.html#simple-http-server
But the request reading is stopped after the headers because of the if condition with \r\n.
Removing this condition and just checking with if not line holds the loop on the next line read.

I have currently no idea what I can do to get the full request with payload.

EDIT: add MRE This is the minimal example where I can reproduce the issue:

main.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from socket import socket, AF_INET, SOCK_STREAM, SOL_SOCKET, SO_REUSEADDR

def main():
  server_socket = socket(AF_INET, SOCK_STREAM)
  server_socket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)

  server_socket.bind(("0.0.0.0", 8080))

  server_socket.listen(5)

  while True:
    # accept connections
    client_server_connection, client_address = server_socket.accept()

    raw_request = client_server_connection.recv(4096)

    print("raw request length: {}".format(len(raw_request)))
    print(raw_request)

    client_server_connection.send(b"HTTP/1.1 200 OK\r\n")
    client_server_connection.send(b"Connection: close\r\n\r\n")

    client_server_connection.sendall(b"""<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>ESP Basic Configuration</title>
  </head>
  <body>
    <h1>ESP Basic Configuration</h1>
    <form action="/" method="post" enctype="application/x-www-form-urlencoded">
      <h3>Network</h3>
      <div>
        <label for="network.ssid">SSID</label>
        <input id="network.ssid" name="network.ssid" type="text" />
      </div>
      <div>
        <label for="network.password">Password</label>
        <input id="network.password" name="network.password" type="password" />
      </div>
      <button type="submit">Save</button>
    </form>
  </body>
</html>
""")

    client_server_connection.close()


if __name__ == "__main__":
  main()

When outputting the raw request with the print statement I get the following info:

raw request length: 536
b'POST / HTTP/1.1\r\nHost: 192.168.0.113:8080\r\nConnection: keep-alive\r\nContent-Length: 39\r\nCache-Control: max-age=0\r\nUpgrade-Insecure-Requests: 1\r\nOrigin: http://192.168.0.113:8080\r\nContent-Type: application/x-www-form-urlencoded\r\nUser-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r\nReferer: http://192.168.0.113:8080/\r\nA'

The request is still truncated to 536 bytes and the request end abruptly.

Upvotes: 1

Views: 662

Answers (1)

Steffen Ullrich
Steffen Ullrich

Reputation: 123260

def __read_from_connection(conn: socket) -> Request:
  log.info("read from connection...")
  raw_request = conn.recv(4096)

You assuming that recv will return all the data you need. This assumption is wrong. TCP has not a concept of messages but is a byte stream, so you need to recv again and again until you got all the data you need. What all data means is defined by the HTTP message format - see the HTTP standard in RFC 7230.

Why the amount of data returned by recv differs between ESP8266 and PC: probably because the ESP8266 has a smaller socket buffer and announces this as window in TCP. This makes the peer send the data in smaller packets.

Apart from that the line ending in the responses must be \r\n not \n.

But the request reading is stopped after the headers because of the if condition with \r\n. Removing this condition and just checking with if not line holds the loop on the next line read.

The empty line (\r\n) marks the end of the HTTP header. To find out the size of the body you need to parse the HTTP header and look for the length information, i.e. Content-length header for a fixed length or Transfer-Encoding: chunked when the body is sent in small chunks and the full length is not known up front. Refer to the standard for the details.

Upvotes: 3

Related Questions