Issues with non-blocking server sockets using polling in MicroPython

Question

I am trying to put together a simple thermometer that provides the temperature on the OLED display as well as via http requests on an ESP8266 using MicroPython.

A Poller Object has been used to prevent the websocket from blocking the loop (so measurements and OLED display can be updated).

#CREATE SOCKET
serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serverSocket.bind(('', 80))
serverSocket.listen(5)

#REGISTER SOCKET TO THE POLLER
pollerObject = select.poll()
pollerObject.register(serverSocket, select.POLLIN)

#PERFORM FIRST MEASUREMENT AT STARTUP
last_meas_time = startup_time
sensor_readings = read_sensor()
print(sensor_readings)
display.display_measurement(str(temp),str(hum))

#LOOP FOREVER
while True:
    
    #ONE MEASUREMENT UPDATE EVERY 30s
    if(time.time() - last_meas_time >= 30):
        sensor_readings = read_sensor()
        print(sensor_readings)
        display.display_measurement(str(temp),str(hum))
        last_meas_time = time.time()
    
    #WAIT UP TO 10s FOR INCOMING CONNECTIONS
    fdVsEvent = pollerObject.poll(10000)
    
    for descriptor, Event in fdVsEvent:
        print()
        print("Got an incoming connection request")
        print("Start processing")
        # Do accept() on server socket or read from a client socket
        conn, addr = serverSocket.accept()
        print('Got a connection from %s' % str(addr))
        request = conn.recv(1024)
        print('Content = %s' % str(request))
        response = web_page()
        conn.send('HTTP/1.1 200 OK\n')
        conn.send('Content-Type: text/html\n')
        conn.send('Connection: close\n\n')
        conn.sendall(response)
        conn.close()

It seems to be working fine for some time, but I found two issues with it where I would appreciate your help:

1. Even though I connect to it only once, 2 or 3 requests are shown as received in shell terminal as you can see below. Why does that happen and how could I address it? Can it be so that the browser waited long enough to send a second or third request?

    MPY: soft reboot
    Connection successful
    ('192.168.1.74', '255.255.255.0', '192.168.1.1', '192.168.1.1')
    b'29.0,24.0'
    
    Got an incoming connection request
    Start processing
    Got a connection from ('192.168.1.64', 58581)
    Content = b'GET / HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nDNT: 1\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
    
    Got an incoming connection request
    Start processing
    Got a connection from ('192.168.1.64', 58582)
    Content = b'GET /favicon.ico HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nDNT: 1\r\nAccept: image/webp,image/apng,image/*,*/*;q=0.8\r\nReferer: http://192.168.1.74/\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'

2. After some long time running I won't be able to connect to it anymore as it will not respond. Is there something obviosly wrong with my approach? This was what I got from the console:

    Got an incoming connection request
    Start processing
    Got a connection from ('192.168.1.64', 59158)
    Content = b'GET / HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nDNT: 1\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
    
    Got an incoming connection request
    Start processing
    Got a connection from ('192.168.1.64', 59157)
    Content = b'GET /favicon.ico HTTP/1.1\r\nHost: 192.168.1.74\r\nConnection: keep-alive\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.66\r\nDNT: 1\r\nAccept: image/webp,image/apng,image/*,*/*;q=0.8\r\nReferer: http://192.168.1.74/\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: pt-BR,pt;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,sv;q=0.5\r\n\r\n'
    
    Got an incoming connection request
    Start processing
    Got a connection from ('192.168.1.64', 59160)
    Content = b''
    Traceback (most recent call last):
      File "main.py", line 104, in 
    OSError: [Errno 104] ECONNRESET
    MicroPython v1.13 on 2020-09-11; ESP module with ESP8266
    Type "help()" for more information.
    >>> 

Line 104 corresponds to:

        conn.sendall(response)

Thanks!

Answer 1

Even though I connect to it only once, 2 or 3 requests are shown as received in shell terminal as you can see below. Why does that happen and how could I address it? Can it be so that the browser waited long enough to send a second or third request?

This depends on how the browser connects to your server. There might be multiple requests the browser is looking for, or the browser has a timeout value for the socket connecting to your server. I don't have any web knowledge, but it looks like two requests for different information. How that information is handled, should be passed onto web_page(). It looks like you are sending the entirety of a web page and not the specific content it is looking for.

After some long time running I won't be able to connect to it anymore as it will not respond. Is there something obviously wrong with my approach?

What might be happening is you have socket.sendall() blocking any new sockets from being created. Also note, even though you have properly closed the socket, the socket may still have data to send. It has been marked closed, but the OS might not have closed it yet.

You are on the right track by using select.poll(). At first glance, it seems that registering your serverSocket with pollerObject (select.poll) would handle future connections. That isn't what is happening. You are registering just the one socket to pollerObject. The severSocket is getting the select.POLLIN event for the incoming connection from the browser. You need a way to add/register new sockets created by serverSocket to pollerObject so you can service other sockets.

Now the best example of what you are trying to do in micropython is to make something similar to the selector example in Python 3 Selectors.

import selectors
import socket

sel = selectors.DefaultSelector()

def accept(sock, mask):
    conn, addr = sock.accept()  # Should be ready
    print('accepted', conn, 'from', addr)
    conn.setblocking(False)
    sel.register(conn, selectors.EVENT_READ, read)

def read(conn, mask):
    data = conn.recv(1000)  # Should be ready
    if data:
        print('echoing', repr(data), 'to', conn)
        conn.send(data)  # Hope it won't block
    else:
        print('closing', conn)
        sel.unregister(conn)
        conn.close()

sock = socket.socket()
sock.bind(('localhost', 1234))
sock.listen(100)
sock.setblocking(False)
sel.register(sock, selectors.EVENT_READ, accept)

while True:
    events = sel.select()
    for key, mask in events:
        callback = key.data
        callback(key.fileobj, mask)

Generally, you won't have to worry about filling the socket transmit buffer with socket.send(), but you should handle it. For now, I would put some debug prints before and after the socket.sendall() since that will block/retry until all the data is sent. In the case that not all the data has sent, you will have to register the socket for a write ready event, and pass the remaining data that needs to be sent. This is a bit more complicated.

Got an incoming connection request
Start processing
Got a connection from ('192.168.1.64', 59160)
Content = b''
Traceback (most recent call last):
  File "main.py", line 104, in 
OSError: [Errno 104] ECONNRESET
MicroPython v1.13 on 2020-09-11; ESP module with ESP8266
Type "help()" for more information.
>>> 

The problem you are running into above is you probably have a socket connection that has timed out. TCP is letting you know the connection has expired. You should handle this with a try except else clause.