Ryan Barrett
Reputation: 11
PICO W / Micropython websocket client can't send to PHP websocket properly
So I've been racking my brain for a few days trying to figure this one out.
In short, I have a PHP-CLI script I wrote for my project that is my websocket server. The script authenticates connections and hands off via a local socket the socket resource to another script that is the 'main' script. I run the services via systemd, everything works ok, my Android app can authenticate and use the websocket fully, my C programs on a Raspberry Pi Zero Wireless can access the websocket fully, my PHP webpage can authenticate and access the websocket server fully.
Recently I wanted to have a go getting a PICO Wireless doing portions of what my Pi Zero's do, just managing one task like controlling my lighting, or aquiring sensor data, which it would relay in the same way to my websocket as the current incarnation on the Pi Zero's do.
I setup my PICO W to access my wifi, which is fine, I have a static IP set and can ping it. I cloned 'danni/uwebsockets' using protocol.py to provide a websocket class to seemingly handle the framing and handshake for this client. I used this library for lack of finding information about anything else capable of providing the websocket client class.
The Micropython code I have going does handshake and will receive json formatted messages from the server after the handshake (I have a field 'mType' : 'AUTH REQUEST' that is sent to indicate the client should send AUTH. When I try to reply with my 'AUTH REPLY' message, the socket doesn't seem to get the entire send as one transmission, causing the unseal and subsequent json.decode to fail and my message is not received (although some data appears to be had at the servers side socket recv)
here's the relevant code portions:
This is the socket_recv that awaits incoming data after the socket upgrade / handshake:
$done = false;
$data = '';
$bytes = '';
$socketData = '';
while(!$done) {
socket_clear_error($newSocketArrayResource);
$bytes = socket_recv($newSocketArrayResource, $data, 1, MSG_DONTWAIT);
$lastError = socket_last_error($newSocketArrayResource);
if ($lastError != 11 && $lastError > 0) {
printf("lastError: %s\n",$lastError);
$done = true;
}else if ($bytes === false) {
printf("Socket: read finished\n");
$done = true;
}else if (intval($bytes) > 0) {
$socketData .= $data;
}else{
usleep(2000);
}
}
printf("socketData %s\n",strlen($socketData));
$socketMessage = $socketHandler->unseal($socketData);
printf("socketMessage %s %s\n",strlen($socketMessage),$socketMessage);
$messageObj = json_decode($socketMessage);
//printf("%s receive msg devID:%s mType:%s socketMessage: %s\n",$newSocketArrayResource, $messageObj->devID,$messageObj->mType,$socketMessage);
if(isset($messageObj->mType)&&$messageObj->mType=="AUTH"){
This is what I see in my logging for the PHP websocket server when the PICO connects:
This is when I use ws.send(msg) a preformatted json string.
Jan 03 19:28:26 rc-think php[6611]: SOCKET Resource id #13 added to client socket array
Jan 03 19:28:26 rc-think php[6611]: h:GET /chatserver HTTP/1.1
Jan 03 19:28:26 rc-think php[6611]: Host: localhost:8090
Jan 03 19:28:26 rc-think php[6611]: User-Agent: PICO-W (MICROPYTHON)
Jan 03 19:28:26 rc-think php[6611]: Sec-WebSocket-Key: Wf5IqdigmE6WoltwZki6nA==
Jan 03 19:28:26 rc-think php[6611]: Sec-WebSocket-Version: 13
Jan 03 19:28:26 rc-think php[6611]: X-Forwarded-For: 192.168.0.99
Jan 03 19:28:26 rc-think php[6611]: X-Forwarded-Host: 192.168.0.22:8090
Jan 03 19:28:26 rc-think php[6611]: X-Forwarded-Server: 127.0.1.1
Jan 03 19:28:26 rc-think php[6611]: Upgrade: WebSocket
Jan 03 19:28:26 rc-think php[6611]: Connection: Upgrade
Jan 03 19:28:26 rc-think php[6611]: [1B blob data]
Jan 03 19:28:26 rc-think php[6611]: HANDSHAKE 127.0.0.1(proxy) for:192.168.0.99 cookie:unset sessid:unset secKey:Wf5IqdigmE6WoltwZki6nA==
Jan 03 19:28:26 rc-think php[6611]: Socket: read finished
Jan 03 19:28:26 rc-think php[6611]: socketData 2
Jan 03 19:28:26 rc-think php[6611]: socketMessage 0
Jan 03 19:28:26 rc-think php[6611]: Socket: read finished
Jan 03 19:28:26 rc-think php[6611]: socketData 93
Jan 03 19:28:26 rc-think php[6611]: [44B blob data]
Jan 03 19:28:26 rc-think php[6611]: [7B blob data]
Jan 03 19:28:26 rc-think php[6611]: [18B blob data]
Jan 03 19:28:26 rc-think php[6611]: [8B blob data]
Jan 03 19:28:26 rc-think php[6611]: [6B blob data]
Jan 03 19:28:26 rc-think php[6611]: [15B blob data]
Notice the blob data over two transmissions above.
This is when I dump a dict to string using json.dumps(data)
Jan 03 19:37:00 rc-think php[6752]: SOCKET Resource id #13 added to client socket array
Jan 03 19:37:00 rc-think php[6752]: h:GET /chatserver HTTP/1.1
Jan 03 19:37:00 rc-think php[6752]: Host: localhost:8090
Jan 03 19:37:00 rc-think php[6752]: User-Agent: PICO-W (MICROPYTHON)
Jan 03 19:37:00 rc-think php[6752]: Sec-WebSocket-Key: SeC2i7a8HqaXKFM6N0De5Q==
Jan 03 19:37:00 rc-think php[6752]: Sec-WebSocket-Version: 13
Jan 03 19:37:00 rc-think php[6752]: X-Forwarded-For: 192.168.0.99
Jan 03 19:37:00 rc-think php[6752]: X-Forwarded-Host: 192.168.0.22:8090
Jan 03 19:37:00 rc-think php[6752]: X-Forwarded-Server: 127.0.1.1
Jan 03 19:37:00 rc-think php[6752]: Upgrade: WebSocket
Jan 03 19:37:00 rc-think php[6752]: Connection: Upgrade
Jan 03 19:37:00 rc-think php[6752]: [1B blob data]
Jan 03 19:37:00 rc-think php[6752]: HANDSHAKE 127.0.0.1(proxy) for:192.168.0.99 cookie:unset sessid:unset secKey:SeC2i7a8HqaXKFM6N0De5Q==
Jan 03 19:37:00 rc-think php[6752]: Socket: read finished
Jan 03 19:37:00 rc-think php[6752]: socketData 2
Jan 03 19:37:00 rc-think php[6752]: socketMessage 0
When my Android app logs in:
Jan 03 19:31:03 rc-think php[6649]: SOCKET Resource id #14 added to client socket array
Jan 03 19:31:03 rc-think php[6649]: h:GET /chatserver HTTP/1.1
Jan 03 19:31:03 rc-think php[6649]: Host: localhost:8090
Jan 03 19:31:03 rc-think php[6649]: User-Agent: Mozilla/5.0 (MYCO; MYCO-CLIENT; en-US; rv:1.0.1a)
Jan 03 19:31:03 rc-think php[6649]: Cookie: MYCOUSERSESS=2d661e2db6904710ad924e9e745f8237
Jan 03 19:31:03 rc-think php[6649]: Sec-WebSocket-Key: Qg+crE7S7cu5XWK84PSRtw==
Jan 03 19:31:03 rc-think php[6649]: Sec-WebSocket-Version: 13
Jan 03 19:31:03 rc-think php[6649]: Sec-WebSocket-Extensions: permessage-deflate
Jan 03 19:31:03 rc-think php[6649]: Accept-Encoding: gzip
Jan 03 19:31:03 rc-think php[6649]: X-Forwarded-For: 192.168.0.192
Jan 03 19:31:03 rc-think php[6649]: X-Forwarded-Host: 192.168.0.22
Jan 03 19:31:03 rc-think php[6649]: X-Forwarded-Server: 127.0.1.1
Jan 03 19:31:03 rc-think php[6649]: Upgrade: WebSocket
Jan 03 19:31:03 rc-think php[6649]: Connection: Upgrade
Jan 03 19:31:03 rc-think php[6649]: [1B blob data]
Jan 03 19:31:03 rc-think php[6649]: HANDSHAKE 127.0.0.1(proxy) for:192.168.0.192 cookie:MYCOUSERSESS=2d661e2db6904710ad924e9e745f8237 sessid:2d661e2db6904710ad924e9e745f8237 secKey:Qg+crE7S7cu5XWK84PSRtw==
Jan 03 19:31:03 rc-think php[6649]: Socket: read finished
Jan 03 19:31:03 rc-think php[6649]: socketData 138
Jan 03 19:31:03 rc-think php[6649]: socketMessage 130 {"mType":"AUTH","uID":"552-5515366868-68","uDatID":"17c55cc6598bed92f79f2d0b70356f1b","sessID":"2d661e2db6904710ad924e9e745f8237"}
Jan 03 19:31:03 rc-think php[6649]: AUTH REQUEST USER Resource id #13 for 552-5515366868-68::17c55cc6598bed92f79f2d0b70356f1b::2d661e2db6904710ad924e9e745f8237
Jan 03 19:31:03 rc-think php[6649]: MYCO:AUTH:DB: connected...
Jan 03 19:31:03 rc-think php[6649]: MYCO:AUTH:U ADDING client ::13::1::17c55cc6598bed92f79f2d0b70356f1b::2d661e2db6904710ad924e9e745f8237
Jan 03 19:31:03 rc-think php[6649]: resource(13) of type (Socket)
Jan 03 19:31:03 rc-think php[6649]: MYCO:AUTH:U socket resource
the 'danni/uwebsockets' file:
"""
Websockets protocol
"""
#import logging
import ure as re
import ustruct as struct
import urandom as random
import usocket as socket
from ucollections import namedtuple
#LOGGER = logging.getLogger(__name__)
# Opcodes
OP_CONT = const(0x0)
OP_TEXT = const(0x1)
OP_BYTES = const(0x2)
OP_CLOSE = const(0x8)
OP_PING = const(0x9)
OP_PONG = const(0xa)
# Close codes
CLOSE_OK = const(1000)
CLOSE_GOING_AWAY = const(1001)
CLOSE_PROTOCOL_ERROR = const(1002)
CLOSE_DATA_NOT_SUPPORTED = const(1003)
CLOSE_BAD_DATA = const(1007)
CLOSE_POLICY_VIOLATION = const(1008)
CLOSE_TOO_BIG = const(1009)
CLOSE_MISSING_EXTN = const(1010)
CLOSE_BAD_CONDITION = const(1011)
URL_RE = re.compile(r'(wss|ws)://([A-Za-z0-9-\.]+)(?:\:([0-9]+))?(/.+)?')
URI = namedtuple('URI', ('protocol', 'hostname', 'port', 'path'))
class NoDataException(Exception):
pass
class ConnectionClosed(Exception):
pass
class Websocket:
"""
Basis of the Websocket protocol.
This can probably be replaced with the C-based websocket module, but
this one currently supports more options.
"""
is_client = True
def __init__(self, sock):
self.sock = sock
self.open = True
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
self.close()
def settimeout(self, timeout):
self.sock.settimeout(timeout)
def read_frame(self, max_size=None):
"""
Read a frame from the socket.
See https://tools.ietf.org/html/rfc6455#section-5.2 for the details.
"""
# Frame header
two_bytes = self.sock.read(2)
if not two_bytes:
raise NoDataException
byte1, byte2 = struct.unpack('!BB', two_bytes)
# Byte 1: FIN(1) _(1) _(1) _(1) OPCODE(4)
fin = bool(byte1 & 0x80)
opcode = byte1 & 0x0f
# Byte 2: MASK(1) LENGTH(7)
mask = bool(byte2 & (1 << 7))
length = byte2 & 0x7f
if length == 126: # Magic number, length header is 2 bytes
length, = struct.unpack('!H', self.sock.read(2))
elif length == 127: # Magic number, length header is 8 bytes
length, = struct.unpack('!Q', self.sock.read(8))
if mask: # Mask is 4 bytes
mask_bits = self.sock.read(4)
try:
data = self.sock.read(length)
except MemoryError:
# We can't receive this many bytes, close the socket
print("Frame of length %s too big. Closing",length)
self.close(code=CLOSE_TOO_BIG)
return True, OP_CLOSE, None
if mask:
data = bytes(b ^ mask_bits[i % 4]
for i, b in enumerate(data))
return fin, opcode, data
def write_frame(self, opcode, data=b''):
"""
Write a frame to the socket.
See https://tools.ietf.org/html/rfc6455#section-5.2 for the details.
"""
fin = True
mask = self.is_client # messages sent by client are masked
length = len(data)
# Frame header
# Byte 1: FIN(1) _(1) _(1) _(1) OPCODE(4)
byte1 = 0x80 if fin else 0
byte1 |= opcode
# Byte 2: MASK(1) LENGTH(7)
byte2 = 0x80 if mask else 0
if length < 126: # 126 is magic value to use 2-byte length header
byte2 |= length
self.sock.write(struct.pack('!BB', byte1, byte2))
elif length < (1 << 16): # Length fits in 2-bytes
byte2 |= 126 # Magic code
self.sock.write(struct.pack('!BBH', byte1, byte2, length))
elif length < (1 << 64):
byte2 |= 127 # Magic code
self.sock.write(struct.pack('!BBQ', byte1, byte2, length))
else:
raise ValueError()
if mask: # Mask is 4 bytes
mask_bits = struct.pack('!I', random.getrandbits(32))
self.sock.write(mask_bits)
data = bytes(b ^ mask_bits[i % 4]
for i, b in enumerate(data))
self.sock.write(data)
def recv(self):
"""
Receive data from the websocket.
This is slightly different from 'websockets' in that it doesn't
fire off a routine to process frames and put the data in a queue.
If you don't call recv() sufficiently often you won't process control
frames.
"""
assert self.open
while self.open:
try:
fin, opcode, data = self.read_frame()
except NoDataException:
return ''
except ValueError:
print("Failed to read frame. Socket dead.")
self._close()
raise ConnectionClosed()
if not fin:
raise NotImplementedError()
if opcode == OP_TEXT:
return data.decode('utf-8')
elif opcode == OP_BYTES:
return data
elif opcode == OP_CLOSE:
self._close()
return
elif opcode == OP_PONG:
# Ignore this frame, keep waiting for a data frame
continue
elif opcode == OP_PING:
# We need to send a pong frame
print("Sending PONG")
self.write_frame(OP_PONG, data)
# And then wait to receive
continue
elif opcode == OP_CONT:
# This is a continuation of a previous frame
raise NotImplementedError(opcode)
else:
raise ValueError(opcode)
def send(self, buf):
"""Send data to the websocket."""
assert self.open
if isinstance(buf, str):
opcode = OP_TEXT
buf = buf.encode('utf-8')
elif isinstance(buf, bytes):
opcode = OP_BYTES
else:
raise TypeError()
self.write_frame(opcode, buf)
def close(self, code=CLOSE_OK, reason=''):
"""Close the websocket."""
if not self.open:
return
buf = struct.pack('!H', code) + reason.encode('utf-8')
self.write_frame(OP_CLOSE, buf)
self._close()
def _close(self):
print("Connection closed")
self.open = False
self.sock.close()
the unseal function in my php:
function unseal($socketData) {
$length = ord($socketData[1]) & 127;
if($length == 126) {
$masks = substr($socketData, 4, 4);
$data = substr($socketData, 8);
}
elseif($length == 127) {
$masks = substr($socketData, 10, 4);
$data = substr($socketData, 14);
}
else {
$masks = substr($socketData, 2, 4);
$data = substr($socketData, 6);
}
$socketData = "";
for ($i = 0; $i < strlen($data); ++$i) {
$socketData .= $data[$i] ^ $masks[$i%4];
}
return $socketData;
}
the current micropython 'send auth' function
def ws_send_auth():
global ws
global devID
global devName
global devType
print("sending AUTH")
data = {
"mType":"AUTH",
"devID":"98",
"devName":"thisName",
"devType":"thisType"
}
msg = '{"message":"AUTH REPLY","mType":"AUTH","devID":"%s","devName":"%s","devType":"%s"}' % (devID,devName,devType)
#ws.send("\r\n\r\n")
#ws.send(msg)
ws.send(json.dumps(data))
anybody have an idea why 'danni/uwebsocket' sends aren't being received properly when all the other languages seem to be able to send? Is this a buffer size / transmission size issue on the PICO that my non-blocking socket isn't handling right? (I've tried buffer sizes on the non-blocking socket of 1-4096 with no effect) Or maybe the write_frame isn't doing what it needs to? I did notice that the socket.write in write_frame occurs over several calls, could this be causing the problem?
UPDATE !!!! :)
I modified the write_frame function from 'danni/uwebsockets' because I suspected that it's multiple writes to the socket was causing the behaviour I had been experiencing, which has resolved the issue
def write_frame(self, opcode, data=b''):
"""
Write a frame to the socket.
See https://tools.ietf.org/html/rfc6455#section-5.2 for the details.
"""
fin = True
mask = self.is_client # messages sent by client are masked
length = len(data)
# Frame header
# Byte 1: FIN(1) _(1) _(1) _(1) OPCODE(4)
byte1 = 0x80 if fin else 0
byte1 |= opcode
# Byte 2: MASK(1) LENGTH(7)
byte2 = 0x80 if mask else 0
random_bits = random.getrandbits(32)
if length < 126: # 126 is magic value to use 2-byte length header
byte2 |= length
thestruct = struct.pack('!BBI', byte1, byte2, random_bits)
elif length < (1 << 16): # Length fits in 2-bytes
byte2 |= 126 # Magic code
thestruct = struct.pack('!BBHI', byte1, byte2, length, random_bits)
elif length < (1 << 64):
byte2 |= 127 # Magic code
thestruct = struct.pack('!BBQI', byte1, byte2, length, random_bits)
else:
raise ValueError()
if mask: # Mask is 4 bytes
mask_bits = struct.pack('!I', random_bits)
data = bytes(b ^ mask_bits[i % 4]
for i, b in enumerate(data))
self.sock.write(thestruct + data)
notice now it does one write at the end with all the framing bytes, mask, and data.
now I see the correct stuff being had after the frame unpack and json_decode
Jan 04 12:04:19 rc-think php[3613]: SOCKET Resource id #13 added to client socket array
Jan 04 12:04:19 rc-think php[3613]: h:GET /chatserver HTTP/1.1
Jan 04 12:04:19 rc-think php[3613]: Host: localhost:8090
Jan 04 12:04:19 rc-think php[3613]: User-Agent: PICO-W (MICROPYTHON)
Jan 04 12:04:19 rc-think php[3613]: Sec-WebSocket-Key: g0Uth/7Y6OVZqFr8aqGgrg==
Jan 04 12:04:19 rc-think php[3613]: Sec-WebSocket-Version: 13
Jan 04 12:04:19 rc-think php[3613]: X-Forwarded-For: 192.168.0.99
Jan 04 12:04:19 rc-think php[3613]: X-Forwarded-Host: 192.168.0.22:8090
Jan 04 12:04:19 rc-think php[3613]: X-Forwarded-Server: 127.0.1.1
Jan 04 12:04:19 rc-think php[3613]: Upgrade: WebSocket
Jan 04 12:04:19 rc-think php[3613]: Connection: Upgrade
Jan 04 12:04:19 rc-think php[3613]: [1B blob data]
Jan 04 12:04:19 rc-think php[3613]: HANDSHAKE 127.0.0.1(proxy) for:192.168.0.99 cookie:unset sessid:unset secKey:g0Uth/7Y6OVZqFr8aqGgrg==
Jan 04 12:04:19 rc-think php[3613]: Socket: read finished
Jan 04 12:04:19 rc-think php[3613]: socketData 84
Jan 04 12:04:19 rc-think php[3613]: socketMessage 78 {"mType": "AUTH", "devName": "thisName", "devType": "thisType", "devID": "98"}
Jan 04 12:04:19 rc-think php[3613]: resource(13) of type (Socket)
Jan 04 12:04:19 rc-think php[3613]: MYCO:AUTH:D socket resource
Jan 04 12:04:19 rc-think php[3613]: AUTH REQUEST DEVICE Resource id #13 98 thisName thisType
now we can move along to getting the websocket recv on the PICO's second coretask
Upvotes: 0
Views: 65
Answers (0)
Related Questions
- How to get the client IP address in PHP
- How do I send a POST request with PHP?
- Raspberry Pi Pico (Micropython)
- Run more than one thread in micropython(pico w)
- Issues Connecting Client from MicroPython to Raspberry Pi Pico Server
- micropython client from raspberry pi pico cannot connect to local host
- How exactly does the websocket fragmention work?