CODEWITHSUNDEEP
python-3.xamazon-web-servicesimage-processingmqttaws-iot
Anudocs
Anudocs

Reputation: 686

Sending image as numpy array over MQTT to AWS IoT Core

I am capturing images from webcam and sending those images as numpy arrays to AWS IoT Core using AWS IoT Device Python SDK. At AWS IoT Core, this numpy array is transferred to a lamda function for image processing(via RESNET model).

But numpy arrays's size is too big to send over MQTT and thats why AWS IoT Core is not receiving it.

So my question is:

  1. Is there a way to increase MQTT TX buffer size in SDK? I couldn't find any.
  2. If not, then what is the best way to send a big numpy array over MQTT and receiving it?

Publishing code:

    cam = cv2.VideoCapture(0)
    ret, frame = cam.read()
    message['image'] = frame.tolist() #numpy array converted to list to make it work with json.dumps
    messageJson = json.dumps(message)
    myAWSIoTMQTTClient.publish(topic, messageJson, 0)

Frame numpy array:

[0:480] :[array([[156, 168, 20...ype=uint8), array([[155, 167, 20...ype=uint8), array([[144, 168, 20...ype=uint8), array([[144, 168, 20...ype=uint8), array([[138, 168, 20...ype=uint8), array([[138, 168, 20...ype=uint8), array([[149, 170, 20...ype=uint8), array([[151, 172, 20...ype=uint8), array([[156, 174, 20...ype=uint8), array([[156, 174, 20...ype=uint8), array([[153, 174, 20...ype=uint8), array([[152, 173, 20...ype=uint8), array([[153, 172, 20...ype=uint8), array([[154, 173, 20...ype=uint8), ...]
dtype:dtype('uint8')
max:222
min:0
shape:(480, 640, 3)
size:921600

Upvotes: 1

Views: 1420

Answers (1)

Mark Setchell
Mark Setchell

Reputation: 207668

You seem a bit stuck. I don't know the AWS side of things, but can show you the imaging side. I am suggesting you convert your image frame (which is a Numpy array) into a JPEG image for transmission because that will take less bandwidth and result in smaller MQTT messages. However, a JPEG is binary so you can't send it as JSON unless you base64-encode it first. In my example below, I show a way to take your image (Numpy array) and:

  • convert it to JPEG. Note that you could equally use PNG - especially if you want lossless - but that will be slower normally. Note you could equally use PIL/Pillow instead of OpenCV. Note that you can vary the quality too, which will affect size.
  • base64-encode it
  • convert to JSON

then I show the reverse process for the receiving end. The actual size/bandwith saving will depend on your image and how compressible it is and the quality loss you are prepared to accept, however I started with a 920kB image and actually represent it in 66kB of JSON.

#!/usr/bin/env python3

import cv2
import numpy as np
from base64 import b64encode

# Get any old image, 640x480 pixels - corresponds to your "frame"
na = cv2.imread('start.jpg', cv2.IMREAD_COLOR)
print(f'DEBUG: Size as Numpy array: {na.nbytes}')

# Convert to "in-memory" JPEG
_, JPEG = cv2.imencode(".jpg", na, [int(cv2.IMWRITE_JPEG_QUALITY), 80])
print(f'DEBUG: Size as JPEG: {JPEG.nbytes}')
JPEG.tofile('DEBUG-original.jpg')

# Base64 encode
b64 = b64encode(JPEG)
print(f'DEBUG: Size as base64: {len(b64)}')
print(f'DEBUG: Start of base64: {b64[:32]}...')

# JSON-encode
message = { "image": b64.decode("utf-8") }
messageJSON = json.dumps(message)
print(f'DEBUG: Start of JSON: {messageJSON[:32]}')

Sample Output

DEBUG: Size as Numpy array: 921600
DEBUG: Size as JPEG: 49456
DEBUG: Size as base64: 65944
DEBUG: Start of base64: b'/9j/4AAQSkZJRgABAQAAAQABAAD/2wBD'...
DEBUG: Start of JSON: {"image": "/9j/4AAQSkZJRgABAQAAA

The receiver side will look like this:

### RECEIVER: All variables suffixed with '_r' to denote receiver ###

import cv2
import numpy as np
from base64 import b64decode

# Extract base64 from JSON
b64_r = json.loads(messageJSON)

# Extract JPEG-encoded image from base64-encoded string
JPEG_r = b64decode(b64_r["image"])

# Decode JPEG back into Numpy array
na_r = cv2.imdecode(np.frombuffer(JPEG_r,dtype=np.uint8), cv2.IMREAD_COLOR)

Note: If you want to convert to greyscale, you need to do that before JPEG-encoding:

# Convert to greyscale
grey = cv2.cvtColor(na, cv2. COLOR_BGR2GRAY)

# JPEG-encode
_, JPEG = cv2.imencode(".jpg", grey, [int(cv2.IMWRITE_JPEG_QUALITY), 80])

Keywords: Python, MQTT, AWS, image processing, JSON, base64, encode, decode, bandwidth, minimise, minimize, reduce, prime.

Upvotes: 3

Related Questions