CODEWITHSUNDEEP
pythonmqttpaho
keen
keen

Reputation: 83

Trouble in sending files in MQTT

I am new to MQTT. I was trying to explore how to send files with MQTT.

I tried to convert a picture (.jpg format) into a byte array. If I directly send the byte array as MQTT payload, it works well. But, I need to send the file information, so I tried to pack into a json object and send, but json only allows string. So, I tried to send the byte array as string.

file = "pic.jpg";
f = open (file, "rb");
fileContent = f.read();
byteArray = bytearray(fileContent);

fileData = {
    'name': file,
    'content': str(byteArray)
}
payload=json.dumps(fileData);

sender.publish(TOPIC, payload=payload, qos=2, retain=False);

Now, when I receive the byte array, I tried to encode the string content to byte array. But, I see, all the newline characters are not formatted correctly, and it does not produce the file.

def onMessage(client, userdata, msg):
    print("=> Message Received");

    fileData = json.loads(msg.payload)

    f = open(fileData['name'], "wb");
    f.write(bytes(fileData['content'], 'utf-8'));

    f.close();

Please help me understand the file data encoding and decoding techniques. Thank you!

Upvotes: 1

Views: 1006

Answers (1)

Mark Setchell
Mark Setchell

Reputation: 207465

As you have correctly discovered, you can't send arbitrary binary data, such as an image, in a JSON.

One option might be to base64-encode the image and put that in a JSON with the image name and other meta-data. But, depending on your environment, you may be paying a heavy price because JPEGs are often 100-300kB and base64-encoding will always inflate that by 33%.

However, there is nothing to stop you sending two parts in one message. So, as a second option, you could send a JSON with the filename and metadata (ownership, dates) followed by the raw binary data. If you want to do that, you could decide to always send say a 4-byte header with the length of the JSON packet encoded in network byte order using htonl() so that the decoder knows how many bytes to strip off the start and interpret as JSON - and thereby to also know where the image starts.

A third option is to leverage the fact that both JPEGs and PNG images have a well-known sequence of bytes both at the start and end, so you could either append or prepend an image to a JSON and find the boundary between the two by searching using a regex within a message. Check the Wikipedia entries for PNG and JPEG:

  • JPEG start = ff d8 ff
  • JPEG end = ff d9
  • PNG start = 89 50 4e 47 0d 0a 1a 0a
  • PNG end = 49 45 4e 44 ae 42 60 82

My preference would normally tend to be the second option, but all 3 should work.

Upvotes: 1

Related Questions