ESP32 - Store RGB from decoded JPEG MCUs in buffer

Question

We are currently working on implementing motion detection for the ESP32-cam. To be able to work with the motion detection we want to access the raw pixels of the image, but also want to have a compressed JPEG to send over a network.

We use the JPEGDEC library were they have an example to draw a decoded JPEG image to an LCD screen. The decode function is supplied a callback function that should handle the drawing of each MCU. However, instead of instantly drawing each MCU we want to store the RGB565 data for the entire image in the variable rgb. This RGB image will then be used for the motion detection.

We have tried to implement the code below but can not get it to work and also have some questions:

1. We want to have the RGB data as uint8_t but the supplied pixels from JPEGDEC (pDraw->pPixels) are of type uint16_t. How can we handle this?
2. As of now we try to allocate enough memory to store the RGB image, but the malloc function returns NULL. Is it correct to allocate this amount below?

#include <JPEGDEC.h>

uint8_t *rgb = (uint8_t*) malloc(sizeof(uint8_t) * WIDTH * HEIGHT);
JPEGDEC jpeg;

void loop()
{
  camera_fb_t* frame = esp_camera_fb_get();

  jpeg.openRAM((uint8_t*) frame->buf, frame->len, drawMCU);
  jpeg.setPixelType(RGB565_LITTLE_ENDIAN);
  jpeg.decode(0, 0, JPEG_SCALE_HALF);
  jpeg.close();

  pixel_counter = 0;
}

int drawMCU(JPEGDRAW *pDraw) {
  for (int i = 0; i < pDraw->iWidth * pDraw->iHeight; i++) {
    rgb[pixel_counter] = pDraw->pPixels[i];
    pixel_counter++;
  }
}

Answer 1

It's not clear, I've done the same thing, extracted pixels to RGB565 format and shown live video stream on 65k color oled SSD1331 without use an external jpeg decoder.

You do not need a decoder at all, I've used the provided functions with the camera driver, search for "img_converters.h".

In the Arduino IDE open examples/ESP32/Camera/CameraWebServer.ino, move to app_httpd.cpp Tab and you can see this file inclusion.

This file contains all functions to convert image formats. In my linux system I found it on ESP32 Arduino core at this position: /home/massimo/Arduino/hardware/espressif/esp32/tools/sdk/include/esp32-camera/img_converters.h

It even provide some functions to do overlay on frames, like draw lines (see facial recognition red and green rects) and to print a text. Please I interested to it, if someone find more infos I want to try.

For a byte buffer depends on size you capture the image, don't expets to capture a full 1600x1200 frame to ESP ram, are a lots of data and you cannot, 2 bytes (RGB565) every pixel are a lots of data and you cannot capture in a single pass, you need to divide example in 8/16/32/64 strips.

With oled that is 96x64 I've set camera resolution at very low resolution, do not have much sense capture a big image and then downscale it, so I've set 160x120 resolution, when I capture the frame, by default it is jpeg, I've converted to RGB565 using a function provided in the img_converters all the frame in one single array, then removed vertical and horizontal odd pixel lines, so the final image result in 80x60 pixels, to complete it finally I pass all entire frame to the oled library I wrote for this oled in just one single command that draw very fast using this command that write a buffer in a single pass:

oled.pushPixels(pixelData, len);

Read from SD and draw on oled require 6-7 milliseconds every frame at fullscreen 96x64.

An image of 80x60 pixels 16 bit color (frame extracted from ESP32CAMto draw on oled) need 80 x 60 x 2 = 9600 bytes and there is no problem, I took it even on ESP8266, I've tried to allocate up 32kb, but just when I started to convert my code to show on 320x240 ILI9341 I tried a lot to manage a single frame, but without success, too much datas, the only way I've stripped it in 8 or 16 chunks per frame.

Because I've managed to read images from SD Card, I read one strip, then I render on display, then load another strip, then render.... etc.... This slow down a lot.

My oled library is very fast, it can show videos (read from uSD) at resolution 96x64 @ near 130 FPS, when I tried the same video (Matrix) but bigger on TFT, it sit down, max 12 FPS on a 320x180 pixel video with esp8266 clocked @ 160Mhz and SPI set to 50Mhz. A lots of data .... 12 FPS but to view on TFT it is incredible, it is not a PC or a smartphone or raspberry, it is a 4$ microcontroller + 6$ (320x240 2.44 ILI9341 touchscreen TFT) + 1$ card reader = touchscreen video player.

With oled I've managed to play videos @ 30 FPS with audio 44100 16 Bit Stereo synced out of external PCM5102A dac and it is fantastic, even with two MAX98357A 3W I2S to make a stereo channels and directly attached to hifi speakers. This way using an audio library I wrote, can play films downoladed from Youtube or other sources (with high quality audio out of the box) on a small color oled that is big like a coin and it is good to make a small ESP iOT smartwatch (but not touchscreen), it works with ESP8266 and with ESP32, not yet tried the external dac on ESP32CAM, it can work, but not in conjunction of SD Card, so I doubt it is capable to playback from SD_MMC, no more pins.

Answer 2

You can't call malloc outside of a function. If you have a setup() function move it there. Also, you need to allocate space for the r, g, and b bytes - which would be 3 times WIDTH * HEIGHT.

Then, extract the RGB bytes from the RGB565 data and store in the array.

uint8_t *rgb = NULL;

void setup() {
    rgb = malloc(WIDTH * HEIGHT * 3);
}

int drawMCU(JPEGDRAW *pDraw)
{
    for (int i = 0; i < pDraw->iWidth * pDraw->iHeight; i++) {
        uint8_t b = (pDraw->pPixels[i] & 0x001F) << 3;
        uint8_t g = (pDraw->pPixels[i] & 0x07E0) >> 3;
        uint8_t r = (pDraw->pPixels[i] & 0xF800) >> 8;
        rgb[i * 3] = r;
        rgb[i * 3 + 1] = g;
        rgb[i * 3 + 2] = b;
    }
}

Or, don't use malloc at all since WIDTH and HEIGHT are known at compile time:

uint8_t rgb[WIDTH * HEIGHT * 3];