CODEWITHSUNDEEP
node.jstypescripthtml5-canvaswebgl
axis7818
axis7818

Reputation: 23

How to convert a Uint8Array to a CanvasImageSource?

In TypeScript (NodeJS), I am struggling to convert a Uint8Array with bitmap image data into a CanvasImageSource type.

More Context

I am working on a typescript library that will be used in the browser as well as NodeJS environments. The library does image operations with WebGL, so in NodeJS environments, I am attempting to leverage headless-gl. My library includes a function (getCanvasImageSource) that returns a CanvasImageSource for clients to use.

A bunch of code was removed for the purposes of asking the question. A WebGL shader will create the desired image on the gl context, which the client can retrieve through the CanvasImageSource. This works as intended in a browser client.

/**
 * Browser version of the library.
 */
export class MyLibrary {
    protected gl!: WebGLRenderingContext;
    protected width: number;
    protected height: number;

    public getCanvasImageSource(): CanvasImageSource {
        return this.gl.canvas;
    }

    /**
     * the GL context can only be created in a browser.
     */
    protected makeGL(): WebGLRenderingContext {
        const canvas = document.createElement('canvas');
        canvas.width = this.width;
        canvas.height = this.height;
        const glContext = canvas.getContext('webgl');
        if (!glContext) {
            throw new Error("Unable to initialize WebGL. This browser or device may not support it.");
        }
        return glContext;
    }
}

import gl from 'gl';

/**
 * A subclass of MyLibrary that overrides the browser-specific functionality.
 */
export class MyHeadlessLibrary extends MyLibrary {
    public getCanvasImageSource(): CanvasImageSource {
        // The canvas is `undefined` from headless-gl.
        // this.gl.canvas === undefined;

        // But, I can read the pixel data as a bitmap.
        const format = this.gl.RGBA;
        const type = this.gl.UNSIGNED_BYTE;
        const bitmapData = new Uint8Array(this.width * this.height * 4);
        this.gl.readPixels(0, 0, this.width, this.height, format, type, bitmapData);

        // This is where I am struggling...
        // Is there a way to convert my `bitmapData` into a `CanvasImageSource`?
    }

    /**
     * Overrides the browser's WebGL context with the headless-gl implementation.
     */
    protected makeGL(): WebGLRenderingContext {
        const glContext = gl(this.width, this.height);
        return glContext;
    }
}

However, I am struggling to find a way to successfully convert the Uint8Array data read from the headless-gl context into a CanvasImageSource object.

Here are some things that I have tried:

1. Return this.gl.canvas

This ends up being undefined in the case of headless-gl.

2. Use JSDOM

JSDOM's canvas does not support the WebGL render context.

3. Use JSDOM to create an HTMLImageElement and set the src to a base64 URL

I haven't quite figured out why, but the promise here does not ever resolve or reject. This results in my library timing out. So perhaps this strategy will work, but there is an issue with my implementation.

This strategy has been used in other areas of the library, but none that involve headless-gl or even WebGL. Just the 2D canvas context.

import gl from 'gl';
import { JSDOM } from 'jsdom';

export class MyHeadlessLibrary extends MyLibrary {
    /**
     * In this attempt, I changed the return type to Promise<CanvasImageSource> here, in MyLibrary, and in the client code.
     */
    public getCanvasImageSource(): Promise<CanvasImageSource> {
        // The canvas is `undefined` from headless-gl.
        // this.gl.canvas === undefined;

        // But, I can read the pixel data as a bitmap.
        const format = this.gl.RGBA;
        const type = this.gl.UNSIGNED_BYTE;
        const bitmapData = new Uint8Array(this.width * this.height * 4);
        this.gl.readPixels(0, 0, this.width, this.height, format, type, bitmapData);

        // Create a DOM and HTMLImageElement.
        const html: string = `<!DOCTYPE html><html><head><meta charset="utf-8" /><title>DOM</title></head><body></body></html>`;
        const dom = new JSDOM(html);
        const img = dom.window.document.createElement('img');

        // Create a base64 data URL
        const buffer = Buffer.from(bitmapData);
        const dataurl = `data:image/bmp;base64,${buffer.toString('base64')}`;

        // Set the image source and wrap the result in a promise
        return new Promise((resolve, reject) => {
            img.onerror = reject;
            img.onload = () => resolve(img);
            img.src = dataurl;
        });
    }
}

Please let me know if something in my code jumps out as a problem, or point me to a potential solution to this problem!

Upvotes: 0

Views: 2043

Answers (2)

user128511
user128511

Reputation:

According to the spec a CanvasImageSource is

typedef (HTMLOrSVGImageElement or
         HTMLVideoElement or
         HTMLCanvasElement or
         ImageBitmap or
         OffscreenCanvas) CanvasImageSource;

So it depends on what your needs are. If you don't need any alpha then one of those is HTMLCanvasElement so given pixels you could do

function pixelsToCanvas(pixels, width, height) {
  const canvas = document.createElement('canvas');
  canvas.width = width;
  canvas.height = height;
  const ctx = canvas.getContext('2d');
  const imgData = ctx.createImageData(width, height);
  imgData.data.set(pixels);
  ctx.putImageData(imgData, 0, 0);

  // flip the image
  ctx.scale(1, -1);
  ctx.globalCompositeOperation = 'copy';
  ctx.drawImage(canvas, 0, -height, width, height);

  return canvas;
}

This should work as long as either you have no alpha or you don't care about lossy alpha. Note: the code assumes you're providing unpremliplied alpha

The issue is pixels could have a pixel like this 255, 192, 128, 0. But because alpha is zero, when you pass that through the function above you'll get 0, 0, 0, 0 in the canvas that comes out because canvases always use premultiplied alpha. That may not be an issue because for most use cases 255, 192, 128, 0 appears as 0,0,0,0 anyway but if you have a special use case then this solution won't work.

Note: You'll need the canvas package

As for your Image from dataURL example this code makes no sense

// Create a base64 data URL
const buffer = Buffer.from(bitmapData);
const dataurl = `data:image/bmp;base64,${buffer.toString('base64')}`;

First off, whether image/bmp is supported or not is browser dependent so it's possible JSDOM doesn't support image/bmp but further a bmp file has a header which the code is not supplying. Without that header there is no way for any API to know what's in the data. If you gave it 256 bytes is that an 8x8 4byte per pixel image? A 16x4 4byte per pixel image? A black and white 32x64 1 bit per pixel image? etc.. You need the header.

Maybe writing the header will make that code work?

function convertPixelsToBMP(pixels, width, height) {
  const BYTES_PER_PIXEL = 4;
  const FILE_HEADER_SIZE = 14;
  const INFO_HEADER_SIZE = 40;

  const dst = new Uint8Array(FILE_HEADER_SIZE + INFO_HEADER_SIZE + width * height * 4);
  
  {
    const data = new DataView(dst.buffer);
    const fileSize = FILE_HEADER_SIZE + INFO_HEADER_SIZE + (width * height * 4);

    data.setUint8 ( 0, 0x42); // 'B'
    data.setUint8 ( 1, 0x4D); // 'M'
    data.setUint32( 2, fileSize, true)
    data.setUint8 (10, FILE_HEADER_SIZE + INFO_HEADER_SIZE);

    data.setUint32(14, INFO_HEADER_SIZE, true);
    data.setUint32(18, width, true);
    data.setUint32(22, height, true);
    data.setUint16(26, 1, true);
    data.setUint16(28, BYTES_PER_PIXEL * 8, true);
  }

  // bmp expects colors in BGRA format
  const pdst = new Uint8Array(dst.buffer, FILE_HEADER_SIZE + INFO_HEADER_SIZE);
  for (let i = 0; i < pixels.length; i += 4) {
    pdst[i    ] = pixels[i + 2];
    pdst[i + 1] = pixels[i + 1];
    pdst[i + 2] = pixels[i + 0];
    pdst[i + 3] = pixels[i + 3];
  }
  return dst;
}

note: this code also assumes you're providing unpremultiplied alpha.

Upvotes: 2

axis7818
axis7818

Reputation: 23

@gman, thank you for the help! And that makes sense that I would need the header for the base64 URL, but I didn't need that anyways. Returning an HTMLCanvasElement is sufficient for my needs. There is some alpha in the image, but premultiplied alpha is not an issue.

The one other thing I ran into was that the resulting image was flipped vertically. I assume this is because of the difference in WebGL & 2D canvas coordinate systems. I solved this by looping through the pixels & swapping rows.

The resulting solution looks like this:

export class MyHeadlessLibrary extends MyLibrary {
    public getCanvasImageSource(): CanvasImageSource {
        // read the pixel data
        const pixels = new Uint8Array(this.width * this.height * 4);
        this.gl.readPixels(0, 0, this.width, this.height, this.gl.RGBA, this.gl.UNSIGNED_BYTE, pixels);

        // create a headless canvas & 2d context
        const html: string = `<!DOCTYPE html><html><head><meta charset="utf-8" /><title>DOM</title></head><body></body></html>`;
        const dom = new JSDOM(html);
        const canvas = dom.window.document.createElement('canvas');
        canvas.width = this.width;
        canvas.height = this.height;
        const ctx = canvas.getContext('2d');
        if (!ctx) {
            throw Error("Unable to create a 2D render context");
        }

        // flip the image
        const bytesPerRow = this.width * 4;
        const temp = new Uint8Array(bytesPerRow);
        for (let y = 0; y < this.height / 2; y += 1) {
            const topOffset = y * bytesPerRow;
            const bottomOffset = (this.height - y - 1) * bytesPerRow;
            temp.set(pixels.subarray(topOffset, topOffset + bytesPerRow));
            pixels.copyWithin(topOffset, bottomOffset, bottomOffset + bytesPerRow);
            pixels.set(temp, bottomOffset);
        }

        // Draw the pixels into the new canvas
        const imgData = ctx.createImageData(this.width, this.height);
        imgData.data.set(pixels);
        ctx.putImageData(imgData, 0, 0);

        return canvas;
    }
}

Upvotes: 1

Related Questions