CODEWITHSUNDEEP
javascriptangularweb
Yuanyi Wu
Yuanyi Wu

Reputation: 329

Is it possible to render a png/jpeg byte array to an element without creating a blob and URL.createObjectURL?

I am writing a front-end application with Angular and grpc-web stream. The goal is to render the image payload from the stream at 60-100 FPS. It was previously implemented in WPF but now I need to switch to a web front-end. I notice that even though the binary image has been transferred to the browser, I still need to create a blob URL for the to download.

My current strategy is:

URL.revokeObjectURL(this.ImageSrc);
const blob = new Blob([res.getImage_asU8()], {type: res.getType()});
this.ImageSrc = URL.createObjectURL(blob);

However, there are two major performance issues 1. CPU usage is very high. I profiled it with CDT and found that new Blob() made up the most script time. 2. Eventhough the object is already in-memory, after I created the blob URL the img element seemed to download the blob from memory again. From the CDT networking monitoring I found that each in-memory blob takes 10-30ms to 'download'. (UPDATE: possibly due to CDT slowing down the performance)

Besides creating the URL for a blob, I also tried using base64 for rendering the image. The image is 800*600 monochrome png, and as expected, base64 does not help in this case.

I wonder whether I can directly render the binary image transferred by a stream onto an element (img, canvas, or svg) without creating the blob or re-downloading the blob from memory. Thank you!

Update April 10

I made a test project for comparing the image updating speed. I meant to compare three usage cases but I don't know how to decode png/jpeg binary into RGBA efficiently in browser JavaScript so I skipped that.

Multi-image comparison

<img> + createObjectURL: img + blob

<canvas> + putImageData: canvas

So, when 10 images are streamed in parallel, the image + blob solution seems faster than the canvas based method.

Single-image comparison

<img> + createObjectURL: img + blob

<canvas> + putImageData: server-side + canvas

Surprisingly, the img is possible to keep up at this rate. The canvas seems too heavy to update at this rate.

Conclusion

<img> + createObjectURL should fit most high FPS situation. ** Do not trust the timing in CDT because CDT will slow down the execution**. I am not doing a very careful comparison here, please let me know if I am not properly using the canvas element. Besides, the heap usage for <img> + createObjectURL is much lower than using the canvas as long as URL.revokeObjectURL is called in time.

Upvotes: 1

Views: 1019

Answers (2)

Strille
Strille

Reputation: 5781

Maybe some of the work can be offloaded to a web worker?

For instance, createImageBitmap can create an ImageBitmap from a Blob, and ImageBitmap implements the Transferable interface, which means it can be transfered between a Web worker and the main thread.

So, maybe look into having a web worker producing ImageBitmap that is then passed to the main thread and rendered to a canvas?

Upvotes: 1

Kaiido
Kaiido

Reputation: 137084

That really depends on where your images are coming from.

The simplest idea here would be to send it as a video stream.

If it is not doable, then you might consider sending the raw bitmap data as Uint8 [R,G,B,A,R,G,B,A], that you'll be able to push directly to a canvas, given you have a fixed width and height.

const width = 300;
const height = 150;
const service = new Worker(getWorkerURL());
const ctx = canvas.getContext('2d');

service.onmessage = e => {
  const img = new ImageData(e.data, 300, 150);
  ctx.putImageData(img,0,0);
};





// fake service
// a simple Worker that will send a message with an Uint8ClampedArray of random data

function getWorkerURL() {
  const content = `
function sendData() {
  const arr = new Uint8ClampedArray(300*150*4);
  for(let i = 0; i<arr.length; i++) {
    if(i%4 === 3) arr[i] = 255;
    arr[i] = Math.random() * 255;
  }
  postMessage(arr, [arr.buffer]);
  if(self.requestAnimationFrame)
    self.requestAnimationFrame(sendData);
  else setTimeout(sendData, 16);
}
sendData();`;
  const blob = new Blob([content], {type: 'application/javascript'});
  return URL.createObjectURL(blob);
}
<canvas id="canvas"></canvas>

If this is not possible either, and you have to send binary encoded images, then there is no better way than going through an <img> element, even if indeed, this implies a lot of asynchronous operations like network ones. You could of course try to parse the binary file yourself from the ArrayBuffer you receive, and to decode the raw pixels values, but my gut says that you'll need a lot of work to be able to compete with built-in implementations.

Upvotes: 1

Related Questions