CODEWITHSUNDEEP
pythonhtmlnumpyvideo-streamingmultipart-mixed-replace
Basj
Basj

Reputation: 46423

Sending RGB image data from Python numpy to a browser HTML page

I need to send realtime image RGB data (in Numpy format) to a HTML page in a browser (web-based GUI), through HTTP. The following code works with the well-known multipart/x-mixed-replace trick: run this and access http://127.0.0.1:5000/video_feed: you will see a video in the browser.

from flask import Flask, render_template, Response
import numpy as np, cv2
app = Flask('')
def gen_frames():  
    while True:
        img = np.random.randint(0, 255, size=(1000, 1000, 3))
        ret, buf = cv2.imencode('.jpg', img)
        frame = buf.tobytes()
        yield (b'--frame\r\nContent-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')
@app.route('/video_feed')
def video_feed():
    return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')
app.run()

However, according to my benchmark, the real performance bottleneck is the cv2.imencode('.jpg', img).

In my real application, if I just generate the image, the CPU is ~ 1% for Python. When I imencode(...), the CPU jumps to 25%, and 15% for Chrome.

I also tried with PNG format but it's similar.

Question: how to efficiently send RGB image data from a numpy array (example: 1000 x 1000 pixels x 3 colors because of RGB) to a browser HTML page?

(Without compression/decompression it might be better, but how?)

enter image description here

Here is the benchmark

          FPS       CPU PYTHON      CPU CHROME
PNG       10.8      20 %            10 %
JPG       14        23 %            12 %
JPG       10.7      16 %            10 %           (with time.sleep to match PNG 10.8 fps)
BMP       19        17 %            23 %
BMP       10.8      8 %             12 %           (with time.sleep to match PNG 10.8 fps)

Upvotes: 2

Views: 1179

Answers (4)

Carlon van spijker
Carlon van spijker

Reputation: 11

Maybe you can try encoding it as base64 for compression of the video/image and then send it to the browser with the base64 mime type e.g.data:image/jpeg;base64.

Upvotes: 0

queeg
queeg

Reputation: 9384

As it stands AVI or MP4 compression would be good quality even for movies, but the compression itself takes too much CPU time to perform it on live data.

If some arbitrary protocol/format were created, one would not just have to program the server but also the client to consume this protocol/format. Therefore still some standard solution should be preferred.

I believe you can find a compromise between compression and CPU load in video conferencing systems, where the live camera data needs to be compressed and streamed via the network. With that in mind I believe here are sources of information that can help pursuing the topic:

Upvotes: 0

Bastian Venthur
Bastian Venthur

Reputation: 16540

According to the docs you could check if the default optimization is turned on:

Default Optimization in OpenCV

Many of the OpenCV functions are optimized using SSE2, AVX, etc. It contains the unoptimized code also. So if our system support these features, we should exploit them (almost all modern day processors support them). It is enabled by default while compiling. So OpenCV runs the optimized code if it is enabled, otherwise it runs the unoptimized code. You can use cv.useOptimized() to check if it is enabled/disabled and cv.setUseOptimized() to enable/disable it.

So try this:

In [5]: cv.useOptimized()         # check for optimization
Out[5]: False
In [7]: cv.setUseOptimized(True)  # turn it on if not already turned on

Upvotes: 0

Booboo
Booboo

Reputation: 44108

Try using the PILLOW module instead and see if that improves performance. My benchmark shows that each iteration of the gen_frames() generator function based on PILLOW requires less than half the CPU of the CV2 version.

from flask import Flask, render_template, Response
from PIL import Image
import numpy as np
from io import BytesIO

app = Flask('')

def gen_frames():
    while True:
        img = np.random.randint(0, 255, size=(1000, 1000, 3), dtype=np.uint8)
        rgb_image = Image.fromarray(img, 'RGB')
        buf = BytesIO()
        rgb_image.save(buf, 'JPEG')
        frame = buf.getbuffer()
        yield (b'--frame\r\nContent-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

@app.route('/video_feed')
def video_feed():
    return Response(gen_frames(), mimetype='multipart/x-mixed-replace; boundary=frame')

app.run()

Upvotes: 1

Related Questions