Pass data from OpenCV C++ to NodeJS/JS | Electron

Question

I am trying to do a POC for Video Processing application with the following stack and struck with passing processed media stream from c++ application to Electron Front end GUI.

  Electron  
     |
   Nodejs
     |
C++ Application

C++ Application will read the IP/Webcam(using OpenCV only to fetch data) and process the input stream(not with OpenCV). I am trying to figure out a way to send that stream from C++ to Electron GUI(NodeJS/JS) with good fps. Right now I compiled my C++ app using node-gyp and installed it as node package.

Also, I don't want to change my C++ Application too much( like including OpenCV as node package), because later I will use that C++ Application alone for integrating with another application.

Answer 1

The Challenge:

We want to execute our heavy lifting code in a separate worker thread while also sending results (stream data chunks) back to the main thread during execution.

NAN (Native Abstractions for Node.js) already provides an approach to do this with (AsyncProgressWorker).

However, we can not know if the HandleProgressCallback is actually invoked during the execution to send back our results. This can happen when our run time is simply to fast and therefore the callback is never executed.

Proposed Solution:

We simply collect our stream output in a stack (StackCollect). We attempt to clear this stack immediately and send the stream results back to the main thread (if possible) - (StackDrain). If we don't have the time to clear the stack immediately we drain (whats left) at the end of the execution run (HandleOKCallback).

Implementation Example:

demo.cpp (our C++ node/electron addon):

#include <nan.h>
#include <node.h>
#include <v8.h>

#include <iostream>
#include <string>
#include <vector>

#include <mutex>

#include <chrono>
#include <thread>

class vSync_File : public Nan::AsyncProgressWorker {
public:
    ~vSync_File();
    vSync_File(Nan::Callback * result, Nan::Callback * chunk);

    void Execute(const Nan::AsyncProgressWorker::ExecutionProgress& chunk);

    void HandleOKCallback();

    void HandleProgressCallback(const char *tout, size_t tout_size);

    //needed for stream data collection
    void StackCollect(std::string & str_chunk, const Nan::AsyncProgressWorker::ExecutionProgress& tchunk);
    //drain stack
    void StackDrain();
private:
    Nan::Callback * chunk;
    //stores stream data - use other data types for different output
    std::vector<std::string> stack;
    //mutex
    std::mutex m;
};

vSync_File::vSync_File(Nan::Callback * result, Nan::Callback * chunk)
: Nan::AsyncProgressWorker(result), chunk(chunk) {}

vSync_File::~vSync_File() {
    delete chunk;
}

void vSync_File::StackCollect(std::string & str_chunk, const Nan::AsyncProgressWorker::ExecutionProgress& tchunk) { 
    std::lock_guard<std::mutex> guardme(m);

    stack.push_back(str_chunk);

    //attempt drain
    std::string dummy = "NA";
    tchunk.Send(dummy.c_str(), dummy.length());
}

//Dump out stream data
void vSync_File::StackDrain() {
  std::lock_guard<std::mutex> guardme(m);

  for (uint i = 0; i < stack.size(); i++) {
    std::string th_chunk = stack[i];
    v8::Local<v8::String> chk = Nan::New<v8::String>(th_chunk).ToLocalChecked();
    v8::Local<v8::Value> argv[] = { chk };

    chunk->Call(1, argv, this->async_resource);
  }
  stack.clear();
}

//Our main job in a nice worker thread
void vSync_File::Execute(const Nan::AsyncProgressWorker::ExecutionProgress& tchunk) {
    //simulate some stream output
    for (unsigned int i = 0; i < 20; i++) {
        std::string out_chunk;
        out_chunk = "Simulated stream data " + std::to_string(i);   

        std::this_thread::sleep_for(std::chrono::milliseconds(300)); //so our HandleProgressCallback is invoked, otherwise we are too fast in our example here

        this->StackCollect(out_chunk, tchunk);
    }   
}

//Back at the main thread - if we have time stream back the output 
void vSync_File::HandleProgressCallback(const char *tout, size_t tout_size) {
  Nan::HandleScope scope;

  this->StackDrain();
}

//Back at the main thread - we are done
void vSync_File::HandleOKCallback () {

  this->StackDrain(); //drain leftovers from stream stack

  v8::Local<v8::String> result_mess = Nan::New<v8::String>("done reading").ToLocalChecked();
  v8::Local<v8::Value> argv[] = { result_mess };
  callback->Call(1, argv, this->async_resource);
}


NAN_METHOD(get_stream_data) {
  Nan::Callback *result = new Nan::Callback(info[0].As<v8::Function>());
  Nan::Callback *chunk = new Nan::Callback(info[1].As<v8::Function>());

  AsyncQueueWorker(new vSync_File(result, chunk));
}


NAN_MODULE_INIT(Init) {
   //we want stream data
   Nan::Set(target, Nan::New<v8::String>("get_stream_data").ToLocalChecked(),
   Nan::GetFunction(Nan::New<v8::FunctionTemplate>(get_stream_data)).ToLocalChecked());
}

NODE_MODULE(stream_c_electron, Init)

index.js (electron implementation example):

const stream_c_electron = require('./build/linux_x64/stream_c_electron.node');

stream_c_electron.get_stream_data(function(res) {
    //we are done
    console.log(res);
}, function(chk) {
    console.log("a line streamed");
    console.log(chk);
});

package.json:

{
  "name": "stream_c_electron",
  "version": "1.0.0",
  "description": "stream from c++ node addon demo",
  "main": "index.js",
  "scripts": {
    "start": "electron .",
    "build_this": "HOME=~/.electron-gyp node-gyp rebuild --target=2.0.8 --arch=x64 --dist-url=https://atom.io/download/electron",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "11AND2",
  "license": "MIT",
  "dependencies": {
    "nan": "2.11.0"
  },
  "devDependencies": {
    "electron": "2.0.8"
  }
}

binding.gyp:

{
  "targets": [
    {
      "target_name": "stream_c_electron",
      "sources": [  "c_src/demo.cpp"  ],
      "conditions": [
        [
        'OS=="linux"',
        {
            "cflags": ["-Wall", "-std=c++11"],
            'product_dir' : 'linux_x64',
            "include_dirs": [
                "<!(node -e \"require('nan')\")"
            ]
        }
        ]
      ]
    }
   ]
}

Answer 2

You have to compile your c++ stuff as a static library with emscripten and load it in via import MyLib from "./MyLib"; or with require and run with node --experimental-modules --napi-modules main.mjs. Basically the idea is that the V8 engine is able to read your native code. It's also incredibly fast compared to pure javascript code.

It's actually pretty easy when you know what to do. Have a look to this sample code. It basically uses native c++ libpng library for javascript. The only tricky thing is actually interfacing c++ with javascript.

https://github.com/skanti/png-decoder-javascript/tree/devel