How to access image pixel data in react-native

Question

I have an Image that was loaded from camera roll or any other source, usually local one.

How can I access its pixel data map to perform some calculations or measurements?

Answer 1

I converted the the byes array buffer to unit8array that will allows to interact with the binary data in the ArrayBuffer as an array of bytes (values from 0 to 255). This makes it easier to work with the frame’s raw binary data (like extracting pixel values ). and it doesn't take time it did calculate the sum of all pixel's luminance in ms for my device. I used 110 as the brightness threshold.

const onLightDetected = Worklets.createRunOnJS((luminanceState) => {
 setIsBrightnessFeedbackVisible(!luminanceState);
});

const frameProcessor = useSkiaFrameProcessor(frame => {
 'worklet';
 runAtTargetFps(1, () => {
   const buffer = frame.toArrayBuffer();
   const data = new Uint8Array(buffer);
   console.log('pixelcount', data.length);
   let luminanceSum = 0;
   const sampleSize =data.length; 
   for (let i = 0; i < sampleSize; i++) {
     luminanceSum += data[i];
   }
   const averageBrightness = luminanceSum / sampleSize;
   console.log('avgbrightness', averageBrightness);
   const luminanceState = averageBrightness > BRIGHTNESS_THRESHOLD;
   void onLightDetected(luminanceState);
});
 const faces = detectFaces(frame);
 frame.render();
});

Answer 2

Thanks Michael. Seems today its much easier

import { RNCamera } from 'react-native-camera';

<RNCamera ref={ref => {this.camera = ref;}} style={styles.preview}></RNCamera>

//Set the options for the camera
const options = {
    base64: true
};
// Get the base64 version of the image
const data = await this.camera.takePictureAsync(options)

Answer 3

There is a way of getting this information using native modules, but I only have Android implementation at the moment. Tested on RN 0.42.3. First of all, you'll need to create a native module in your app. Assuming that application is initialized with the name SampleApp, create new directory in your React Native project android/app/src/main/java/com/sampleapp/bitmap with two files in it:

android/app/src/main/java/com/sampleapp/bitmap/BitmapReactPackage.java

package com.sampleapp;

import com.facebook.react.ReactPackage;
import com.facebook.react.bridge.JavaScriptModule;
import com.facebook.react.bridge.NativeModule;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.uimanager.ViewManager;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

public class BitmapReactPackage implements ReactPackage {

  @Override
  public List<Class<? extends JavaScriptModule>> createJSModules() {
    return Collections.emptyList();
  }

  @Override
  public List<ViewManager> createViewManagers(ReactApplicationContext reactContext) {
    return Collections.emptyList();
  }

  @Override
  public List<NativeModule> createNativeModules(
                              ReactApplicationContext reactContext) {
    List<NativeModule> modules = new ArrayList<>();

    modules.add(new BitmapModule(reactContext));

    return modules;
  }

}

android/app/src/main/java/com/sampleapp/bitmap/BitmapModule.java

package com.sampleapp;

import com.facebook.react.bridge.NativeModule;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactContext;
import com.facebook.react.bridge.ReactContextBaseJavaModule;
import com.facebook.react.bridge.ReactMethod;
import com.facebook.react.bridge.Promise;
import com.facebook.react.bridge.WritableNativeArray;
import com.facebook.react.bridge.WritableNativeMap;

import android.graphics.Bitmap;
import android.graphics.BitmapFactory;

import java.io.IOException;

public class BitmapModule extends ReactContextBaseJavaModule {

  public BitmapModule(ReactApplicationContext reactContext) {
    super(reactContext);
  }

  @Override
  public String getName() {
    return "Bitmap";
  }

  @ReactMethod
  public void getPixels(String filePath, final Promise promise) {
    try {
      WritableNativeMap result = new WritableNativeMap();
      WritableNativeArray pixels = new WritableNativeArray();

      Bitmap bitmap = BitmapFactory.decodeFile(filePath);
      if (bitmap == null) {
        promise.reject("Failed to decode. Path is incorrect or image is corrupted");
        return;
      }

      int width = bitmap.getWidth();
      int height = bitmap.getHeight();

      boolean hasAlpha = bitmap.hasAlpha();

      for (int x = 0; x < width; x++) {
        for (int y = 0; y < height; y++) {
          int color = bitmap.getPixel(x, y);
          String hex = Integer.toHexString(color);
          pixels.pushString(hex);
        }
      }

      result.putInt("width", width);
      result.putInt("height", height);
      result.putBoolean("hasAlpha", hasAlpha);
      result.putArray("pixels", pixels);

      promise.resolve(result);

    } catch (Exception e) {
      promise.reject(e);
    }

  }

}

As you can see in the second file there is a method getPixels, which will be available from JS as a part of Bitmap native module. It accepts a path to an image file, converts the image to an internal Bitmap type, which allows to read image pixels. All image pixels are read one by one and saved to array of pixels in a form of hex strings (because React Native does not allow to pass hex values through the bridge). These hex strings have 8 characters, 2 characters per ARGB channel: first two characters is a hex value for alpha channel, second two - for red, third two - for green and the last two - for blue channel. For example, the value ffffffff - is a white color and ff0000ff - is a blue color. For convenience, image width, height and presence of alpha channel are returned along with array of pixels. Method returns a promise with an object:

{
  width: 1200,
  height: 800,
  hasAlpha: false,
  pixels: ['ffffffff', 'ff00ffff', 'ffff00ff', ...]
}

Native module also has to be registered in the app, modify android/app/src/main/java/com/sampleapp/MainApplication.java and add new module in there:

@Override
protected List<ReactPackage> getPackages() {
  return Arrays.<ReactPackage>asList(
      new MainReactPackage(),
      new BitmapReactPackage() // <---
  );
}

How to use from JS:

import { NativeModules } from 'react-native';

const imagePath = '/storage/emulated/0/Pictures/blob.png';

NativeModules.Bitmap.getPixels(imagePath)
  .then((image) => {
    console.log(image.width);
    console.log(image.height);
    console.log(image.hasAlpha);


    for (let x = 0; x < image.width; x++) {
      for (let y = 0; y < image.height; y++) {
        const offset = image.width * y + x;
        const pixel = image.pixels[offset];
      }
    }
  })
  .catch((err) => {
    console.error(err);
  });

I need to mention, that it performs pretty slow, most likely because of transfering a huge array through the bridge.