CODEWITHSUNDEEP
reactjsreact-nativetensorflowdeep-learningtensorflow.js
yudhiesh
yudhiesh

Reputation: 6809

How do I crop faces based on bounding boxes in Tensorflow.js?

I need to crop faces which are detected in BlazeFace Model then send the image over to a custom model I made. I have already implemented the face detection with the bounding boxes but am stuck at cropping the face out.

I have the coordinates of the landmarks and the bottomRight and topLeft but I do not know how to do so. In python with tensorflow their exist functions to do so but with tensorflow.js I can't find anything for this.

Rendering Bounding Boxes on Face

    const faces = await bfModel
      .estimateFaces(tensor, returnTensors)
      .catch(e => console.log(e));
    console.log(faces);

    // Faces is an array of objects
    if (!isEmpty(faces)) {
      setModelFaces({ faces });
    }

  const renderBoundingBoxes = () => {
    const { faces } = modelFaces;
    const scale = {
      height: styles.camera.height / tensorDims.height,
      width: styles.camera.width / tensorDims.width
    };
    const flipHorizontal = Platform.OS === "ios" ? false : true;
    if (!isEmpty(faces)) {
      return faces.map((face, i) => {
        const { topLeft, bottomRight } = face;
        const bbLeft = topLeft.dataSync()[0] * scale.width;
        const boxStyle = Object.assign({}, styles.bbox, {
          left: flipHorizontal
            ? previewWidth - bbLeft - previewLeft
            : bbLeft + previewLeft,
          top: topLeft.dataSync()[1] * scale.height + 20,
          width:
            (bottomRight.dataSync()[0] - topLeft.dataSync()[0]) * scale.width,
          height:
            (bottomRight.dataSync()[1] - topLeft.dataSync()[1]) * scale.height
        });

        return <View style={boxStyle}></View>;
        1;
      });
    }
  };

Output from console.log(faces):

Array [
  Object {
    "bottomRight": Tensor {
      "dataId": Object {},
      "dtype": "float32",
      "id": 220600,
      "isDisposedInternal": false,
      "kept": false,
      "rankType": "1",
      "scopeId": 426282,
      "shape": Array [
        2,
      ],
      "size": 2,
      "strides": Array [],
    },
    "landmarks": Tensor {
      "dataId": Object {},
      "dtype": "float32",
      "id": 220602,
      "isDisposedInternal": false,
      "kept": false,
      "rankType": "2",
      "scopeId": 426286,
      "shape": Array [
        6,
        2,
      ],
      "size": 12,
      "strides": Array [
        2,
      ],
    },
    "probability": Tensor {
      "dataId": Object {},
      "dtype": "float32",
      "id": 220592,
      "isDisposedInternal": false,
      "kept": false,
      "rankType": "1",
      "scopeId": 426249,
      "shape": Array [
        1,
      ],
      "size": 1,
      "strides": Array [],
    },
    "topLeft": Tensor {
      "dataId": Object {},
      "dtype": "float32",
      "id": 220599,
      "isDisposedInternal": false,
      "kept": false,
      "rankType": "1",
      "scopeId": 426280,
      "shape": Array [
        2,
      ],
      "size": 2,
      "strides": Array [],
    },
  },
]

Upvotes: 1

Views: 1914

Answers (2)

rhiskey
rhiskey

Reputation: 73

I dont sure but it can work:

let img_4d = tf.expandDims(inputTF3D, axis=0)
const start = predictions[id].topLeft;
const end = predictions[id].bottomRight;
let boxes = tf.concat([start, end]);
boxes = tf.reshape(boxes,[-1, 4])
let crop = tf.image.cropAndResize(img_4d, boxes, [0], [IMAGE_HEIGHT, IMAGE_WIDTH])

https://github.com/tensorflow/tfjs/issues/3914

Upvotes: 0

edkeveked
edkeveked

Reputation: 18381

image can be cropped using tf.image.cropAndResize. The tensor should be a 4d tensor. If the image is a 3d tensor, it first needs to be expanded. The crop expected height and width should be passed as argument to copAndResize

boxes = tf.concat([topLeftTensor, bottomRightTensor]).reshape([-1, 4])
crop = tf.image.cropAndResize(images, boxes, [0], [height, width])

Upvotes: 3

Related Questions