CODEWITHSUNDEEP
reactjstensorflowkerasdeep-learningtensorflow.js
yudhiesh
yudhiesh

Reputation: 6809

Tensorflow.js cropping image return zeros tensor

I have a custom model which takes in cropped faces from BlazeFace Model then outputs a prediction of 3 classes.

Before sending them to my custom model I resize the cropped faces to be of shape [1,224,224,3]

Output at every prediction:

Float32Array [
  6.522771905936864e-11,
  3.698188456857654e-12,
  1,
]

Code for resizing the cropped faces and making predictions:

 const getPrediction = async tensor => {
    if (!tensor) {
      console.log("Tensor not found!");
    }
    // Load both models 
    const bfModel = await blazeFaceModel;
    const returnTensors = true;
    const faces = await bfModel
      .estimateFaces(tensor, returnTensors)
      .catch(e => console.log(e));
    // Reshape tensor from rank 3 to rank 4
    const tensorReshaped = tensor.reshape([1, 224, 224, 3]);
    const scale = {
      height: styles.camera.height / tensorDims.height,
      width: styles.camera.width / tensorDims.width
    };

    // Faces is an array of objects
    if (!isEmpty(faces)) {
      // Setting faces in state
      setModelFaces({ faces });
    } 
    //Looping over array of objects in faces
    faces.map((face, i) => {
      const { topLeft, bottomRight } = face;
      const width = Math.floor(
        bottomRight.dataSync()[0] - topLeft.dataSync()[0] * scale.width
      );
      const height = Math.floor(
        bottomRight.dataSync()[1] - topLeft.dataSync()[1] * scale.height
      );
      const boxes = tf
        .concat([topLeft.dataSync(), bottomRight.dataSync()])
        .reshape([-1, 4]); 
      // Cropping out faces from original tensor 
      const crop = tf.image.cropAndResize(
        tensorReshaped,
        boxes,
        [0],
        [height, width]
      );
      // Resize cropped faces to [1,224,224,3]
      const alignCorners = true;
      const imageResize = tf.image.resizeBilinear(
        crop,
        [224, 224],
        alignCorners
      );
      makePrediction(imageResize);
    });
  };
// Make predictions on the tensors
  const makePrediction = async image => {
    if (!image) {
      console.log("No input!");
    }
    const model = await loadedModel;
    const prediction = await model.predict(image, { batchSize: 1 });
    if (!prediction || isEmpty(prediction)) {
      console.log("Prediction not available");
    }
    console.log(prediction);
    console.log(prediction.dataSync());
  };

EDIT

I tried changing the batch size when making predictions to 1 and still the same issue

I tried reconverting the keras model to tfjs format and still the same issue

I tried disposing of the tensor after making a prediction but still there is an error

So i printed out the tensors of the resized faces and its a lot of 0's

Tensor before prediction
Tensor
    [[[[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      ...
      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]]]]
undefined
Tensor during prediction
Tensor
    [[[[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      ...
      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]]]]
undefined

Upvotes: 1

Views: 1028

Answers (1)

edkeveked
edkeveked

Reputation: 18381

boxes of tf.image.cropAndResize are normalized coordinates between 0 and 1. Therefore topLeft and bottomRight should be normalized by using [imageWidth, imageHeight]

normalizedTopLeft = topLeft.div(tensor.shape.slice(-3, -2)) 
// slice will get [h, w] of a tensor of shape [b, h, w, ch] or [h, w, ch]

// do likewise for bottomRight
// use normalizedTopLeft instead of topLeft for cropping

Upvotes: 1

Related Questions