CODEWITHSUNDEEP
c#androidopencvtensorflowtensorflowsharp
Robert Kaa Frank
Robert Kaa Frank

Reputation: 53

How to transform Byte[](decoded as PNG or JPG) to Tensorflows Tensor

I'am trying to use Tensorflowsharp in a Project in Unity.

The problem i'm facing is that for the transform you usually use a second Graph to transform the input into a tensor. The used functions DecodeJpg and DecodePng are not supported on Android so how can you transform that input into a tensor ? 

private static void ConstructGraphToNormalizeImage(out TFGraph graph, out TFOutput input, out TFOutput output, TFDataType destinationDataType = TFDataType.Float)
{

    const int W = 224;
    const int H = 224;
    const float Mean = 117;
    const float Scale = 1;
    graph = new TFGraph();
    input = graph.Placeholder(TFDataType.String);
    output = graph.Cast(graph.Div(
        x: graph.Sub(
            x: graph.ResizeBilinear(
                images: graph.ExpandDims(
                    input: graph.Cast(
                        graph.DecodeJpeg(contents: input, channels: 3), DstT: TFDataType.Float),
                    dim: graph.Const(0, "make_batch")),
                size: graph.Const(new int[] { W, H }, "size")),
            y: graph.Const(Mean, "mean")),
        y: graph.Const(Scale, "scale")), destinationDataType);
}

Other solutions seem to create non accurate results.

Maybe somehow with a Mat object?

and my EDIT: I implemented something comparabel in c# in Unity and it works partially. It is just not accurate at all. How am i gonna find out the Mean? And i could not find anything about the rgb order.? I'm really new to this so maybe i have just overlooked it. (on Tensorflow.org) Using MobileNet trained in 1.4.

  public TFTensor transformInput(Color32[] pic, int texturewidth, int textureheight)
    {
        const int W = 224;
        const int H = 224;
        const float imageMean = 128;
        const float imageStd = 128;

        float[] floatValues = new float[texturewidth * textureheight * 3];

        for (int i = 0; i < pic.Length; ++i)
        {
            var color = pic[i];
            var index = i * 3;

            floatValues[index] = (color.r - imageMean) / imageStd;
            floatValues[index + 1] = (color.g - imageMean) / imageStd;
            floatValues[index + 2] = (color.b - imageMean) / imageStd;

        }
        TFShape shape = new TFShape(1, W, H, 3);
        return TFTensor.FromBuffer(shape, floatValues, 0, floatValues.Length);
    }

Upvotes: 3

Views: 4577

Answers (2)

Andrei Ashikhmin
Andrei Ashikhmin

Reputation: 2451

You probably didn't crop and scale your image before putting it into @sladomic function.

I managed to hack together a sample of using TensorflowSharp in Unity for object classification. It works with model from official Tensorflow Android example, but also with my self-trained MobileNet model. All you need is to replace the model and set your mean and std, which in my case were all equal to 224.

Upvotes: 2

sladomic
sladomic

Reputation: 876

Instead of feeding the byte array and then use DecodeJpeg, you could feed the actual float array, which you can get like this:

https://github.com/tensorflow/tensorflow/blob/3f4662e7ca8724f760db4a5ea6e241c99e66e588/tensorflow/examples/android/src/org/tensorflow/demo/TensorFlowImageClassifier.java#L134

float[] floatValues = new float[inputSize * inputSize * 3];
int[] intValues = new int[inputSize * inputSize];

bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());
for (int i = 0; i < intValues.length; ++i) {
      final int val = intValues[i];
      floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd;
      floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd;
      floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd;
}

Tensor<Float> input = Tensors.create(floatValues);

In order to use "Tensors.create()" you need to have at least Tensorflow version 1.4.

Upvotes: 5

Related Questions