CODEWITHSUNDEEP
c#tensorflowunity-game-engine
Grant Allan
Grant Allan

Reputation: 319

Converting color struct to tensor for input into an onnx model

In Unity, I'm working to feed input from the computer's webcam into a model I've already trained to recognize faces. I already have the webcam streaming frames into my code and my model loaded and ready to take input. But having never worked with C# until this project, I'm struggling to turn the webcam's texture into a tensor.

Here's my code so far:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

using System.Runtime.InteropServices;
using UnityEngine.UI;
using Unity.Barracuda;

public class EmotionModel : MonoBehaviour
{
    public RawImage rawImage;
    public NNModel dummyModel;

    private WebCamTexture camTexture;

    // Start is called before the first frame update
    void Start()
    {
        // Use cam_devices to allow a person to select their desired camera
        // For debugging purposes, prints available devices to the console
        WebCamDevice[] cam_devices = WebCamTexture.devices;
        for (int i = 0; i < cam_devices.Length; i++)
            Debug.Log($"Webcam available: {cam_devices[i].name}");

        // Assuming the first available WebCam is desired
        camTexture = new WebCamTexture(cam_devices[0].name);
        rawImage.texture = camTexture;
        rawImage.material.mainTexture = camTexture;
        if (camTexture != null)
            Debug.Log($"Streaming [{cam_devices[0].name}]");
            camTexture.Play();
    }

    // Update is called once per frame
    void Update()
    {
        Debug.Log("");
        Debug.Log("");
        Debug.Log("EmotionModel Update Called");

        // Create worker for the model
        IWorker worker = dummyModel.CreateWorker();

        // Get current frame
        // They're float values of 0 to 1. Which is what we want; our
        // input needs to be normalized to be 0 to 1 values anyways.
        Color[] frameColors = camTexture.GetPixels();
        Debug.Log($"Red: {frameColors[0].r}");
        Debug.Log($"Green: {frameColors[0].g}");
        Debug.Log($"Blue: {frameColors[0].b}");
        /* SOMEHOW convert this stupid thing into the tensor down there ↓ */

        // Execute neural network with specific input and get results back
        //Tensor inputImage = new Tensor(1, frameColors[0], frameColors[1], frameColors[2], 3);
        Tensor inputImage = new Tensor(1, 224, 224, 3);
        var outputPreds = worker.Execute(inputImage).PeekOutput();

        // Access values of the output tensor causing the main thread to block until neural network execution is done
        var predIndex = outputPreds.ArgMax()[0];
        
        Debug.Log($"Predicted Index: {predIndex}");
        Debug.Log($"Prediction Confidence: {outputPreds[predIndex]}");

        // Disposes
        try {
            worker.Dispose();
        } catch (NullReferenceException e) {
            Debug.Log("Failed to dispose of worker (does worker exist?).");
        }
        try {
            inputImage.Dispose();
        } catch (NullReferenceException e) {
            Debug.Log("Failed to dispose of inputImage tensor (does inputImage exist?).");
        }
    }
}

As you can see in this line:

Color[] frameColors = camTexture.GetPixels();

I've converted the texture into a Color structure. That lets me access the pixel values; you can me printing out the rgb values of the first pixel right below it. But I can't figure out where to go from there.

Upvotes: 0

Views: 437

Answers (1)

derHugo
derHugo

Reputation: 90679

I think you simply want to use another constructor.

There is Tensor(UnityEngine.Texture srcTexture, int channels = 3, string n = "") which looks exactly like what you are trying to achieve.

So probably simply

var inputImage = new Tensor(camTexture);

Upvotes: 0

Related Questions