CODEWITHSUNDEEP
pythonpytorch
user171780
user171780

Reputation: 3078

What are the expected values in the input in Pytorch?

I am new to Pytorch, following the tutorials. I want to implement a regresor for a nonlinear function with 4 real inputs and 2 real outputs. I cannot find anywhere what is the supposed range for the inputs and outputs. They should go between -1 and 1? Between 0 and 1? Can it be anything?

More details

I have written the following simple parameterized model for experimenting:

class Net(torch.nn.Module):
    def __init__(self, n_inputs: int, n_outputs: int, n_hidden_layers: int, n_nodes_per_hidden_layer: int):
        n_inputs = int(n_inputs)
        n_outputs = int(n_outputs)
        n_hidden_layers = int(n_hidden_layers)
        n_nodes_per_hidden_layer = int(n_nodes_per_hidden_layer)
        if any([i<=0 for i in [n_inputs,n_outputs,n_hidden_layers,n_nodes_per_hidden_layer]]):
            raise ValueError(f'All n_inputs, n_outputs, n_hidden_layers and n_nodes_per_hidden_layer must be greater than 0.')
        super().__init__()
        self.input_layer = torch.nn.Linear(n_inputs, n_nodes_per_hidden_layer)
        self.hidden_layers = [torch.nn.Linear(n_nodes_per_hidden_layer, n_nodes_per_hidden_layer) for i in range(n_hidden_layers)]
        self.output_layer = torch.nn.Linear(n_nodes_per_hidden_layer, n_outputs)

    def forward(self, x):
        x *= .1
        activation_function = torch.nn.functional.relu
        x = activation_function(self.input_layer(x))
        for idx,layer in enumerate(self.hidden_layers):
            x = activation_function(layer(x))
        x = self.output_layer(x)
        return x

and I instantiate it in this way:

dnn = Net(
    n_inputs = 4, # Defined by the number of observables (charge of each channel now).
    n_outputs = 2, # x and y.
    n_hidden_layers = 3, # Choose yourself.
    n_nodes_per_hidden_layer = 66, # Choose yourself.
)

My input x is data that distributes in a weird way from 0 to 1, my outputs are 2 values in the range 1e-2 +- 100e-6. I have tried x -= .5 and different scalings too, but cannot make it work. I don't get any error, just it does not seem to learn what it is supposed to learn.

I know that this model should work because I have used it with similar data that distributes in a similar way but the inputs in the range 0-100e-12 using x *= 1e9 and it was performing reasonably well. I don't know why, however.

Upvotes: 0

Views: 524

Answers (1)

Szymon Maszke
Szymon Maszke

Reputation: 24681

Data transformation

They should go between -1 and 1? Between 0 and 1? Can it be anything?

They can be any real valued numbers, but in general we standardize input values using mean and standard deviation (so the result has 0 mean and 1 variance) like this (for two dimensional data that you have, assuming samples are zeroth dimension and features are the first dimension):

import torch

samples, features = 128, 4
data = torch.randn(samples, features)
std, mean = torch.std_mean(data, dim=0, keepdim=True)

normalized = (data - mean) / std

In general neural networks work best with normalized input with similar ranges.

And this transformation could (and should as it will make it easier for nn.Linear layer output) also be applied to your regression target as it's reversible.

Other things

  • Use torch.nn.MSELoss for regression
  • Use standard optimizer (like Adam) with default learning rates (you can fine tune it later)
  • Make sure your pipeline works correctly

Upvotes: 1

Related Questions