Finding patterns in time series with PyTorch

Question

I started PyTorch with image recognition. Now I want to test (very basically) with pure NumPy arrays. I struggle with getting the setup to work, so basically I have vectors with values between 0 and 1 (normalized curves). Those vectors are always of length 1500 and I want to find e.g. "high values at the beginning" or "sine wave-like function", "convex", "concave" etc. stuff like that, so just shapes of those curves.
My training set consists of many vectors with their classes; I have chosen 7 classes. The net should be trained to classify a vector into one or more of those 7 classes (not one hot).

I'm struggling with multiple issues, but first my very basic Net

class Net(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(Net, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.rnn = nn.RNN(input_dim, hidden_dim, layer_dim)
        self.fc = nn.Linear(self.hidden_dim, output_dim)

    def forward(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(1), self.hidden_dim).requires_grad_()
        out, h0 = self.rnn(x, h0.detach())
        out = out[:, -1, :]
        out = self.fc(out)
        return out

network = Net(1500, 70, 20, 7)
optimizer = optim.SGD(network.parameters(), lr=learning_rate, momentum=momentum)

This is just a copy-paste from an RNN demo. Here is my first issue. Is an RNN the right choice? It is a time series, but then again it is an image recognition problem when plotting the curve.

Now, this here is an attempt to batch the data. The data object contains all training curves together with the correct classifiers.

def train(epoch):
    network.train()
    network.float()
    batching = True
    index = 0
    # monitor the cummulative loss for an epoch
    cummloss = []
    # start batching some curves
    while batching:
        optimizer.zero_grad()


        # here I start clustering come curves to a batch and normalize the curves
        _input = []
        batch_size = min(len(data)-1, index+batch_size_train) - index
        for d in data[index:min(len(data)-1, index+batch_size_train)]:
            y = np.array(d['data']['y'], dtype='d')
            y = np.multiply(y, y.max())
            y = y[0:1500]
            y = np.pad(y, (0, max(1500-len(y), 0)), 'edge')
            if len(_input) == 0:
                _input = y
            else:
                _input = np.vstack((_input, y))
        input = torch.from_numpy(_input).float()
        input = torch.reshape(input, (1, batch_size, len(y)))
        target = np.zeros((1,7))
        # the correct classes have indizes, to I create a vector with 1 at the correct locations
        for _index in np.array(d['classifier']):
            target[0,_index-1] = 1
        target = torch.from_numpy(target)


        # get the result form the network
        output = network(input)
        # is this a good loss function?
        loss = F.l1_loss(output, target)
        loss.backward()
        cummloss.append(loss.item())
        optimizer.step()
        index = index + batch_size_train
        if index > len(data):
            print(np.mean(cummloss))
            batching = False

for e in range(1, n_epochs):
    print('Epoch: ' + str(e))
    train(0)

The problem I'm facing right now is, the loss doesn't change very little, even with hundreds of epochs.

Are there existing examples of this kind of problem? I didn't find any, just pure png/jpg image recognition. When I convert the curves to png then I have a little issue to train a net, I took densenet and it worked just fine but it seems to be super overkill for this simple task.

Answer 1

Image Recognition is different from Time-Series data. In the imaging domain your data-set might have more similarity with problems like Activity-Recognition, Video-Recognition which have temporal component. So, I'd recommend looking into some models for those.

As for the current model, I'd recommend using LSTM instead of RNN. And also for classification you need to use an activation function in your final layer. This should softmax with cross entropy based loss or sigmoid with MSE loss.

Keras has a Timedistributed model which makes it easy to handle time components. You can use a similar approach with Pytorch by applying linear layers followed by LSTM.

Look into these for better undertsanding ::

Activity Recognition : https://www.narayanacharya.com/vision/2019-12-30-Action-Recognition-Using-LSTM

https://discuss.pytorch.org/t/any-pytorch-function-can-work-as-keras-timedistributed/1346

How to implement time-distributed dense (TDD) layer in PyTorch

Activation Function ::

https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html

Answer 2

This is just a copy-paste from an RNN demo. Here is my first issue. Is an RNN the right choice?

In theory what model you choose does not matter as much as "How" you formulate your problem.

But in your case the most obvious limitation you're going to face is your sequence length: 1500. RNN store information across steps and typically runs into trouble over long sequence with vanishing or exploding gradient.

LSTM net have been developed to circumvent this limitations with memory cell, but even then in the case of long sequence it will still be limited by the amount of information stored in the cell.

You could try using a CNN network as well and think of it as an image.

Are there existing examples of this kind of problem?

I don't know but I might have some suggestions : If I understood your problem correctly, you're going from a (1500, 1) input to a (7,1) output, where 6 of the 7 positions are 0 except for the corresponding class where it's 1.

I don't see any activation function, usually when dealing with multi class you don't use the output of the dense layer to compute the loss you apply a normalizing function like softmax and then you can compute the loss.

Answer 3

From your description of features you have in the form of sin like structures, the closes thing that comes to mind is frequency domain. As such, if you have and input image, just transform it to the frequency domain by a Fourier transform and use that as your feature input.

Might be best to look for such projects on the internet, one such project that you might want to read the research paper or video from this group (they have some jupyter notebooks for you to try) or any similar works. They use the furrier features, that go though a multi layer perceptron (MLP).

I am not sure what exactly you want to do, but seems like a classification task, you would use RNN if you want your neural network to work with a sequence. To me it seems like the 1500 dimensions are independent, and as such can be just treated as input.

Regarding the last layer, for a classification problem it usually is a probability distribution obtained by applying softmax (if only the classification is distinct - i.e. probability sums up to 1), in which, given an input, the net gives a probability of it being from each class. If we are predicting multiple classes we are going to use sigmoid as the last layer of the neural network.

Regarding your loss, there are many losses you can try and see if they are better. Once again, for different features you have to know what exactly is the measurement of distance (a.k.a. how different 2 things are). Check out this website, or just any loss function explanations on the net.

So you should try a simple MLP on top of fourier features as a starting point, assuming that is your feature vector.