Retrain Artificial Neural Network for PHP5

Question

So in my web app I managed to connect a <canvas> to ANN for PHP5 (http://ann.thwien.de/index.php/Main_Page). I train it like this:

<?php
require_once 'ANN/Loader.php';

$data = $_POST['data'];

$data = array_map('floatval', $data);

use ANN\Network;
use ANN\Values;

try
{
    $objNetwork = Network::loadFromFile('xor.dat');
}
catch(Exception $e)
{
    print 'Creating a new one...';

    $objNetwork = new Network;

    $objValues = new Values;

    $objValues->train();
    call_user_func_array(array($objValues, 'input'), $data);
    $objValues->output(1);

    $objValues->saveToFile('values_xor.dat');

    unset($objValues);
}

try
{
    $objValues = Values::loadFromFile('values_xor.dat');
}
catch(Exception $e)
{
    die('Loading of values failed');
}

$objNetwork->setValues($objValues); // to be called as of version 2.0.6

$boolTrained = $objNetwork->train();

print ($boolTrained)
    ? 'Network trained'
    : 'Network not trained completely. Please re-run the script';

$objNetwork->saveToFile('xor.dat');

$objNetwork->printNetwork();

And if I need to check data against the Neural Network, I can do that like this:

<?php
require_once 'ANN/Loader.php';

$data = $_POST['data'];

$data = array_map('floatval', $data);

use ANN\Network;
use ANN\Values;

try
{
    $objNetwork = Network::loadFromFile('xor.dat');
}
catch(Exception $e)
{
    die('Network not found');
}

try
{
    $objValues = Values::loadFromFile('values_xor.dat');
}
catch(Exception $e)
{
    die('Loading of values failed');
}

call_user_func_array(array($objValues, 'input'), $data);

$objNetwork->setValues($objValues);

print_r($objNetwork->getOutputs());

However, what I want ultimately, is that if the Neural Network recognizes a gesture, it trains with that data. So, ideally it becomes even more accurate over time. I've been busting my head for days trying to figure this out, but I've got nothing.

Answer 1

Observations

Most classical ANN architectures follow the procedure of

• operating two modes: training and mapping

after training the weights (and hence the network's memory) are fixed.

There are some models which attempt to add new knowledge to an existing data set. Have a look at Adaptive Resonance Theory, ART neural networks. However they don't necessarily assimilate the new knowledge. They may add a new output or set of neurons to classify the new item.

The goto ANN for pattern recognition, which your problem seems to be is a Feed Forward network/ multi-layer peceptron, trained by Back propagation (BP).

I looked at your link and it appears to be using a "multilayer perceptron", so you may be stuck with this architecture.

Problem

Your problem here is that if your network which has been trained on (N) patterns and produced (O) outputs, is presented with a new pattern (p). It classifies (P) as belonging to the same pattern (n) or group of patterns(ns) that produced an output (o).

I don't know if your training data is adding a 1 training pattern = 1 output or several training patterns = 1 output.

Solution

The brute force method is to completely retrain your network with (p) added to your training data.

There are other options.

BP follows gradient decent so at the end of each training cycle you will get an error. BP usually stops when this error is below a chosen level error_t. (see note at end). This is not a great way of doing it and leads to guesswork and over/under training. Assuming it has actually reached a target level and not got caught in a local minima.

Initially your error is going to be high, as it gets close to your error_t you can save a copy of the weights. These represent your network in a partially trained state (error_p).

When you want to introduce a pattern (p) you can restart your training with these partially trained weights, instead of the initial or new randomised ones. The error should initially up from (error_p) and then start to decline. Save the new partially trained weights when it reaches error_p again. The let it finish its training and put it back to work.

Alternatively. (1) When you want to include (p). Retrain your network only using either (p) or the group of training patterns to which you want to add (p). This is going to alter the recognition of all your other pattern-output pairs.

Note that your trained network originality took the new pattern p and the error on it was sufficiently low that it said "I know you". This means that when you present trained weights and p to your training algorithm, it may already be below your target error and you will only get 1 training cycle.

So I suggest only doing 1 training cycle as mentioned at (1). Then testing on all your training data + p. If the error is below error_t consider it trained, if not let it run on until you get back down to error_t.

note at end

Stopping a BP at an arbitrary error is the basic way of training. Doing this can leave the network not trained enough so it does not recognise what you want or "over fitted" so it only recognises your training patterns and reject your real world patterns.

A different method is to have a a training set of patterns and a validation set. You train the network with the training data but don't look at the error as a stop flag. Instead after each training cycle you show the network the validation pattens and if it recognises them you stop training. This is your validation error_v.

It gets a bit worse.

Supposing you have N training patterns. At the start of every training cycle you show the network pattern[1]. The network produces an error (Desired Output - Actual Output), you back propagate this error thought then network and it means that next time it sees pattern[1] the error should be lower. What it has done here is modify the weights a little.

Now yo do the same for pattern[2] ,..., and so on until pattern[N].

Each time the network adjusts its weight for pattern[n], it affects the recognition of the previous pattern[1 to n-1]. So by the time you have gotten to pattern[N]. A lot of damage can have occurred to the previous learning. The worst damage theoretically should have happened to pattern[1]'s learning but it actually depends on shared similarity between the patterns and the mathematics of the pattern variables vs. the output variables.

The reason I am telling you this is that if you are going to represent a new pattern to a trained network you are performing a bit of this.

You can fix/alleviate it by presenting your training patterns in a random order (or stepped order) on each training cycle, so pattern[1] is not always seen first and pattern[N] is not always seen last.