CODEWITHSUNDEEP
neural-network
KansaiRobot
KansaiRobot

Reputation: 9912

Neural Network Retraining

I am coding a simple Neural Network, but I have thought of one issue that is bothering me. This NN is for finding categories in the input. To better understand this, say the categories are "the numbers" (0,1,2...9).

To implement this the output layer is 10 nodes. Say I train this NN with several input -output pairs and save the learned weights somewhere. As the learning process takes quite a lot of time, after that I go and take a break. Come fresh the next day and re-start learning with new input -output pairs. So fair so goo

But what happen if on that time, I decide that I want to recognize hexadecimals (0,1,...9,A,B,,,E,F)... ergo the categories are increasing.

I suspect that would imply changing the structure of the NN and therefore I should retrain the NN from scratch.

Is this so?

Any comment, advice or your share of experience will be greatly appreciated

EDIT: This question has been marked as duplicate. I read the other question and although similar, my question is more concrete. While the other question speaks in generalities and the answer also is quite general- mine is very concrete as I use an example:

If I train a NN to recognize decimal numbers and later on decide to add data to make it recognize hexadecimals, can this be possible? How? Do I have to retrain the whole NN? In other words, does the structure of the NN needs to stay stationary with 10 OR 16 outputs since the beginning?

I would very much appreciate for a concrete answer to this. Thanks

Upvotes: 0

Views: 3578

Answers (1)

villasv
villasv

Reputation: 6831

A few considerations

  1. Your training set and testing set should have the same distribution

    Unless you have some way of specifying sample weights like some algorithms can you should at all costs avoid training on biased data. This is true for machine learning in general, not only neural networks.

  2. Resuming training from a previous session is equivalent of using good initial values

    Technically, you're just using the previous network as initial value instead of a random value. You should keep training in the whole dataset as always, to avoid a biased network.

Short Answer

Yes, you should always retrain your network if by retrain, you mean doing a training routine with the full dataset.

If you just mean retrain as doing a really long training iteration, it isn't your choice anyway. You must always train the network until the training error and testing error (or cross validated error) converge. If you reuse the previously trained network, that will probably happen faster.

You see, this is true no matter what kind of model change. If you change the network architecture, or the dataset, or both (your example), or some other parameter.

Of course, if you change the network architecture, you're going to have a bit of trouble on reusing the previous network. You could reuse the learned parameters from nodes that were kept and randomly initialize the parameters for the new nodes.

Upvotes: 1

Related Questions