CODEWITHSUNDEEP
matlabmachine-learningneural-network
user3123955
user3123955

Reputation: 2901

Creating a basic feed forward perceptron neural network for multi-class classification

I am new to neural networks and I want to create a feed forward neural network for mutli-class classification. I am allowed to use any code that is publicly available but not any MATLAB ToolBox as i don't have access to it (so no neural network toolbox). The goal is to classify the data into one of 10 classes. Here is the data set, the class is defined by the three letter code in the last column.

When creating a neural network, do you simply define the amount of nodes and have each node in layer i connect to every single node in layer i+1? And then simply have them learn the weights themselves?

Also is there a source I could follow that has MATLAB code for creating a neural network with any amount of input, any number of nodes, and does multi-class classification that is feed forward.

Upvotes: 2

Views: 7775

Answers (2)

greeness
greeness

Reputation: 16104

One thing that should help is to use cross-entropy error instead of classification error or mean-squared error (MSE) for such multi-class problem (especially for evaluation). This is a nice article that explains the idea. I will quote its example here:

Suppose we are predicting a person’s political party affiliation (democrat, republican, other) from independent data such as age, sex, annual income, and so on. ...

Now suppose you have just three training data items. Your neural network uses softmax activation for the output neurons so that there are three output values that can be interpreted as probabilities. For example suppose the neural network’s computed outputs, and the target (aka desired) values are as follows:

computed       | targets              | correct?
-----------------------------------------------
0.3  0.3  0.4  | 0  0  1 (democrat)   | yes
0.3  0.4  0.3  | 0  1  0 (republican) | yes
0.1  0.2  0.7  | 1  0  0 (other)      | no

This neural network has classification error of 1/3 = 0.33. Notice that the NN just barely gets the first two training items correct and is way off on the third training item. Now see another output as below:

computed       | targets              | correct?
-----------------------------------------------
0.1  0.2  0.7  | 0  0  1 (democrat)   | yes
0.1  0.7  0.2  | 0  1  0 (republican) | yes
0.3  0.4  0.3  | 1  0  0 (other)      | no

This NN also has a classification error of 1/3 = 0.33. But this second NN is much better than the first because it nails the first two training items and just barely misses the third training item. To summarize, classification error is a very crude measure of error. See below for a comparison of the classification error and Average cross-entropy error in the two cases:

Neural Network | classification error   | Average cross-entropy error
--------------------------------------------------------------------
NN1            |        0.33            |     1.38
NN2            |        0.33            |     0.64

To use cross-entropy error in the training, you need to use a different cost function. See details here.

enter image description here where m is the number of training examples and k is the number of classes; y is the label; x is the feature vector; \theta is the weight parameter.

Upvotes: 1

ASantosRibeiro
ASantosRibeiro

Reputation: 1257

A general introduction to neural networks (it seems you still need to learn a bit what they are): http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html

Read this document which explain how feedforward networks with backpropagation work (maths are important): http://galaxy.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html

Here you have one implementation in matlab with comments: http://anoopacademia.wordpress.com/2013/09/29/back-propagation-algorithm-using-matlab/

Regarding your questions:

1) "When creating a neural network, do you simply define the amount of nodes and have each node in layer i connect to every single node in layer i+1?" Depend of the network you use. In simple fully-connected feedforward neural nets, yes.

2)"And then simply have them learn the weights themselves?" That's the general idea. You have some data of which you know their classes (supervised learning), which you will give to the neural network to learn the pattern, and after learning is finished you use this updated weights to classify new, unseen data.

Upvotes: 2

Related Questions