CODEWITHSUNDEEP
neural-networklayeractivation-function
SMS
SMS

Reputation: 382

Neural network - exercise

I am currently learning for myself the concept of neural networks and I am working with the very good pdf from http://neuralnetworksanddeeplearning.com/chap1.html

There are also few exercises I did, but there is one exercise I really dont understand, at least one step

Task: There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01. enter image description here

I found also the solution, as can be seen on the second image enter image description here

I understand why the matrix has to have this shape, but I really struggle to understand the step, where the user calculates

0.99 + 3*0.01
4*0.01

I really don't understand these two steps. I would be very happy if someone can help me understand this calculation

Thank you very much for help

Upvotes: 3

Views: 1692

Answers (1)

viceriel
viceriel

Reputation: 883

Output of previous layer is 10x1(x). Weight matrix is 4x10. New output layer will be 4x1. There are two assumption first:

  • x is 1 only at one row. xT= [1 0 0 0 0 0 0 0 0 0]. If you multiple this vector with matrix W your output will be yT=[0 0 0 0], because there is only 1 in x. After multiplication by W will be this only 1 multiple by 0th column of W which are zeroes.

  • Second assumption is, what if x is not 1 anymore, instead of one x can be xT=[0.99 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01]. And if you perform multiplication of x with first row of W result is 0.05(I believe here is typo). When xT=[0.01 0.99 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01] after multiplication with first row of W result is 1.03. Because:

0.01*0 + 0.99*1 + 0.01*0 + 0.01*1 + 0.01*0 + 0.01*1 + 0.01*0 + 0.01*1 + 0.01*0 + 0.01*1 = 1.03

So I believe there is a typo, because author probably assume 4 ones at first row of W, which is not true, because there is 5 ones. Because if there was 4 ones at first first row, than really results will be 0.04 for 0.99 at first row of x and 1.02 for 0.99 at second row of x.

Upvotes: 3

Related Questions