CODEWITHSUNDEEP
pythonnumpymachine-learningneural-networksoftmax
WhiteSolstice
WhiteSolstice

Reputation: 651

Neural Network with Softmax Output

I am currently learning coding for Multi-Layered Perception. For this MLP I attempted to use logistic sigmoidal for my hidden layers and Softmax for my output with an assumption of two class labels.

import theano
from theano import tensor as T
import numpy as np
import matplotlib.pyplot as plt


alpha = 0.1
#Alpha value
alpha = 2*alpha/2
no_iters = 1 #Trying to get 1 iteration to work first.

#Weight matrix to hidden layer (2 input into 2 neuron)
w_h = np.array([ [1.0, 2.0],
              [-2.0, 0.0] ])

#Bias to hidden layer need ( 2 Hidden Layer neurons)
b_h = np.array([3.0, -1])

#Weight matrix to output layer (2 input into 1 neuron)
w_o = np.array([[1.0],
                [1.0]])

#Bias to output layer (Only 1 bias for one output neuron)
b_o = np.array([-2.0])

# X Input Array (No of data rows, No of inputs)
x = np.array([[1.0, 2.0],
              [-2.0, 3.0]])

#Desired Outputs(2 data row = 2 desired output (Rows))
d = np.array([[0.0],
             [1.0]])

#Assume 2 class labels for the 2 data rows
k = np.array([[1.0, 0.0],
             [0.0, 1.0]])

for iter in range(no_iters):
    #Hidden Layer Functions
    s = np.dot(x,w_h)+ b_h
    z = 1.0/(1 + np.exp(-s))


    #Output Layer Functions (Softmax)
    u = np.dot(z, w_o)+b_o
    u_max = np.max(u, axis=1, keepdims=True)
    p = np.exp(u-u_max)/np.sum(np.exp(u-u_max), axis=1, keepdims=True)
    y = np.argmax(p, axis=1)

    #SoftMax Delta O
    delta_o = k - p

    #Delta for input layer (DZ = differentiation of function)
    dz = z*(1-z)
    delta_h = np.dot(delta_o, np.transpose(w_o))*dz

    #Assign new weight and bias to output layer
    dw = -np.dot(np.transpose(z),delta_o)
    db = -np.sum(delta_o, axis=0)
    w_o = w_o - dw * alpha
    b_o = b_o - db * alpha

    #Assign new weight and bias to hidden layer
    w_h = w_h + alpha*np.dot(np.transpose(x), delta_h)
    b_h = b_h + alpha*np.sum(np.transpose(delta_h), axis=1)

    print(z)
    print(y)

When the code is executed there will be issues with matrix dot product for delta_h = np.dot(delta_o, np.transpose(w_o))*dz. Due to delta_o being a 2x2 matrix and transpose(w_o) being a 1x2 matrix.

Am I using the wrong formula to approach this problem?

Upvotes: 0

Views: 528

Answers (1)

Subham Mukherjee
Subham Mukherjee

Reputation: 789

You cannot multiply two tensors of different sizes. What you can do is you can get the mean of the error vector you are getting and do element-wise modification in the weights. That will not affect the performance and will solve the error I hope.

Upvotes: 1

Related Questions