neural network from scratch in python using sigmoid activation

Question

i am new to python,trying to learn machine learning in python.i have tried to write a neural network from scratch with one hidden layer on the famous iris dataset.this is a three class classifier with out put as one hot vectors.i have also taken help from already written algos for help.for instance i used the same training set as my testing set.

it is a huge code to go through,i would like you to tell me, that how do we subtract 'y' output( which is one hot vector) of dimensions (150,3) and my out y softmax will be of vector (150,21).this is my biggest problem.i tried to look online everyone have used this method but since i am weak in python i don't understand it.this is the line of code delta3[range(m1), y] -= 1 arrays used as indices must be of integer (or boolean) type if m1 is sie of(150) and if i give size m1(150,3) then delta3[range(m1), y] -= 1 TypeError: range() integer end argument expected, got tuple.

remember m1=150
my y vector=150,3
softmax=150,21

my code is

#labels or classes
#1=iris-setosa
#2=iris-versicolor
#0=iris-virginica

#features
#sepallength
#sepalwidth
#petallengthcm
#petalwidth


import pandas as pd
import matplotlib.pyplot as plt
import csv
import numpy as np
df=pd.read_csv('Iris.csv')

df.convert_objects(convert_numeric=True)
df.fillna(0,inplace=True)

df.drop(['Id'],1,inplace=True)
#function to convert three labels into values 0,1,2
def handle_non_numericaldata(df):
    columns=df.columns.values
    for column in columns:
        text_digit_vals={}
        def convert_to_int(val):
            return text_digit_vals[val]
        if df[column].dtype!=np.int64 and df[column].dtype!=np.float:       
            column_contents=df[column].values.tolist()
            unique_elements=set(column_contents)
            x=0
            for unique in unique_elements:
                if unique not in text_digit_vals:
                    text_digit_vals[unique]=x
                    x+=1
            df[column]=list(map(convert_to_int,df[column]))
     return(df)
handle_non_numericaldata(df)    

x=np.array(df.drop(['Species'],1).astype(float))
c=np.array(df['Species'])


n_values=(np.max(c)+1)
y=(np.eye(n_values)[c])

m1=np.size(c)

theta=np.ones(shape=(4,1))
theta2=np.ones(shape=(1,21))
#no of examples "m"

#learning rate alpha
alpha=0.01
#regularization parameter
lamda=0.01
for i in range(1,1000):
    z1=np.dot(x,theta)  
    sigma=1/(1+np.exp(-z1))

    #activation layer 2.
    a2=sigma
    z2=np.dot(a2,theta2)

    probs=np.exp(z2)

    softmax=probs/np.sum(probs,axis=1,keepdims=True)
    delta3=softmax



    delta3[range(m1), y] -= 1

    A2=np.transpose(a2)
    dw2 = (A2).dot(delta3)
    W2=np.transpose(theta2)

    delta2=delta3.dot(W2)*sigma*(1-sigma)
    X2=np.transpose(x)
    dw1=np.dot(X2,delta2)


    dw2=dw2-lamda*theta2
    dw1=dw1-lamda*theta

    theta =theta -alpha* dw1

     theta2= theta2-alpha * dw2
    correct_logprobs=0
    correct_logprobs=correct_logprobs-np.log(probs[range(m1),y])
    data_loss=np.sum(correct_logprobs)
    data_loss+=lamda/2*(np.sum(np.square(theta))+ np.square(theta2))
    loss=1./m1*data_loss
    if 1000%i==0:
        print("loss after iteration%i:%f",loss)


final1=x.dot(theta)
sigma=1/(1+np.exp(-final1))
z2=sigma.dot(theta2)
exp_scores=np.exp(z2)
probs=exp_scores/np.sum(exp_scores,axis=1,keepdims=True)
print(np.argmax(probs,axis=1))

Answer 1

What is a Neural Network?

The term ‘Neural’ has origin from the human (animal) nervous system’s basic functional unit ‘neuron’ or nerve cells present in the brain and other parts of the human (animal) body. A neural network is a group of algorithms that certify the underlying relationship in a set of data similar to the human brain. The neural network helps to change the input so that the network gives the best result without redesigning the output procedure

Now in code Example:_

import numpy as np

#assign input values

 input_value=np.array([[0.26,0.77,0.25],[0.42,0.8,0.25],[0.56,0.53,0.25],[0.29,0.79,0.25]])
    input_value.shape

#assign output values

 output=np.array([0.644045,0.651730,0.707523,0.644395])
    output=output.reshape(4,1)
    output

#assign weights

 weights=np.array([[0.1],[0.1],[0.1]])
    weights.shape
    weights

#add bias

 bias=0.3

#activation function

 def sigmoid_func(x):
        return 1/(1+np.exp(-x))

#derivative of sigmoid function

def der(x):

return sigmoid_func(x)*(1-sigmoid_func(x))

#updating weights

 for epochs in range(10000):
        input_arr=input_value
        #print(input_arr)
        weighted_sum=np.dot(input_arr,weights)+bias

### CALCULATION OF PRE ACTIVATION FUNCTION

first_output=sigmoid_func(weighted_sum)
#print(first_output)
error=first_output - output
#print(error)
total_error=np.square(np.subtract(first_output,output)).mean()
#print total error
first_der=error
second_der=der(first_output)
derivative=first_der*second_der
t_input=input_value.T
final_derivative=np.dot(t_input,derivative)
#update Weigths
weights=weights-0.05*final_derivative
#update bias
for i in derivative:
    bias=bias-0.05*i
    
    
print(weights)
print(bias)

#prediction for 1st item

pred=np.array([0.26,0.77,0.25])
result=np.dot(pred,weights)+bias
res=sigmoid_func(result)
print(res)

#prediction for 2nd item

pred=np.array([0.42,0.8,0.25])
result=np.dot(pred,weights)+bias
res=sigmoid_func(result)
print(res)
    

#prediction for 3rd item

  pred=np.array([0.56,0.53,0.25])
    result=np.dot(pred,weights)+bias
    res=sigmoid_func(result)
    print(res)

#prediction for 4th item

 pred=np.array([0.29,0.79,0.25])
    result=np.dot(pred,weights)+bias
    res=sigmoid_func(result)
    print(res)

Answer 2

In Python range generates a tuple of numbers from x to y with range(x, y). If you generate something like range(10) then it is the same as (0, 1, 2, 3, 4, 5, 6, 7, 8, 9). Lists in Python need an integer index such as list[0] or list[4], not list[0, 4], however, there is a built-in thing in Python that allows access from index x to index y in a list here is the syntax: list[0:4]. This will return every value from 0 to 3 in the list. Such as if a list is list = [0,10,3,4,12,5,3] then list[0:4] will return [0,10,3,4].

Try taking a look at list data structures in Python on the Python Docs. As well as Understanding Generators in Python.

I think what your looking for is something like this: delta3 = [[z-1 for z in delta3[x:y]] for x in range(m1)]. This list comprehension uses two generations both, [x-1 for x in l], which subtracts one from every element in the list, and [l[x:y] for x in range(m)], which generates a list of lists with values through x to y in a range of m. Though I'm not sure I understand what your end goal is, fully.