CODEWITHSUNDEEP
numpymachine-learningdeep-learningperceptron
anki
anki

Reputation: 765

Decision boundary in perceptron not correct

I was preparing some code for a lecture and re-implemented a simple perceptron: 2 inputs and 1 output. Aim: a linear classifier.

Here's the code that creates the data, setups the perceptron and trains it:

from ipywidgets import interact
import numpy as np
import matplotlib.pyplot as plt

# Two randoms clouds
x = [(1,3)]*10+[(3,1)]*10
x = np.asarray([(i+np.random.rand(), j+np.random.rand()) for i,j in x])

# Colors
cs = "m"*10+"b"*10
# classes
y = [0]*10+[1]*10



class Perceptron:
    def __init__(self):
        self.w = np.random.randn(3)
        self.lr = 0.01

    def train(self, x, y, verbose=False):
        errs = 0.

        for xi, yi in zip(x,y):
            x_ = np.insert(xi, 0, 1)
            r = self.w @ x_

            ######## HERE IS THE MAGIC HAPPENING #####
            r = r >= 0
            ##########################################

            err = float(yi)-float(r)

            errs += np.abs(err)

            if verbose:
                print(yi, r)

            self.w = self.w + self.lr * err * x_

        return errs

    def predict(self, x):
        return np.round(self.w @ np.insert(x, 0, 1, 1).T)

    def decisionLine(self):
        w = self.w
        slope =  -(w[0]/w[2]) / (w[0]/w[1])
        intercept = -w[0]/w[2]
        return slope, intercept

p = Perceptron()

line_properties = []
errs = []

for i in range(20):
    errs.append(p.train(x, y, True if i == 999 else False))
    line_properties.append(p.decisionLine())

print(p.predict(x)) # works like a charm!



@interact
def showLine(i:(0,len(line_properties)-1,1)=0):
    xs = np.linspace(1, 4)
    a, b = line_properties[i]

    ys = a * xs + b

    plt.scatter(*x.T)
    plt.plot(xs, ys, "k--")

At the end, I am calculating the decision boundary, i.e. the linear eq. separating class 0 and 1. However, it seems to be off. I tried inversion etc but have no clue what is wrong. Interestingly, if I change the learning rule to

self.w = self.w + self.lr * err / x_

i.e. dividing by x_, it works properly - I am totally confused. Anyone an idea?

Solved for real

Now I added one small, but very important part to the Perceptron that I just forgot (and maybe others may forget it as well). You have to do the thresholded activation! r = r >= 0 - and now it is centered on 0 and then it does work - this is basically the answer below. If you don't do this, you have to change the classes to get again the center at 0. Currently, I prefer having the classes -1 and 1 as this gives a better decision line (centered) instead of a line that is very close to one of the data clouds.

Before:

enter image description here

Now:

enter image description here

Upvotes: 0

Views: 230

Answers (1)

Ben Reiniger
Ben Reiniger

Reputation: 12592

You are creating a linear regression (not logistic regression!) with targets 0 and 1. And the line you plot is the line where the model predicts 0, so it should ideally cut through the cloud of points labeled 0, as in your first plot.

If you don't want to implement the sigmoid for logistic regression, then at least you will want to display a boundary line that corresponds to a value of 0.5 rather than 0.

As for inverting the weights providing a plot that looks like what you want, I think that's just a coincidence of this data.

Upvotes: 1

Related Questions