Tensorflow multi-variable logistic regression not working

Question

I am trying to create a program which will classify a point as either 1 or 0 using Tensorflow. I am trying to create an oval shape around the center of this plot, where the blue dots are:

Everything in the oval should be classified as 1, every thing else should be 0. In the graph above, the blue dots are 1s and the red x's are 0s.

However, every time I try to classify a point, it always choses 1, even if it was a point I trained it with, saying it was 0.

My question is simple: Why is the guess always 1, and what am I doing wrong or should do differently to fix this problem? This is my first machine learning problem I have tried without a tutorial, so I really don't know much about this stuff.

I'd appreciate any help you can give, thanks!

Here's my code:

#!/usr/bin/env python3

import tensorflow as tf
import numpy
import matplotlib.pyplot as plt

training_in = numpy.array([[0, 0], [1, 1], [2, 0], [-2, 0], [-1, -1], [-1, 1], [-1.5, 1],   [3, 3], [3, 0], [-3, 0], [0, -3], [-1, 3], [1, -2], [-2, -1.5]])
training_out = numpy.array([1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0])

def transform_data(x):
    return [x[0], x[1], x[0]**2, x[1]**2, x[0]*x[1]]

new_training_in = numpy.apply_along_axis(transform_data, 1, training_in)

feature_count = new_training_in.shape[1]

x = tf.placeholder(tf.float32, [None, feature_count])
y = tf.placeholder(tf.float32, [None, 1])

W = tf.Variable(tf.zeros([feature_count, 1]))
b = tf.Variable(tf.zeros([1]))

guess = tf.nn.softmax(tf.matmul(x, W) + b)

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(tf.matmul(x, W) + b, y))

opti = tf.train.GradientDescentOptimizer(0.01).minimize(cost)

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

for i in range(1000):
    for (item_x, item_y) in zip(new_training_in, training_out):
        sess.run(opti, feed_dict={ x: [item_x], y: [[item_y]]})

print(sess.run(W))
print(sess.run(b))

plt.plot(training_in[:6, 0], training_in[:6, 1], 'bo')
plt.plot(training_in[6:, 0], training_in[6:, 1], 'rx')

results = sess.run(guess, feed_dict={ x: new_training_in })

for i in range(training_in.shape[0]):
    xx = [training_in[i:,0]]
    yy = [training_in[i:,1]]
    res = results[i]

    # this always prints `[ 1.]`
    print(res)

    # uncomment these lines to see the guesses
    # if res[0] == 0:
    #     plt.plot(xx, yy, 'c+')
    # else:
    #     plt.plot(xx, yy, 'g+')

plt.show()

Answer 1

The problem occurs when you use softmax_cross_entropy_with_logits. In your concrete case, both logits and labels should have shape [batch_size, number_of_labels=2].

Note that your tensors logits=tf.matmul(x, W) + b and labels=y have shape [batch_size, 1], so Tensorflow is assuming that number_of_labels=1. That's why your guess is always the same.

A) You could solve this problem by encoding training_out as a one-hot vector. I recommend using np.eye() to achieve that:

training_out = [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]
training_out = numpy.eye(2)[training_out]

Then, you will need to make the following changes:

y = tf.placeholder(tf.float32, [None, 2])
W = tf.Variable(tf.zeros([feature_count, 2]))
b = tf.Variable(tf.zeros([2]))
...
for i in range(1000):
    for (item_x, item_y) in zip(new_training_in, training_out):
        sess.run(opti, feed_dict={x: [item_x], y: [item_y]})
...
results = sess.run(guess, feed_dict={x: new_training_in})[:,1]

B) Alternatively, you could use sparse_softmax_cross_entropy_with_logits, which allows labels to have shape [batch_size]. I've tweaked your code to make it work in this way:

import tensorflow as tf
import numpy
import matplotlib.pyplot as plt

training_in = numpy.array(
    [[0, 0], [1, 1], [2, 0], [-2, 0], [-1, -1], [-1, 1], [-1.5, 1], [3, 3], [3, 0], [-3, 0], [0, -3], [-1, 3], [1, -2],
     [-2, -1.5]])
training_out = numpy.array([1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0])

def transform_data(x):
    return [x[0], x[1], x[0] ** 2, x[1] ** 2, x[0] * x[1]]

new_training_in = numpy.apply_along_axis(transform_data, 1, training_in)

feature_count = new_training_in.shape[1]

x = tf.placeholder(tf.float32, [None, feature_count])
y = tf.placeholder(tf.int32, [None])

W = tf.Variable(tf.zeros([feature_count, 2]))
b = tf.Variable(tf.zeros([2]))

guess = tf.nn.softmax(tf.matmul(x, W) + b)

cost = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(tf.matmul(x, W) + b, y))

opti = tf.train.GradientDescentOptimizer(0.01).minimize(cost)

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

for i in range(1000):
    for (item_x, item_y) in zip(new_training_in, training_out):
        sess.run(opti, feed_dict={x: [item_x], y: [item_y]})

print(sess.run(W))
print(sess.run(b))

plt.plot(training_in[:6, 0], training_in[:6, 1], 'bo')
plt.plot(training_in[6:, 0], training_in[6:, 1], 'rx')

results = sess.run(guess, feed_dict={x: new_training_in})

for i in range(training_in.shape[0]):
    xx = [training_in[i:, 0]]
    yy = [training_in[i:, 1]]
    res = results[i]

    # this always prints `[ 1.]`
    print(res)

    # uncomment these lines to see the guesses
    if res[0] == 0:
        plt.plot(xx, yy, 'c+')
    else:
        plt.plot(xx, yy, 'g+')
plt.show()