CODEWITHSUNDEEP
machine-learningtensorflowdeep-learning
Transcendental
Transcendental

Reputation: 929

A Simple Network on TensorFlow

I was trying to train a very simple model on TensorFlow. Model takes a single float as input and returns the probability of input being greater than 0. I used 1 hidden layer with 10 hidden units. Full code is shown below:

import tensorflow as tf
import random 

# Graph construction

x = tf.placeholder(tf.float32, shape = [None,1])
y_ = tf.placeholder(tf.float32, shape = [None,1])

W = tf.Variable(tf.random_uniform([1,10],0.,0.1))
b = tf.Variable(tf.random_uniform([10],0.,0.1))

layer1 = tf.nn.sigmoid( tf.add(tf.matmul(x,W), b) )

W1 = tf.Variable(tf.random_uniform([10,1],0.,0.1))
b1 = tf.Variable(tf.random_uniform([1],0.,0.1))

y = tf.nn.sigmoid( tf.add( tf.matmul(layer1,W1),b1) )

loss = tf.square(y - y_)

train_step = tf.train.GradientDescentOptimizer(0.01).minimize(loss)

# Training

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    N = 1000
    while N != 0:
        batch = ([],[])
        u = random.uniform(-10.0,+10.0)
        if u >= 0.:
            batch[0].append([u])
            batch[1].append([1.0])
        if  u < 0.:
            batch[0].append([u])
            batch[1].append([0.0])

        sess.run(train_step, feed_dict = {x : batch[0] , y_ : batch[1]} )
        N -= 1

    while(True):
        u = raw_input("Give an x\n")
        print sess.run(y, feed_dict = {x : [[u]]})

The problem is, I am getting terribly unrelated results. Model does not learn anything and returns irrelevant probabilities. I tried to adjust learning rate and change variable initialization, but I did not get anything useful. Do you have any suggestions?

Upvotes: 0

Views: 149

Answers (1)

fabmilo
fabmilo

Reputation: 48330

You are computing only one probability what you want is to have two classes:

  • greater/equal than zero.
  • less than zero.

So the output of the network will be a tensor of shape two that will contain the probabilities of each class. I renamed y_ in your example to labels:

labels = tf.placeholder(tf.float32, shape = [None,2])

Next we compute the cross entropy between the result of the network and the expected classification. The classes for positive numbers would be [1.0, 0] and for negative numbers would be [0.0, 1.0]. The loss function becomes:

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, labels)
loss = tf.reduce_mean(cross_entropy)

I renamed the y to logits as that is a more descriptive name.

Training this network for 10000 steps gives:

Give an x
3.0
[[ 0.96353203  0.03686807]]
Give an x
200
[[ 0.97816485  0.02264325]]
Give an x
-20
[[ 0.12095013  0.87537241]]

Full code:

import tensorflow as tf
import random

# Graph construction

x = tf.placeholder(tf.float32, shape = [None,1])
labels = tf.placeholder(tf.float32, shape = [None,2])

W = tf.Variable(tf.random_uniform([1,10],0.,0.1))
b = tf.Variable(tf.random_uniform([10],0.,0.1))

layer1 = tf.nn.sigmoid( tf.add(tf.matmul(x,W), b) )

W1 = tf.Variable(tf.random_uniform([10, 2],0.,0.1))
b1 = tf.Variable(tf.random_uniform([1],0.,0.1))

logits = tf.nn.sigmoid( tf.add( tf.matmul(layer1,W1),b1) )

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, labels)

loss = tf.reduce_mean(cross_entropy)

train_step = tf.train.GradientDescentOptimizer(0.01).minimize(loss)

# Training

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    N = 1000
    while N != 0:
        batch = ([],[])
        u = random.uniform(-10.0,+10.0)
        if u >= 0.:
            batch[0].append([u])
            batch[1].append([1.0, 0.0])
        if  u < 0.:
            batch[0].append([u])
            batch[1].append([0.0, 1.0])

        sess.run(train_step, feed_dict = {x : batch[0] , labels : batch[1]} )

        N -= 1

    while(True):
        u = raw_input("Give an x\n")
        print sess.run(logits, feed_dict = {x : [[u]]})

Upvotes: 2

Related Questions