CODEWITHSUNDEEP
tensorflowlinear-regressiongradient-descent
brain_team
brain_team

Reputation: 31

My TensorFlow Gradient Descent diverges

import tensorflow as tf
import pandas as pd
import numpy as np

def normalize(data):
    return data - np.min(data) / np.max(data) - np.min(data)

df = pd.read_csv('sat.csv', skipinitialspace=True)
x_reading = df['reading_score']
x_math = df['math_score']
x_reading, x_math = np.array(x_reading[df.reading_score != 's']), np.array(x_math[df.math_score != 's'])

x_data = normalize(np.float32(np.array([x_reading, x_math])))

y_writing = df[['writing_score']]
y_data = normalize(np.float32(np.array(y_writing[df.writing_score != 's'])))

W = tf.Variable(tf.random_uniform([1, 2], -.5, .5)) #float32
b = tf.Variable(tf.ones([1]))
y = tf.matmul(W, x_data) + b

loss = tf.reduce_mean(tf.square(y - y_data.T))
optimizer = tf.train.GradientDescentOptimizer(0.005)
train = optimizer.minimize(loss)

init = tf.initialize_all_variables()

with tf.Session() as sess:
    sess.run(init)

    for step in range(1000):
        sess.run(train)
        print step, sess.run(W), sess.run(b), sess.run(loss)

Here's my code. My sat.csv contains a data of reading, writing and math scores at SAT. As you can guess, the difference between the features is not that big.

This is a part of sat.csv. 

DBN,SCHOOL NAME,Num of Test Takers,reading_score,math_score,writing_score
01M292,HENRY STREET SCHOOL FOR INTERNATIONAL STUDIES,29,355,404,363
01M448,UNIVERSITY NEIGHBORHOOD HIGH SCHOOL,91,383,423,366
01M450,EAST SIDE COMMUNITY SCHOOL,70,377,402,370
01M458,FORSYTH SATELLITE ACADEMY,7,414,401,359
01M509,MARTA VALLE HIGH SCHOOL,44,390,433,384
01M515,LOWER EAST SIDE PREPARATORY HIGH SCHOOL,112,332,557,316
01M539,"NEW EXPLORATIONS INTO SCIENCE, TECHNOLOGY AND MATH HIGH SCHOOL",159,522,574,525
01M650,CASCADES HIGH SCHOOL,18,417,418,411
01M696,BARD HIGH SCHOOL EARLY COLLEGE,130,624,604,628
02M047,47 THE AMERICAN SIGN LANGUAGE AND ENGLISH SECONDARY SCHOOL,16,395,400,387

I've only used math, writing and reading scores. My goal for the code above is to predict the writing score if I give math and reading scores.

I've never seen Tensorflow's gradient descent model diverges with this such simple data. What'd be wrong?

Upvotes: 0

Views: 516

Answers (1)

rmeertens
rmeertens

Reputation: 4451

Here are a few options you could try:

  • Normalise you input and output data
  • Set smaller initial values for your weights
  • Use a lower learning rate
  • Divide your loss by the amount of samples you have (not putting your data in a placeholder is already uncommon).

Let me know what (if any) of these options helped and good luck!

Upvotes: 1

Related Questions