CODEWITHSUNDEEP
pythontensorflowlinear-regression
goelakash
goelakash

Reputation: 2519

Getting very high values in linear regression

I am trying to make a simple MLP to predict values of a pixel of an image - original blog . Here's my earlier attempt using Keras in python - link

I've tried to do the same in tensorflow, but I am getting very large output values (~10^12) when they should be less than 1.

Here's my code:

import numpy as np
import cv2
from random import shuffle
import tensorflow as tf

'''
Image preprocessing
'''
image_file = cv2.imread("Mona Lisa.jpg")

h = image_file.shape[0]
w = image_file.shape[1]

preX = []
preY = []

for i in xrange(h):
    for j in xrange(w):
        preX.append([i,j])
        preY.append(image_file[i,j,:].astype('float32')/255.0)

print preX[:5], preY[:5]
zipped = [i for i in zip(preX,preY)]
shuffle(zipped)

X_train = np.array([i for (i,j) in zipped]).astype('float32')
Y_train = np.array([j for (i,j) in zipped]).astype('float32')

print X_train[:10], Y_train[:10]

'''
Tensorflow code
'''

def weight_variable(shape):
  initial = tf.truncated_normal(shape, stddev=0.1)
  return tf.Variable(initial)

def bias_variable(shape):
  initial = tf.constant(0.1, shape=shape)
  return tf.Variable(initial)

x = tf.placeholder(tf.float32, shape=[None,2])
y = tf.placeholder(tf.float32, shape=[None,3])



'''
Layers
'''

w1 = weight_variable([2,300])
b1 = bias_variable([300])
L1 = tf.nn.relu(tf.matmul(X_train,w1)+b1)

w2 = weight_variable([300,3])
b2 = bias_variable([3])
y_model = tf.matmul(L1,w2)+b2


'''
Training
'''

# criterion
MSE = tf.reduce_mean(tf.square(tf.sub(y,y_model)))

# trainer
train_op = tf.train.GradientDescentOptimizer(learning_rate = 0.01).minimize(MSE)

nb_epochs = 10

init = tf.initialize_all_variables()
sess = tf.Session()

sess.run(init)
cost = 0

for i in range(nb_epochs):
    sess.run(train_op, feed_dict ={x: X_train, y: Y_train})
    cost += sess.run(MSE, feed_dict ={x: X_train, y: Y_train})

cost /= nb_epochs
print cost


'''
Prediction
'''

pred = sess.run(y_model,feed_dict = {x:X_train})*255.0
print pred[:10]

output_image = []
index = 0

h = image_file.shape[0]
w = image_file.shape[1]

for i in xrange(h):
    row = []

    for j in xrange(w):
        row.append(pred[index])
        index += 1

    row = np.array(row)
    output_image.append(row)

output_image = np.array(output_image)
output_image = output_image.astype('uint8')
cv2.imwrite('out_mona_300x3_tf.png',output_image)

Upvotes: 1

Views: 201

Answers (2)

Shagas
Shagas

Reputation: 605

First of all, I think that instead of running the train_op and then the MSE you can run both ops in a list and reduce your computational cost significantly. 

for i in range(nb_epochs):
cost += sess.run([MSE, train_op], feed_dict ={x: X_train, y: Y_train})

Secondly, I suggest always writing out your cost function so you can see what is going on during the training phase. Either manually print it out or use tensorboard to log your cost and plot it (you can find examples on the official tf page). You can also monitor your weights to see that they aren't blowing up.

A few things you can try: Reduce learning rate, add regularization to weights. Check that your training set (pixels) really consist of the values that you expect them to.

Upvotes: 1

Itamar Katz
Itamar Katz

Reputation: 9645

You give the input layer weights and the output layer weights the same names w and b, so it seems something goes wrong in the gradient-descent procedure. Actually I'm surprised tensorflow doesn't issue an error or at leas a warning (or am I missing something?)

Upvotes: 0

Related Questions