Convolutional Neural Network in Tensorflow with Own Data for Prediction

Question

I am a beginner in CNN and Tensorflow. I am trying to implement convolutional neural network in tensorflow with own data for prediction but I am having some problems. I converted Deep MNIST for Experts tutorials to this. Deep MNIST for Experts is classification, but I am trying to do regression. Another problem is, this code give me accuracy=1 for each step. What is the cause of the error? How can I convert this code for regression?

Data set:

Year_Month_Day,Hour_Minute,Temperature,Relative_humidity,Pressure,Total_Precipitation,Snowfall_amount,Total_cloud_cover,High_cloud_cover,Medium_cloud_cover,Low_cloud_cover,Shortwave_Radiation,Wind_speed_10m,Wind_direction_10m,Wind_speed_80m,Wind_direction_80m,Wind_speed_900m,Wind_direction_900m,Wind_Gust_10m,Difference
2016-10-24,23.00,15.47,76.00,1015.40,0.00,0.00,100.00,26.00,100.00,100.00,0.00,6.88,186.01,12.26,220.24,27.60,262.50,14.04,2.1 
2016-10-24,22.00,16.14,73.00,1014.70,0.00,0.00,10.20,34.00,0.00,2.00,0.00,6.49,176.82,11.97,201.16,24.27,249.15,7.92,0.669999 
..... 
..... 
.....
2016-10-24,18.00,20.93,56.00,1012.20,0.00,0.00,100.00,48.00,15.00,100.00,91.67,6.49,146.31,12.10,149.62,17.65,163.41,8.64,1.65 
2016-10-24,17.00,21.69,50.00,1012.10,0.00,0.00,100.00,42.00,10.00,100.00,243.86,9.50,142.70,12.77,139.57,19.08,144.21,32.40,0.76 

Code:

import tensorflow as tf
import pandas as pandas
from sklearn import cross_validation
from sklearn import preprocessing
from sklearn import metrics

sess = tf.InteractiveSession()

data = pandas.read_csv("tuna.csv")
print(data[-2:])
#X=data.copy(deep=True)

X=data[['Relative_humidity','Pressure','Total_Precipitation','Snowfall_amount','Total_cloud_cover','High_cloud_cover','Medium_cloud_cover','Low_cloud_cover','Shortwave_Radiation','Wind_speed_10m','Wind_direction_10m','Wind_speed_80m','Wind_direction_80m','Wind_speed_900m','Wind_direction_900m','Wind_Gust_10m']].fillna(0)
Y=data[['Temperature']]

number_of_samples=X.shape[0]
elements_of_one_sample=X.shape[1]


print("number of samples", number_of_samples)
print("elements_of_one_sample", elements_of_one_sample)
train_x, test_x, train_y, test_y = cross_validation.train_test_split(X, Y, test_size=0.1, random_state=42)

print("train_x.shape=", train_x.shape)
print("train_y.shape=", train_y.shape)
print("test_x.shape=", test_x.shape)
print("test_y.shape=", test_y.shape)

epoch = 0 # counter for number of rounds training network
last_cost = 0 # keep track of last cost to measure difference
max_epochs = 2000 # total number of training sessions
tolerance = 1e-6 # we stop when diff in costs less than that
batch_size = 50 # we batch the data in groups of this size
num_samples = train_y.shape[0] # number of samples in training set
num_batches = int( num_samples / batch_size ) # compute number of batches, given
print("############################## num_samples", num_samples)       
print("############################## num_batches", num_batches)       

x = tf.placeholder(tf.float32, shape=[None, 16])
y_ = tf.placeholder(tf.float32, shape=[None, 1])

# xW + b
W = tf.Variable(tf.zeros([16,1]))
b = tf.Variable(tf.zeros([1]))

sess.run(tf.initialize_all_variables())

# y = softmax(xW + b)
y = tf.nn.softmax(tf.matmul(x,W) + b)

# lossはcross entropy
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))

train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)


for n in range( num_batches ):
  batch_x = train_x[ n*batch_size : (n+1)*batch_size ]
  batch_y = train_y[ n*batch_size : (n+1)*batch_size ]
  train_step.run( feed_dict={x: batch_x, y_: batch_y} )


correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print(accuracy.eval(feed_dict={x: test_x, y_: test_y}))


# To create this model, we're going to need to create a lot of weights and biases. 
# One should generally initialize weights with a small amount of noise for symmetry 
# breaking, and to prevent 0 gradients
def weight_variable(shape):
  initial = tf.truncated_normal(shape, stddev=0.1)
  return tf.Variable(initial)

# Since we're using ReLU neurons, it is also good practice to initialize them 
# with a slightly positive initial bias to avoid "dead neurons." Instead of doing 
# this repeatedly while we build the model, let's create two handy functions 
# to do it for us.
def bias_variable(shape):
  initial = tf.constant(0.1, shape=shape)
  return tf.Variable(initial)


# https://www.tensorflow.org/versions/master/api_docs/python/nn.html#conv2d
def conv2d(x, W):
  return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')


# https://www.tensorflow.org/versions/master/api_docs/python/nn.html#max_pool
def max_pool_2x2(x):
  return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],strides=[1, 2, 2, 1], padding='SAME')


W_conv1 = weight_variable([2, 2, 1, 32])
b_conv1 = bias_variable([32])


x_image = tf.reshape(x, [-1,4,4,1])

h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)

W_conv2 = weight_variable([2, 2, 32, 64])
b_conv2 = bias_variable([64])

h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)


W_fc1 = weight_variable([1 * 1 * 64, 1024])
b_fc1 = bias_variable([1024])

h_pool2_flat = tf.reshape(h_pool2, [-1, 1*1*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)


W_fc2 = weight_variable([1024, 1])
b_fc2 = bias_variable([1])

y_conv=tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)

# loss
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)

# accuracy
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# train
sess.run(tf.initialize_all_variables())
for i in range(20000):
  if i%100 == 0:
    batch_x = train_x[ n*batch_size : (n+1)*batch_size ]
    batch_y = train_y[ n*batch_size : (n+1)*batch_size ]
    train_accuracy = accuracy.eval(feed_dict={x:batch_x, y_: batch_y, keep_prob: 1.0})
    print("step %d, training accuracy %g"%(i, train_accuracy))
  train_step.run(feed_dict={x: batch_x, y_: batch_y, keep_prob: 0.5})

# result
print("test accuracy %g"%accuracy.eval(feed_dict={
    x: test_x, y_: test_y, keep_prob: 1.0}))

Output:

number of samples 1250
elements_of_one_sample 16
train_x.shape= (1125, 16)
train_y.shape= (1125, 1)
test_x.shape= (125, 16)
test_y.shape= (125, 1)
############################## num_samples 1125
############################## num_batches 22
1.0
step 0, training accuracy 1
step 100, training accuracy 1
step 200, training accuracy 1
step 300, training accuracy 1
step 400, training accuracy 1
....
....
....
step 19500, training accuracy 1
step 19600, training accuracy 1
step 19700, training accuracy 1
step 19800, training accuracy 1
step 19900, training accuracy 1
test accuracy 1

I am quite new to neural nets and machine learning so pardon me for any mistakes, thanks in advance.

Answer 1

You've got a loss function of cross entropy, which is a loss function specifically designed for classification. If you want to do regression, you need to start with a loss function that penalizes prediction error (L2 error is a great place to start).

For prediction, the rightmost layer of the network needs to have linear units (no activation function). The number of neurons in the rightmost layer should correspond to the number of values you're predicting (if it's a simple regression problem where you're predicting a single value of y given a vector of inputs x, then you just need a single neuron in the right-most layer). Right now, you've got a softmax layer on the back end of the network, which is also specifically used for classification tasks.

Basically - you need to swap your softmax for a linear neuron and change your loss function to something like L2 error (aka mean-squared error).