CODEWITHSUNDEEP
pythontensorflowconvolution
larts
larts

Reputation: 67

How to train convolution filters in tensorflow

I'm trying to train a neural network that will be activated by horisontal lines (and in future, by corners).

I draw an image that contains both vertical and horisontal lines (see link below). Then in photoshop I draw marks on top of horisontal lines, the places where I want the net to activate.

Original and label images look like this: original lbl_hor

Now the task seems easy, but I can't get tensorflow to train such a net. Here is my code:

input = tf.convert_to_tensor(original, tf.float32)
label = tf.convert_to_tensor(lbl_hor, tf.float32)

W1 = tf.Variable(tf.random_normal([5,5,1,1]))
b1 = tf.Variable(0.1)
W2 = tf.Variable(tf.random_normal([5,5,1,1]))
b2 = tf.Variable(0.1)

l1 = tf.nn.conv2d(input, W1, [1,1,1,1], 'SAME')
l1 = l1 + b1
l1 = tf.nn.relu(l1)

l2 = tf.nn.conv2d(l1, W2, [1,1,1,1], 'SAME')
l2 = l2 + b2
l2 = tf.nn.relu(l2)

loss = tf.square(l2 - label)
loss_sum = tf.reduce_sum(loss)
train_op = tf.train.AdamOptimizer(0.01).minimize(loss)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

for i in range(0, 100):
    sess.run(train_op)

The network gives every time unpredictable output, sometimes just black. I tried to multiply the label by some constant so that the activations have more weight - does not have much effect...

Can you help me to figure out how to train such a network or find what's wrong with my code?

Upvotes: 0

Views: 116

Answers (1)

larts
larts

Reputation: 67

Finally figured out the answer. It's embarrassing, but in case someone needs it, posting it here.

First, I should make sure that my labels and the network output are normalized:

output_normalized = tf.nn.l2_normalize(l2)
label_normalized = tf.nn.l2_normalize(label)

Second, tensor tf.square(l2 - label) is of shape [1, 200, 200, 200] - that's clearly not right. So I modified it further:

output_normalized = tf.reshape(tf.nn.l2_normalize(l2), [200, 200])
label_normalized = tf.nn.l2_normalize(label)

loss = tf.square(output_normalized - label_normalized)

Here is new code, that works completely fine:

input = tf.convert_to_tensor(original, tf.float32)
label = tf.convert_to_tensor(lbl_hor, tf.float32)

W1 = tf.Variable(tf.ones([5,5,1,1]))
b1 = tf.Variable(0.1)
W2 = tf.Variable(tf.ones([5,5,1,1]))
b2 = tf.Variable(0.1)

l1 = tf.nn.conv2d(input, W1, [1,1,1,1], 'SAME')
l1 = l1 + b1
l1 = tf.nn.relu(l1)

l2 = tf.nn.conv2d(l1, W2, [1,1,1,1], 'SAME')
l2 = l2 + b2
l2 = tf.nn.relu(l2)

output_normalized = tf.reshape(tf.nn.l2_normalize(l2), [200, 200])
label_normalized = tf.nn.l2_normalize(label)

loss = tf.square(output_normalized - label_normalized)
loss_sum = tf.reduce_sum(loss)
train_op = tf.train.AdamOptimizer(0.1).minimize(loss_sum)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

for i in range(0, 100):
    sess.run(train_op)

For some reason, tf.random_normal as initial values does not produce stable learning every time, so I have used tf.ones instead. Not sure why it happens, but that works...

Upvotes: 1

Related Questions