In Tensorflow, What kind of neural network should I use?

Question

I am doing Tensorflow tutorial, getting what TF is. But I am confused about what neural network should I use in my work. I am looking at Single Layer Neural Network, CNN, RNN, and LSTM RNN.

There is a sensor which measures something and represents the result in 2 boolean ways. Here, they are Blue and Red, like this:

the sensor gives result values every 5minutes. If we pile up the values for each color, we can see some patterns:

number inside each circle represents the sequence of result values given from sensor. (107 was given right after 106) when you see from 122 to 138, you can see decalcomanie-like pattern.

I want to predict the next boolean value before the sensor result. I may do supervised learning using past results. But I'm not sure which neural network or method is suitable. Thinking that this work needs pattern using past results (have to see context), and memorize past results, maybe LSTM RNN (long-short term memory recurrent neural network) would be suitable one. Could you tell me what is the right one?

Answer 1

Yes, an RNN (recurrent neural network) fits the task of accumulating state along along a sequence in order to predict its next element. LSTM (long short-term memory) is a particular design for the recurrent pieces of the network that has turned out to be very successful in avoiding numerical challenges from long-lasting recurrences; see colah's much-cited blogpost for more. (Alternatives to the LSTM cell design exist but I would only fine tune that much later, possibly never.)

The TensorFlow RNN codelab explains LSTM RNNs for the case of language models, which predict the (n+1)-st word of a sentence from the preceding n words, for each n (like for each timestep in your series of measurements). Your case is simpler than language models in that you only have two words (red and blue), so if you read anything about embeddings of words, ignore it.

You also mentioned other types of neural networks. These are not aimed at accumulating state along a sequence, such as your boolean sequence of red/blue inputs. However, your second image suggests that there might be pattern in the sequence of counts of successive red/blue values. You could try using the past k counts as input to a plain feed-forward (i.e., non-recursive) neural network that predicts the probability of the next measurement having the same color as the current one. - Maybe that works with a single layer, or maybe two or even three work better; experimentation will tell. This is a less fancy approach than an RNN, but if it works good enough, it gives you a simpler solution with fewer technicalities to worry about.

CNNs (convolutional neural networks) would not be my first choice here. These aim to discover a set of fixed-scale features at various places in the input, for example, some texture or curved edge anywhere in an image. But you only want to predict one next item that extends your input sequence. A plain neural network (see above) may discover useful patterns on the k previous values, and training it with all earlier partial sequences will help it find those patterns. The CNN approach would help to discover them during prediction at long-gone parts of the input; I have no intuition why that would help.

Answer 2

So it sounds like you need to process a sequences of images. You could actually use both CNN and RNN together. I did this a month ago when I was training a network to swipe left or right on tinder using the sequence of profile pictures. What you would do is pass all of the images through a CNN and then into the RNN. Below is part of the code for my tinder bot. See how I distribute the convolutions over the sequence and then push it through the RNN. Finally I put a softmax classifier on the last time step to make the prediction, however in your case I think you will distribuite the prediction in time since you want the next item in the sequence.

self.input_tensor = tf.placeholder(tf.float32, (None, self.max_seq_len, self.img_height, self.img_width, 3), 'input_tensor')
self.expected_classes = tf.placeholder(tf.int64, (None,))
self.is_training = tf.placeholder_with_default(False, None, 'is_training')
self.learning_rate = tf.placeholder(tf.float32, None, 'learning_rate')
self.tensors = {}
activation = tf.nn.elu
rnn = tf.nn.rnn_cell.LSTMCell(256)
with tf.variable_scope('series') as scope:
    state = rnn.zero_state(tf.shape(self.input_tensor)[0], tf.float32)
    for t, img in enumerate(reversed(tf.unpack(self.input_tensor, axis = 1))):
        y = tf.map_fn(tf.image.per_image_whitening, img)
        features = 48
        for c_layer in range(3):
            with tf.variable_scope('pool_layer_%d' % c_layer):
                with tf.variable_scope('conv_1'):
                    filter = tf.get_variable('filter', (3, 3, y.get_shape()[-1].value, features))
                    b = tf.get_variable('b', (features,))
                    y = tf.nn.conv2d(y, filter, (1, 1, 1, 1), 'SAME') + b
                    y = activation(y)
                    self.tensors['img_%d_conv_%d' % (t, 2 * c_layer)] = y

                with tf.variable_scope('conv_2'):
                    filter = tf.get_variable('filter', (3, 3, y.get_shape()[-1].value, features))
                    b = tf.get_variable('b', (features,))
                    y = tf.nn.conv2d(y, filter, (1, 1, 1, 1), 'SAME') + b
                    y = activation(y)
                    self.tensors['img_%d_conv_%d' % (t, 2 * c_layer + 1)] = y

                y = tf.nn.max_pool(y, (1, 3, 3, 1), (1, 3, 3, 1), 'SAME')
                self.tensors['pool_%d' % c_layer] = y
                features *= 2
                print(y.get_shape())

        with tf.variable_scope('rnn'):
            y = tf.reshape(y, (-1, np.prod(y.get_shape().as_list()[1:])))
            y, state = rnn(y, state)
            self.tensors['rnn_%d' % t] = y

        scope.reuse_variables()

    with tf.variable_scope('output_classifier'):
        W = tf.get_variable('W', (y.get_shape()[-1].value, 2)) 
        b = tf.get_variable('b', (2,))
        y = tf.nn.dropout(y, tf.select(self.is_training, 0.5, 1.0))
        y = tf.matmul(y, W) + b
        self.tensors['classifier'] = y