What's the equivalent of this Keras code in TensorFlow?

Question

The code is as below and runs perfectly:

import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Dropout

xData = np.array([[5, 3, 7], [1, 2, 6], [8, 7, 6]], dtype=np.float32)
yTrainData = np.array([[1], [0], [1]], dtype=np.float32)

model = Sequential()

model.add(Dense(64, input_dim=3, activation='relu'))

model.add(Dropout(0.2))

model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

model.fit(xData, yTrainData, epochs=10, batch_size=128, verbose=2)

xTestData = np.array([[2, 8, 1], [3, 1, 9]], dtype=np.float32)

resultAry = model.predict(xTestData)
print("Cal result: %s" % resultAry)

I can't work out the code in TensowFlow, something I've written is like this:

import tensorflow as tf
import numpy as np

xData = np.array([[5, 3, 7], [1, 2, 6], [8, 7, 6]], dtype=np.float32)
yTrainData = np.array([[1], [0], [1]], dtype=np.float32)

x = tf.placeholder(tf.float32)
yTrain = tf.placeholder(tf.float32)

w = tf.Variable(tf.ones([64]), dtype=tf.float32)
b = tf.Variable(tf.zeros([1]), dtype=tf.float32)

y = tf.nn.relu(w * x + b)

w1 = tf.Variable(tf.ones([3]), dtype=tf.float32)
b1 = tf.Variable(0, dtype=tf.float32)

y1 = tf.reduce_mean(tf.nn.sigmoid(w1 * y + b1))

loss = tf.abs(y1 - tf.reduce_mean(yTrain))

optimizer = tf.train.AdadeltaOptimizer(0.1)

train = optimizer.minimize(loss)

init = tf.global_variables_initializer()
sess = tf.Session()

sess.run(init)

for i in range(10):

    for j in range(3):
        result = sess.run([loss, y1, yTrain, x, w, b, train], feed_dict={x: xData[j], yTrain: yTrainData[j]})

        if i % 10 == 0:
            print("i: %d, j: %d, loss: %10.10f, y1: %f, yTrain: %s, x: %s" % (i, j, float(result[0]), float(result[1]), yTrainData[j], xData[j]))

result = sess.run([y1, loss], feed_dict={x: [1, 6, 0], yTrain: 0})
print(result)

But I will got the following error while running,

Traceback (most recent call last):
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1327, in _do_call
    return fn(*args)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1306, in _run_fn
    status, run_metadata)
  File "C:\Python36\lib\contextlib.py", line 88, in __exit__
    next(self.gen)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64] vs. [3]
         [[Node: mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](Variable/read, _arg_Placeholder_0_0)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "testidc.py", line 36, in <module>
    result = sess.run([loss, y1, yTrain, x, w, b, train], feed_dict={x: xData[j], yTrain: yTrainData[j]})
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
    run_metadata_ptr)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1321, in _do_run
    options, run_metadata)
  File "C:\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64] vs. [3]
         [[Node: mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](Variable/read, _arg_Placeholder_0_0)]]

Caused by op 'mul', defined at:
  File "testidc.py", line 15, in <module>
    y = tf.nn.relu(w * x + b)
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\variables.py", line 705, in _run_op
    return getattr(ops.Tensor, operator)(a._AsTensor(), *args)
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\math_ops.py", line 865, in binary_op_wrapper
    return func(x, y, name=name)
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1088, in _mul_dispatch
    return gen_math_ops._mul(x, y, name=name)
  File "C:\Python36\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1449, in _mul
    result = _op_def_lib.apply_op("Mul", x=x, y=y, name=name)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Incompatible shapes: [64] vs. [3]
         [[Node: mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](Variable/read, _arg_Placeholder_0_0)]]

The main reason is the shape of W, must be the same as x in TensowFlow, but in Keras, the hidden Dense layer could have more nodes than the input(such as 64 in the example).

I need help for the equivalent TensorFlow code instead of the Keras one. Thanks.

Answer 1

This is a couple of months old but it's worth noting that there is absolutely no reason to not use keras with tensorflow. It's even part of the tensorflow library now!

So if you want full control of your tensors but still want to use keras' layers, you can easily achieve that by using keras as-is:

x = tf.placeholder(tf.float32, [None, 1024])
y = keras.layers.Dense(512, activation='relu')(x)

For more on that, keras' creator made a pretty cool post about it.

Answer 2

This is an example that uses the tf.estimator.Estimator framework:

import tensorflow as tf
import numpy as np

# The model
def model(features):
  dense = tf.layers.dense(inputs=features['x'], units=64, activation=tf.nn.relu)
  dropout = tf.layers.dropout(dense, 0.2)
  logits = tf.layers.dense(inputs=dropout, units=1, activation=tf.nn.sigmoid)
  return logits

# Stuff needed to use the tf.estimator.Estimator framework
def model_fn(features, labels, mode):
  logits = model(features)

  predictions = {
    'classes': tf.argmax(input=logits, axis=1),
    'probabilities': tf.nn.softmax(logits)
  }

  if mode == tf.estimator.ModeKeys.PREDICT:
    return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)

  loss = tf.losses.softmax_cross_entropy(onehot_labels=labels, logits=logits)

  # Configure the training op
  if mode == tf.estimator.ModeKeys.TRAIN:
    optimizer = tf.train.RMSPropOptimizer(learning_rate=1e-4)
    train_op = optimizer.minimize(loss, tf.train.get_or_create_global_step())
  else:
    train_op = None

  accuracy = tf.metrics.accuracy(
      tf.argmax(labels, axis=1), predictions['classes'])

  metrics = {'accuracy': accuracy}

  # Create a tensor named train_accuracy for logging purposes
  tf.identity(accuracy[1], name='train_accuracy')
  tf.summary.scalar('train_accuracy', accuracy[1])

  return tf.estimator.EstimatorSpec(
      mode=mode,
      predictions=predictions,
      loss=loss,
      train_op=train_op,
      eval_metric_ops=metrics)

# Setting up input for the model
def input_fn(mode, batch_size):
  # function that processes your input and returns two tensors "samples" and "labels"
  # that the estimator will use to fetch input batches.
  # See https://www.tensorflow.org/get_started/input_fn for how to write this function.
  return samples, labels

# Using the model
def main():
  # Create the Estimator
  classifier = tf.estimator.Estimator(
    model_fn=model_fn, model_dir='some_dir')

  # Train the model

  # NOTE: I use this to make it compatible with your example, but you should 
  #       defnitely set up your own input_fn above
  train_input_fn = tf.estimator.inputs.numpy_input_fn(
    x={"x": np.array([[5, 3, 7], [1, 2, 6], [8, 7, 6]], dtype=np.float32)},
    y=np.array([[1], [0], [1]]),
    num_epochs=10,
    batch_size=128,
    shuffle=False)

  classifier.train(
      input_fn=train_input_fn,
      steps=20000, # change as needed
      )

  # Predict on new data
  predict_input_fn = tf.estimator.inputs.numpy_input_fn(
    x={"x": np.array([[5, 3, 7], [1, 2, 6], [8, 7, 6]], dtype=np.float32)},
    num_epochs=1,
    batch_size=1,
    shuffle=False)

  predictions_iterator = classifier.predict(
      input_fn=predict_input_fn)
  print('Predictions results:')
  for pred in predictions_iterator:
    print(pred)

There is quite bit going on here, so I'll try to explain the blocks one by one.

The model

The model is defined as a composition of tf.layers in a separate model function. This is done to keep the actual model_fn (which is required by the Estimator framework) independent of the model architecture. The function takes a features parameter, which is the output of a call to input_fn (see below). In this example, since we're using tf.estimator.inputs.numpy_input_fn, features is a dictionary with item x:input_tensor. We use the input tensor as input for our model graph.

model_fn

This function is required by the framework and is used to generate a specification for your Estimator that is dependent on the mode the estimato is being used for. Typically, an estimator used for prediction will have less operations than when it's used for training (you don't have the loss, optimizer, etc). This function takes care of adding all that is necessary to your model graph for the three possible modes of operation (prediction, evaluation, training).

Breaking it down to logical pieces, we have:

• Prediction: we only need the model graph, the predictions and the corresponding predicted labels (we could skip the labels, but having it here is handy).
• Evaluation: we need everything for prediction plus: a loss function, some metric to evaluate on and optionally some summaries to visualize the metrics in Tensorboard.
• Training: we need everything for evaluation plus: a training operation from an optimizer (in your sample, RMSProp)

input_fn

This is where we provide the input to our estimator. Have a look at Building Input Functions with tf.estimator for a guide on how your custom input_fn should look like. For the example, we'll use the numpy_input_fn function from the framework. Note that usually one input_fn handles all operation modes according to a mode parameter. Since we're using numpy_input_fn, we need two different instances of it for training and prediction to provide the data as needed.

main

Here we actually train and use the estimator. Firstly, we get an Estimator instance with the model_fn we specified, then we call train() and wait for the training to be over. Once that is done, calling predict() returns an iterable that you can use to get the prediction results for all the samples in the dataset you're predicting.