GP + Tensorflow training

Question

I am trying to train a GPR model and a tensorflow model together. The training part has no issue. But for prediction using the trained model I receive a type error in a tf.placeholder op.

pred, uncp=sess.run([my, yv], feed_dict={X:xtr})

The code is similar to the 2nd example from https://gpflow.readthedocs.io/en/master/notebooks/advanced_usage.html

import numpy as np
import tensorflow as tf

import gpflow
float_type = gpflow.settings.float_type
gpflow.reset_default_graph_and_session()

def cnn_fn(x, output_dim):    
    out= tf.layers.dense(inputs=x, units=output_dim, activation=tf.nn.relu)
    print(out)    
    return out

N = 150
xtr = np.random.rand(N,1)
ytr = np.sin(12*xtr) + 0.66*np.cos(25*xtr) + np.random.randn(N,1)*0.1 + 3
xtr = np.random.rand(N,28)
print(xtr.shape, ytr.shape)
nepoch=50
gp_dim=xtr.shape[1]
print(gp_dim)
minibatch_size = 16

X = tf.placeholder(tf.float32, [None, gp_dim]) 
Y = tf.placeholder(tf.float32, [None, 1])

with tf.variable_scope('cnn'):
    f_X = tf.cast(cnn_fn(X, gp_dim), dtype=float_type)

k = gpflow.kernels.Matern52(gp_dim)
gp_model = gpflow.models.GPR(f_X, tf.cast(Y, dtype=float_type), k)

loss = -gp_model.likelihood_tensor
m, v = gp_model._build_predict(f_X)
my, yv = gp_model.likelihood.predict_mean_and_var(m, v)

with tf.variable_scope('adam'):
    opt_step = tf.train.AdamOptimizer(0.001).minimize(loss)

tf_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='adam')
tf_vars += tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='cnn')

## initialize
sess = tf.Session()
sess.run(tf.variables_initializer(var_list=tf_vars))
gp_model.initialize(session=sess)

for i in range(nepoch):
    shind=np.array(range(len(xtr)))
    np.random.shuffle(shind)
    for j in range(int(len(xtr)/minibatch_size)):
        ind=shind[j*minibatch_size: (j+1)*minibatch_size]
        sess.run(opt_step, feed_dict={X:xtr[ind], Y:ytr[ind]})

Executing the code above runs fine. But adding the following line gives an error:

 pred, uncp=sess.run([my, yv], feed_dict={X:xtr})

with the following error:

<ipython-input-25-269715087df2> in <module>
----> 1 pred, uncp=sess.run([my, yv], feed_dict={X:xtr})

[...]

InvalidArgumentError: You must feed a value for placeholder tensor 'Placeholder_1' with dtype float and shape [?,1]
     [[node Placeholder_1 (defined at <ipython-input-24-39ccf45cd248>:2)  = Placeholder[dtype=DT_FLOAT, shape=[?,1], _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Answer 1

The reason your code fails is because you are not actually feeding in a value to one of the placeholders. This is easier to spot if you actually give them names:

X = tf.placeholder(tf.float32, [None, gp_dim], name='X') 
Y = tf.placeholder(tf.float32, [None, 1], name='Y')

Tensorflow requires the entire compute graph to be well-defined, and the GPR model you are using depends on both X and Y. If you run the following line, it works fine:

pred, uncp = sess.run([my, yv], feed_dict={X: xtr, Y: ytr})

Update: as user1018464 pointed out in the comment, you are using the GPR model, in which the predictions directly depend on the training data (e.g. see equations (2.22) and (2.23) on page 16 of http://www.gaussianprocess.org/gpml/chapters/RW2.pdf). Hence you will need to pass in both xtr and ytr to make predictions.

Other models such as SVGP represent the function through "inducing features", commonly "pseudo input/output" pairs that summarise the data, in which case you won't need to feed in the original input values at all (I got it wrong when I first answered).

You could set up the model as follows:

gp_model = gpflow.models.SVGP(f_X, tf.cast(Y, dtype=float_type), k, gpflow.likelihoods.Gaussian(), xtr.copy(), num_data=N)

Then pred, uncp=sess.run([my, yv], feed_dict={X:xtr}) works as expected.

If you want to predict at different points Xtest, you need to set up a separate placeholder, and reuse the cnn (note the reuse=True in the variable_scope with the same name), as in example 2 of the notebook:

Xtest = tf.placeholder(tf.float32, [None, Mnist.input_dim], name='Xtest')

with tf.variable_scope('cnn', reuse=True):
    f_Xtest = tf.cast(cnn_fn(Xtest, gp_dim), dtype=float_type)

Set up the model as before using f_X, but use f_Xtest in the call to _build_predict:

m, v = gp_model._build_predict(f_Xtest)
my, yv = gp_model.likelihood.predict_mean_and_var(m, v)

Now you need to pass in both X, Y, and Xtest into the session's run():

pred, uncp = sess.run([my, yv], feed_dict={X: xtr, Y: Ytr, Xtest: xtest})

where xtest is the numpy array with points at which you want to predict.

Answer 2

The GPflow manages TensorFlow sessions for you and you don't need to create your own TF session, when you use GPflow alone. In your case, tf.layers.dense makes new variables, which should be initialized and I would advise to use a session which were created by GPflow. Essentially, you need to replace these lines

## initialize
sess = tf.Session()
sess.run(tf.variables_initializer(var_list=tf_vars))
gp_model.initialize(session=sess)

with:

sess = gpflow.get_default_session()
sess.run(tf.variables_initializer(var_list=tf_vars)

or wrap your code with default session context:

with tf.Session() as session:
    ... build and run