How to load trained TensorFlow model to predict with different batch size?

Question

This shall be a quite simple question but indeed caught me.

My purpose is to load a trained TF model, then create a session to run the predict/inference, and evaluate an appropriate batch_size for my device environment.

I prepared two types of model: frozen model and saved model.

For the frozen model, I successfully loaded and get the GraphDef by .ParseFromString(), and optimized the GraphDef by TensorRT, but the batch_size of input node is fixed to 1 (1*299*299*3). Seems like the batch_size can't be configured when exporting then freeze model, and can't be changed afterwards because it's append-only.

For the saved model, the dimension of input node is ? (?*299*299*3). Seems it shall be able to accept any size of batch. But after I loaded the saved model, then input a 299*299*3 image, and get below ERROR:

ValueError: Cannot feed value of shape (299, 299, 3) for Tensor u'input:0', which has shape '(?, 299, 299, 3)'

I tried to expand a axis=0 to the input image, but still get error:

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'import/input' with dtype float and shape [?,299,299,3]

 [[Node: import/input = Placeholder[dtype=DT_FLOAT, shape=[?,299,299,3], _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Overall, this is supposed to be a quite simple question, but I'm not sure how to solve it.

Any idea will be welcome.

Thanks,

---

The script with frozen model, the placeholder is fixed to (1*299*299*3), so it only accepts "args.batch_size" to be 1.

    g = ops.Graph()
with g.as_default():
    inp, out = importer.import_graph_def(graph_def=f32_graph, return_elements=['Placeholder','InceptionV3/Logits/SpatialSqueeze'])
    inp = inp.outputs[0]
    out = out.outputs[0]
    print tf.shape(inp)
    if args.image:
        import numpy as np
        func = TestKit.preprocess_func['tensorflow'][args.network]
        img = func(args.image)
        img = np.expand_dims(img, axis = 0)
    batch_input = img
    for i in range(int(args.batch_size)-1):
        batch_input = np.concatenate((batch_input, img), axis=0)
    print len(batch_input)
    gpu_options = cpb2.GPUOptions(per_process_gpu_memory_fraction=0.625)
    with csess.Session(config=cpb2.ConfigProto(gpu_options=gpu_options), graph=g) as sess:
        t0 = time.time()
        for _ in range(2000):
            val = sess.run(out, {inp: batch_input})
            predict = np.squeeze(val)
            top_indices = predict.argsort()[-5:][::-1]
            result = [(i, predict[i]) for i in top_indices]
        t1 = time.time()
        print result
        print 'Duration:', str(t1-t0)

---

The script of saved model, "input:0" node is (?*299*299*3) but can't accept any shape of input image as mentioned above.

    with tf.Graph().as_default():                                                                                                                       
    with tf.Session(graph=tf.Graph()) as sess:
        tf.saved_model.loader.load(sess, ['serve'], './')
        for op in sess.graph.get_operations():
            print str(op.name)

        if args.image:
            import numpy as np
            func = TestKit.preprocess_func['tensorflow'][args.network]
            img = func(args.image)
            img = np.expand_dims(img, axis = 0)                                                                           
        inp, out = importer.import_graph_def(graph_def=sess.graph.as_graph_def(), return_elements=['input','InceptionV3/Logits/SpatialSqueeze'])
        t0 = time.time()                                                                                                               
        for _ in range(1000):
            val = sess.run(out, {'input:0':img})
            predict = np.squeeze(val)
            top_indices = predict.argsort()[-5:][::-1]
            result = [(i, predict[i]) for i in top_indices]
        t1 = time.time()
        print result
        print 'Duration:', str(t1-t0)

Answer 1

I finally solved the problem. The point is freeze model will not change the batch size because the graph is append-only, so once the original graph is generated with input placeholder (1 * 299 * 299 * 3), it will never be changed.

Finally, I tried to regenerate the original binary graph (binary .pbtxt) with input placeholder (None * 299 * 299 * 3). Then convert this binary graph to freeze model, the freeze model then can have the input placeholder with dimension (? * 299 * 299 * 3). Batch input is supported now.

Thanks,