How to pass list/array of request objects to tensorflow serving in one server call?

Question

After loading the wide and deep model, i was able to make prediction for one request object using the map of features and then serializing it to string for predictions as shown below-

is there a way we can create a batch of requests objects and send them for prediction to tensorflow server?

Code for single prediction looks like this-

for (each feature in feature list) {
    Feature feature = null;
    feature = Feature.newBuilder().setBytesList(BytesList.newBuilder().addValue(ByteString.copyFromUtf8("dummy string"))).build();

        if (feature != null) {
            inputFeatureMap.put(name, feature);
        }

}

//Converting features(in inputFeatureMap) corresponding to one request into 'Features' Proto object

                Features features = Features.newBuilder().putAllFeature(inputFeatureMap).build();
                inputStr = Example.newBuilder().setFeatures(features).build().toByteString();
        }

        TensorProto proto = TensorProto.newBuilder()
                    .addStringVal(inputStr)
                    .setTensorShape(TensorShapeProto.newBuilder().addDim(TensorShapeProto.Dim.newBuilder().setSize(1).build()).build())
                    .setDtype(DataType.DT_STRING)
                    .build();

        PredictRequest req = PredictRequest.newBuilder()
                    .setModelSpec(ModelSpec.newBuilder()
                            .setName("your serving model name")
                            .setSignatureName("serving_default")
                            .setVersion(Int64Value.newBuilder().setValue(modelVer)))
                    .putAllInputs(ImmutableMap.of("inputs", proto))
                    .build();

        PredictResponse response = stub.predict(req);

        System.out.println(response.getOutputsMap());  

Is there a way we can send the list of Features Object for predictions, something similar to this-

List<Features> = {someway to create array/list of inputFeatureMap's which can be converted to serialized string.}

Answer 1

For anyone stumbling here, I found a simple workaround with Example proto to do batch request. I will borrow code from this question and modify it for the batch.

Features features =
    Features.newBuilder()
        .putFeature("Attribute1", feature("A12"))
        .putFeature("Attribute2", feature(12))
        .putFeature("Attribute3", feature("A32"))
        .putFeature("Attribute4", feature("A40"))
        .putFeature("Attribute5", feature(7472))
        .putFeature("Attribute6", feature("A65"))
        .putFeature("Attribute7", feature("A71"))
        .putFeature("Attribute8", feature(1))
        .putFeature("Attribute9", feature("A92"))
        .putFeature("Attribute10", feature("A101"))
        .putFeature("Attribute11", feature(2))
        .putFeature("Attribute12", feature("A121"))
        .putFeature("Attribute13", feature(24))
        .putFeature("Attribute14", feature("A143"))
        .putFeature("Attribute15", feature("A151"))
        .putFeature("Attribute16", feature(1))
        .putFeature("Attribute17", feature("A171"))
        .putFeature("Attribute18", feature(1))
        .putFeature("Attribute19", feature("A191"))
        .putFeature("Attribute20", feature("A201"))
        .build();
Example example = Example.newBuilder().setFeatures(features).build();

String pfad = System.getProperty("user.dir") + "\\1511523781";
try (SavedModelBundle model = SavedModelBundle.load(pfad, "serve")) {
  Session session = model.session();
  final String xName = "input_example_tensor";
  final String scoresName = "dnn/head/predictions/probabilities:0";

  try (Tensor<String> inputBatch = Tensors.create(new byte[][] {example.toByteArray(), example.toByteArray(), example.toByteArray(), example.toByteArray()});
      Tensor<Float> output =
          session
              .runner()
              .feed(xName, inputBatch)
              .fetch(scoresName)
              .run()
              .get(0)
              .expect(Float.class)) {
    System.out.println(Arrays.deepToString(output.copyTo(new float[4][2])));
  }
}

Essentially you can pass each example as an object in byte[4][] and you will receive the result in the same shape float[4][2]