CODEWITHSUNDEEP
tensorflowtensorflow-serving
tech_a_break
tech_a_break

Reputation: 163

Tensorflow Serving Batching parameters

How to do performance tuning of batching using max_batch_size, batch_timeout_micros, num_batch_threads and other parameters? Tried using these parameters with the Query client, it doesn't work.

In the below example I have 100 images and I want to batch in size of 10. The query runs for all images instead of 10.

bazel-bin/tensorflow_serving/example/demo_batch --server=localhost:9000 --max_batch_size=10

Also, for batch scheduling how to make it run every 10 secs after the first batch is done? Thanks.

Upvotes: 4

Views: 3034

Answers (1)

Tuoyu
Tuoyu

Reputation: 81

I have met the same problem like you.

And I checked the source code of tf-serving, these parameters are in an protobuf file which defined in:

serving/tensorflow_serving/servables/tensorflow/session_bundle_config.proto

And I found the example conf file in:

serving/tensorflow_serving/servables/tensorflow/testdata/batching_config.txt

And I believe you could follow the batching_config.txt format, the parameters config should be work.

Hope it helps.

max_batch_size { value: 1024 }
batch_timeout_micros { value: 0 }
max_enqueued_batches { value: 1000000 }
num_batch_threads { value: 8 }
allowed_batch_sizes : 1
allowed_batch_sizes : 2
allowed_batch_sizes : 8
allowed_batch_sizes : 32
allowed_batch_sizes : 128
allowed_batch_sizes : 256
allowed_batch_sizes : 512
allowed_batch_sizes : 1024

Upvotes: 4

Related Questions