How to handle dynamic shapes in trtexec commands in tensorrt

Question

How do I write the trtexec command to compile an engine to receive input from dynamic shapes?

When the onnx model was compiled into the tensorrt engine using the trtexec command, it automatically became an overriding shape in the 1x1 shape. If the input shape is not fixed, a shape such as -1 is usually specified. Is there no setting for dynamic shape in tensorrt?

trtexec --onnx=model.onnx
--saveEngine=model.plan

After the trtexec command, it can be confirmed that 1x1 becomes an overriding shape as shown below.

[03/24/2023-10:34:14] [I] Finish parsing network model [03/24/2023-10:34:14] [W] Dynamic dimensions required for input: input_ids, but no shapes were provided. Automatically overriding shape to: 1x1
[03/24/2023-10:34:14] [W] Dynamic dimensions required for input: attention_mask, but no shapes were provided. Automatically overriding shape to: 1x1

I want to set the shape in a dynamic shape as shown below

trtexec --onnx=model.onnx --shapes=input_ids:1x-1,attention_mask:1x-1 --saveEngine=model.plan

ex) 1x-1 : 1=Batch size, -1=undefined number of tokens may be entered.

Since the input is fixed at 1x1, i cannot receive the result of the tensorrt engine unless it is 1x1 when I give the input of the model.

Currently, the shape is automatically determined with a 1x1 shape. Since the number of input tokens in the onnx model is not fixed, I would like to compile it with a tensor RT engine that can receive an dynamic shape (ex: 1x-1).

Answer 1

You need to specify minimum, maximum, and optimum shapes:

trtexec --onnx=model.onnx --minShapes=input_ids:1x1,attention_mask:1x1 \
                          --optShapes=input_ids:1x512,attention_mask:1x512 \
                          --maxShapes=input_ids:1x1024,attention_mask:1x1024 \
--saveEngine=model.plan