Carter Wang
Carter Wang

Reputation: 27

How to get TTFT and TPOT in VLLM?

I deploy VLLM with docker. I would like to get the TTFT (time to first token) and TPOT (time per output token) for each prompt request, instead of just their statistics.

Upvotes: 0

Views: 28

Answers (0)

Related Questions