Reputation: 27
I deploy VLLM with docker. I would like to get the TTFT (time to first token) and TPOT (time per output token) for each prompt request, instead of just their statistics.
Upvotes: 0
Views: 28