← Notes

The three measurements that turn "it's slow" into a diagnosis

Time to first token, total generation time and output token count separate a chat that is slow from waiting from a chat that is slow from writing.

"It's slow" is a legitimate complaint and a useless diagnosis: it does not say what to fix. The model serving metrics documentation names the missing pieces.

The first is time to first token, defined as "the time elapsed between submitting a prompt and receiving the first token of the model's response". It is the silent wait: the provider queue, prompt processing and, where there is retrieval, the search over the base all fit inside it. The second is total generation time, which runs afterwards, token by token. The sum of the two is the latency the visitor feels.

The third measurement is not about time and is the one that most explains the second: how many tokens the answer has. A reply that comes out in three paragraphs takes some multiple of the time of one that comes out in three lines, on the same infrastructure and the same model.

With all three, each suspicion becomes a test. A long wait before the first word appears points to the queue, the prompt or the search. A fast first word and a reply that drags points to output length or to the generation speed of the chosen model. Those are different fixes, and none of them is guessable from the feeling of slowness.