← Notes

The baseline is what makes "scale" a sentence with meaning

Without a prior measurement, scaling has no success criterion: there is no number to say whether the change worked, and the feeling of improvement is the worst judge available.

Scaling is always a comparison — faster than before, cheaper per answer than before, holding more than before. Without the "before", the word has no referent.

The "before" of a chat is short and well known. Time to first token, defined in the serving metrics documentation as "the time elapsed between submitting a prompt and receiving the first token of the model's response", is the silent wait. Generation time is what runs afterwards. The output token count explains much of the second. Three values, collectable with simple instrumentation, over a handful of real conversations.

The reason to collect them first is not rigour for its own sake. It is that the public guidance for building with models is to "find the simplest solution possible, and only increasing complexity when needed", and needed is an empirical claim. Without a baseline there is no evidence of need — there is only the intuition of whoever is proposing, which is exactly what the rule asks not to be the criterion.

There is also a practical effect of sequence. A baseline taken after the change is no longer a comparison; it becomes the new state, and the gain is left undemonstrable. Measuring is the one step of the project that cannot be done out of order.