From POST request to streamed response โ the serving infrastructure around the inference engine.
The server reads the body, parses JSON, validates the schema, and normalises options into an internal request object. Unknown models, malformed messages, or impossible sampling settings get rejected here before the runtime spends tokenizer, CPU, or GPU time.
The model does not consume role/content arrays directly. The provider renders them into the prompt syntax that checkpoint expects, inserting role markers, separators, BOS tokens, and an assistant prefix so generation begins in the correct conversational state.
The formatted prompt becomes token IDs. This is the same tokenizer logic from Chapter 2, but now applied to the template-expanded string. From this point onward the provider mostly tracks integer IDs, token counts, buffer lengths, and byte offsets.
The scheduler decides whether this request can enter the live batch now. It checks queue depth, token budget, and available KV-cache pages. If capacity exists, it allocates slots; if not, it queues the request or applies backpressure such as HTTP 429.
Prefill runs the entire prompt through the model in one wide pass. Every layer writes keys and values for all prompt positions into the KV cache, and the hidden state at the last prompt token produces the first logits for generation.
After prefill, generation becomes a tight one-token loop. The server feeds back the newest token, runs one-position decode, scores the vocabulary, samples with temperature and top-p, appends the winner, and checks whether EOS, max_tokens, or a stop sequence has been reached.
If stream=true, the provider serialises each sampled token into the OpenAI-compatible SSE wire format and flushes it immediately. That is why chat UIs appear to type word by word: the server is sending incremental deltas, not waiting for the full sentence.
When generation finishes or the client disconnects, the provider tears the request down. KV pages go back to the allocator, metrics are updated, cancellations are reconciled, and the stream closes cleanly so the scheduler can admit more waiting work.
Protect the queue before GPU memory is the thing saying no.
Reject oversize or malformed requests before tokenization and admission.
Track cache pages explicitly; reclaim aggressively when requests finish or cancel.
Under load, prefer shorter queues, lower concurrency, or smaller models over collapse.
Separate liveness from readiness so a pod can be alive but not accepting work.
Use a draft model to reduce latency when verification cost still nets a win.
Smaller weights can raise throughput if the accuracy trade-off is acceptable.
Shard big models across GPUs when one device cannot hold weights or bandwidth demand.
Static batches are simpler; continuous batching usually keeps expensive devices busier.
Clients disappear. The provider must stop wasted decode work quickly and safely.
Most providers expose a small compatibility layer above the inference engine. Matching the request schema and streaming format matters because it lets existing SDKs, tools, and agents talk to the server without knowing whether the backend is vLLM, TGI, llama.cpp, or something custom.