Chapter 16

Build the Provider

From POST request to streamed response โ€” the serving infrastructure around the inference engine.

HTTP request to token stream
POST /v1/chat/completions
{"model":"llama-7b","messages":[{"role":"user","content":"The capital of France is"}],"stream":true,"temperature":0.7,"top_p":0.95}
state: one OpenAI-style request arrives and asks the provider to stream a reply.
โ†“
Receive Request [HTTP POST / OpenAI-compatible API]

The server reads the body, parses JSON, validates the schema, and normalises options into an internal request object. Unknown models, malformed messages, or impossible sampling settings get rejected here before the runtime spends tokenizer, CPU, or GPU time.

๐Ÿ”Ž provider detail
Extracted fields: model, messages[], stream, temperature, top_p, plus defaults such as max_tokens or stop. This is also where auth, rate-limit keys, and request IDs are attached.
state: raw JSON โ†’ req{model:"llama-7b",messages:1,stream:true,temp:0.7,top_p:0.95}
โ†“
Apply Chat Template [Jinja2 / model-specific formatting]

The model does not consume role/content arrays directly. The provider renders them into the prompt syntax that checkpoint expects, inserting role markers, separators, BOS tokens, and an assistant prefix so generation begins in the correct conversational state.

๐Ÿ”Ž provider detail
Example render:
<|user|>\nThe capital of France is\n<|assistant|>\n

Different models want different wrappers, which is why template selection is part of serving, not an afterthought.
state: messages[] โ†’ "<|user|>\nThe capital of France is\n<|assistant|>\n"
โ†“
Tokenize [BPE]

The formatted prompt becomes token IDs. This is the same tokenizer logic from Chapter 2, but now applied to the template-expanded string. From this point onward the provider mostly tracks integer IDs, token counts, buffer lengths, and byte offsets.

๐Ÿ”Ž provider detail
Illustrative only: exact IDs depend on tokenizer build and chat template revision. Serving stacks care about length, special tokens, and byte-safe detokenization on the way back out.
state: prompt string โ†’ [1, 32001, 1045, 3382, 310, 3444, 338, 32002]
โ†“
Admit Request [Scheduler / Queue / Continuous Batching]

The scheduler decides whether this request can enter the live batch now. It checks queue depth, token budget, and available KV-cache pages. If capacity exists, it allocates slots; if not, it queues the request or applies backpressure such as HTTP 429.

๐Ÿ”Ž provider detail
Continuous batching: active requests do not all start together. Prefill and decode work from different requests are interleaved so finished requests can leave and new ones can join without draining the whole batch first.
state: req#418 admitted; KV pages reserved; active batch now contains older decode jobs plus this new prefill job.
โ†“
Prefill [First forward pass]

Prefill runs the entire prompt through the model in one wide pass. Every layer writes keys and values for all prompt positions into the KV cache, and the hidden state at the last prompt token produces the first logits for generation.

๐Ÿ”Ž provider detail
Prefill is compute-heavy: all prompt tokens are processed together, which is why time-to-first-token rises with prompt length even when later decode steps only add one token at a time.
state: prompt_len=8; KV cache[layer][0..7] populated; first next-token logits ready.
โ†“
Decode loop โ€” repeat until stop
Decode Loop [One token at a time]

After prefill, generation becomes a tight one-token loop. The server feeds back the newest token, runs one-position decode, scores the vocabulary, samples with temperature and top-p, appends the winner, and checks whether EOS, max_tokens, or a stop sequence has been reached.

๐Ÿ”Ž provider detail
Hot-path sketch:
forward(last_token) โ†’ logits โ†’ sample โ†’ append โ†’ test stop.

In real providers this loop is interleaved across many requests sharing the same accelerator.
state: generated IDs accumulate one by one โ†’ [..., 3681] then [..., 3681, 29889]
detokenized so far โ†’ " Paris."
no stop yet? decode the next token and keep the request in the active batch
โ†“
Stream Tokens [SSE / chunked response]

If stream=true, the provider serialises each sampled token into the OpenAI-compatible SSE wire format and flushes it immediately. That is why chat UIs appear to type word by word: the server is sending incremental deltas, not waiting for the full sentence.

๐Ÿ”Ž provider detail
Example event:
data: {"choices":[{"delta":{"content":" Paris"}}]}

The network path may be SSE framing over HTTP/1.1 chunked transfer or equivalent streaming behaviour over HTTP/2.
state: client receives partial output immediately โ†’ " Paris", then later "."
โ†“
Cleanup [Release resources]

When generation finishes or the client disconnects, the provider tears the request down. KV pages go back to the allocator, metrics are updated, cancellations are reconciled, and the stream closes cleanly so the scheduler can admit more waiting work.

๐Ÿ”Ž provider detail
Typical epilogue: emit data: [DONE], record latency and tokens/sec, decrement active-request counters, and free the request context so memory is reusable for the next admission decision.
state: req#418 complete; KV slots freed; metrics updated; connection closed with [DONE].
Final effect: the provider turned one HTTP call into a streamed token sequence without rerunning the whole prompt on every step.
Hardening concerns

Rate limiting / backpressure

Protect the queue before GPU memory is the thing saying no.

Validation & prompt limits

Reject oversize or malformed requests before tokenization and admission.

KV budget & eviction

Track cache pages explicitly; reclaim aggressively when requests finish or cancel.

Graceful degradation

Under load, prefer shorter queues, lower concurrency, or smaller models over collapse.

Health checks

Separate liveness from readiness so a pod can be alive but not accepting work.

Speculative decoding

Use a draft model to reduce latency when verification cost still nets a win.

Quantization

Smaller weights can raise throughput if the accuracy trade-off is acceptable.

Tensor parallelism

Shard big models across GPUs when one device cannot hold weights or bandwidth demand.

Continuous vs static batching

Static batches are simpler; continuous batching usually keeps expensive devices busier.

Timeout & cancellation

Clients disappear. The provider must stop wasted decode work quickly and safely.

The OpenAI-compatible API surface
API Surface [Routes and wire format]

Most providers expose a small compatibility layer above the inference engine. Matching the request schema and streaming format matters because it lets existing SDKs, tools, and agents talk to the server without knowing whether the backend is vLLM, TGI, llama.cpp, or something custom.

๐Ÿ”Ž provider detail
Common endpoints:
POST /v1/chat/completions โ†’ chat messages in, streamed deltas out
POST /v1/completions โ†’ raw prompt completion
GET /v1/models โ†’ available model list

OpenAI-compatible mostly means the same JSON fields and the same SSE event shape.
state: compatibility lives at the HTTP boundary; behind it, providers can schedule, batch, quantize, and stream however they like.
โ† Ch15: Database-Optimized Inference chapter index Glossary โ†’