Chapter 17

Implementation Reference

The exact shapes, layouts, and gotchas that matter when you stop reading and start building.

Weights / file format

Model Loading [GGUF format]

Find tensors by name, not file order: common patterns are attn_q, attn_k, attn_v, attn_output, ffn_gate, ffn_up, ffn_down.
Tensor shapes are stored in GGUF tensor info; read dimensions exactly as listed and keep row-major ordering intact.
All modern GGUF files are little-endian.
Tensor payloads are aligned to 32-byte boundaries; trust the recorded offset.
Core metadata such as d_model, n_layers, n_heads, and vocab_size lives in header KV pairs.

┌─────────────────────────────────────┐
│ Magic: "GGUF"                       │
│ Version: 3                          │
│ Metadata KV pairs (architecture,    │
│   n_layers, n_heads, d_model, ...)  │
│ Tensor info (name, shape, offset)   │
│ ─────────────────────────────────── │
│ Tensor data (aligned, contiguous)   │
└─────────────────────────────────────┘

time-saver: Parse metadata and tensor info first, then map tensors by name. Never hardcode offsets.

Text in / token IDs out

Tokenizer

Byte fallback means unknown characters decompose into raw byte tokens instead of crashing the pipeline.
Handle BOS and EOS explicitly; some models require BOS on every prompt.
Special tokens cover padding, role markers, tool markers, or chat-template sentinels.
The merge table comes from GGUF metadata or a paired tokenizer.json.

prompt text
  ↓ normalize / template
subwords from merge table
  ↓ if unknown span appears
byte fallback tokens 0x00..0xFF
  ↓
token IDs + special IDs (BOS / EOS / PAD / <|system|> ...)

time-saver: Most “broken tokenizer” bugs are really missing BOS/EOS rules or missing byte fallback.

Vocabulary table

Embeddings

Prefill should gather N token rows in one batch, not one token at a time.
Check whether embeddings are tied to lm_head or stored as a separate output matrix.

token_ids [5]  →  E[128000 × 4096]  →  X[5 × 4096]
                  (copy 5 rows)

time-saver: Embedding lookup is a gather, not a matrix multiply.

Critical execution order

One Transformer Layer [the exact execution order]

Pre-norm block: RMSNorm happens before attention and before the FFN.
RoPE is applied to Q and K only.
Residuals happen after attention output and after FFN output.

Input [batch × 4096]
    ↓
RMSNorm
    ↓
Q = x × W_Q    [4096 × 4096]
K = x × W_K    [4096 × 1024]  (GQA: fewer K heads)
V = x × W_V    [4096 × 1024]  (GQA: fewer V heads)
    ↓
Split Heads
  Q: [batch × 32 heads × seq × 128]
  K: [batch × 8 heads × seq × 128]
  V: [batch × 8 heads × seq × 128]
    ↓
RoPE (applied to Q and K only)
    ↓
Attention (per head)
    ↓
Merge Heads (concatenate back)
  [batch × seq × 4096]
    ↓
Wo = attn_out × W_O    [4096 × 4096]
    ↓
+ Residual (add input back)
    ↓
RMSNorm
    ↓
Gate = x × W_gate   [4096 × 11008]
Up   = x × W_up     [4096 × 11008]
    ↓
SiLU(Gate) × Up      (element-wise)
    ↓
Down = result × W_down  [11008 × 4096]
    ↓
+ Residual (add input back)
    ↓
Output [batch × 4096]

time-saver: If outputs diverge, verify execution order before debugging math kernels.

Shape changes, not extra math

Attention [head reshaping]

Reshape: reinterpret flat [4096] as [32 × 128] for Q.
Transpose: rearrange to [heads, sequence, head_dim] before dot products.
Concat: join all head outputs back into one [4096] vector.

4096 hidden dimension
    ↓
32 Q heads × 128 dims/head = 4096  ✓
 8 K heads × 128 dims/head = 1024  (GQA)
 8 V heads × 128 dims/head = 1024  (GQA)

critical insight: These are not multiplications. They're just changing how you interpret the same block of memory.

The memory bill

KV Cache [the exact memory layout]

Store keys and values per layer, KV head, position, and dimension.
Allocate for max_seq up front or page it explicitly.
Decode appends one position per token; old positions stay live until eviction or teardown.

cache[layer][head][position][dimension]

Example for Llama-7B:
  32 layers × 8 KV heads × max_seq × 128 floats

At position 5 with 2048 max context:
  Total cache per sequence:
    32 × 8 × 2048 × 128 × 2 (K+V) × 2 bytes (FP16)
  = 256 MB
  ≈ 2 GB at batch size 8

critical insight: The KV cache is why long conversations use more memory. Every token you've ever generated stays cached until the conversation ends or you evict it.

Decode is not prefill

Decode [the one thing that saves weeks of confusion]

For token N+1, recompute Q for the new token only.
Read historical K and V from cache for all previous positions.
Attention work is now “one query row against the whole cached history.”

During decode (generating token N+1):

  Only Q is recomputed for the new token.
  K and V come from cache.

That's it. That's why decode is fast per-token
but slow in wall-clock: you're only computing
one row of Q, but reading ALL cached K and V
for the attention comparison.

New token → Q_new [1 × 32 × 128]
Cache     → K_all [32 × 8 × (N) × 128]   (all previous)
Cache     → V_all [32 × 8 × (N) × 128]   (all previous)

Attention: score Q_new against all K_all
           blend V_all by those scores
           → one new output row

time-saver: Decode latency is dominated by reading the cache and weights, not by recomputing the whole prompt.

Vocabulary projection

lm_head [vocabulary scoring]

Use the final hidden state row only.
Conceptually it is one dot product per vocabulary row.
In practice this is a GEMV or small GEMM, but the loop mental model is correct.

for each token_id in 0..131071:
    score[token_id] = dot(hidden_state, vocab_row[token_id])

That's literally it.
131,072 dot products of length 4,096.
The highest score wins.

time-saver: Think “scan the dictionary” first; optimise to GEMV after the reference path works.

From logits to one token

Sampling

Mask banned or structurally illegal tokens before softmax.
Temperature, top-k, and top-p all operate on the candidate set before the final pick.
Greedy decode is just “skip randomness and take argmax.”

1. logits       — the raw scores from lm_head (131,072 numbers)
2. masking      — set banned tokens to -infinity
3. ÷ temperature — divide scores to sharpen or flatten
4. top-k        — keep only the K highest, zero the rest
5. softmax      — convert to probabilities (0–1, sum to 1)
6. top-p        — keep tokens until cumulative prob ≥ p
7. sample       — pick one randomly, weighted by probability
   (or greedy: just take the highest)

time-saver: Probabilities do not exist until softmax. Everything before that is logit surgery.

Whole engine lifecycle

Runtime [the complete lifecycle]

load owns weights and tokenizer state.
prefill fills the KV cache for the entire prompt.
decode loops token-by-token until a stop condition triggers teardown.

load     — read GGUF, allocate weight memory, parse tokenizer
prefill  — forward pass on all prompt tokens, fill KV cache
decode   — loop: forward 1 token, score vocab, sample, emit
stream   — send each token to the client as it's produced
stop     — EOS token, max_tokens, or stop sequence hit
destroy  — free KV cache, free weight memory, close connection

time-saver: Treat load, prefill, and decode as separate phases in code and profiling; they bottleneck for different reasons.

← Ch16: Build the Provider chapter index Glossary →