We begin with six words:
The capital of France is
We don't ask why the model chooses Paris. We simply watch what physically happens.
Frame the chapter correctly: this is not a philosophy of language. It is a register dump. We follow one prompt through a frozen runtime the way you would follow one packet through a NIC, one query through a query plan, or one instruction through a pipeline stage by stage.
Step 1: Characters to tokens
Start with the actual character stream. At the front of the engine there is no concept of truth, geography, or capitals. There is only text and a tokenizer that knows how to split that text into discrete pieces the model was trained to consume.
T h e โต c a p i t a l โต o f โต F r a n c e โต i s
The tokenizer has a fixed merge table, learned during training and frozen at inference time. This stage is a dictionary lookup, not a neural network. Operationally, it scans left to right and keeps trying to match the longest piece in its vocabulary. When it sees The, it finds a stored entry for that exact span. When it moves forward and sees a leading space plus the next word, it may match capital as one piece because the space-prefixed form was common enough in the training corpus to earn its own entry. That is what byte-pair encoding looks like when stripped of mythology: a deterministic table walk over a frozen lexicon.
"The" โ vocabulary entry 450 " capital" โ vocabulary entry 3007 " of" โ vocabulary entry 310 " France" โ vocabulary entry 3444 " is" โ vocabulary entry 338
"The capital of France is"
โ
BPE merge table lookup
โ
[450, 3007, 310, 3444, 338]Five integers. That's all the model will ever see. The original characters are gone.
That sentence matters. From this point onward, nothing inside the network has access to the raw spelling of France or Paris. If the tokenizer had split the string differently, every subsequent memory access would change. Different IDs mean different rows, different projections, different dot products, and eventually different logits. The rest of the pipeline is numerically heavy, but its first branching decision happened here.
Step 2: Token IDs โ Embedding vectors
Each token ID is a primary key into a table.
Embedding table E Shape: [131,072 rows ร 4,096 columns] Row 0: [-0.012, 0.034, -0.067, 0.019, ...] Row 1: [ 0.041, -0.055, 0.023, 0.088, ...] ... Row 450: [-0.221, 0.738, 0.042, -0.156, ...] โ "The" ... Row 3007: [ 0.519, -0.103, 0.271, 0.044, ...] โ " capital" ...
This is an indexed lookup. Not a computation. Not a transformation. We copy row 450 into position 0 of our working buffer. We copy row 3007 into position 1. And so on.
Working buffer X โ shape: [5 tokens ร 4,096 floats]
Position 0 ("The"): [-0.221, 0.738, 0.042, -0.156, ... ] โ copied from row 450
Position 1 (" capital"): [ 0.519, -0.103, 0.271, 0.044, ... ] โ copied from row 3007
Position 2 (" of"): [-0.088, 0.412, -0.331, 0.207, ... ] โ copied from row 310
Position 3 (" France"): [ 0.672, -0.041, 0.183, -0.295, ... ] โ copied from row 3444
Position 4 (" is"): [ 0.114, 0.556, -0.089, 0.378, ... ] โ copied from row 338Five rows. Each 4,096 numbers wide. 20,480 floats total. This is the starting state of our execution.
Think of this like loading registers from RAM into an execution buffer. The embedding matrix is persistent storage. The working buffer is live state. At this instant nothing has been inferred yet. We have only materialized the learned representation for each token into active memory so the arithmetic core can begin. If you inspected device memory at this point, you would see dense rows of floats, not symbols.
Step 3: RoPE โ Rotate by position
The embedding table has no concept of order. If we shuffled these five rows, the table wouldn't know.
RoPE fixes this by rotating pairs of numbers based on position.
Take two adjacent numbers from position 4 (" is"):
Before: [0.114, 0.556]
Position 4, dimension pair 0
Angle ฮธ = 4 ร 0.01 = 0.04 radians โ 2.3ยฐ
cos(0.04) = 0.9992
sin(0.04) = 0.0400
After:
new_a = 0.114 ร 0.9992 โ 0.556 ร 0.0400 = 0.0917
new_b = 0.114 ร 0.0400 + 0.556 ร 0.9992 = 0.5601
Result: [0.0917, 0.5601]That's it. A 2D rotation. Different pairs rotate at different frequencies โ fast-varying pairs change rapidly with position, slow-varying pairs change gradually. This creates a unique rotational signature for each position.
Every pair of numbers in the vector gets rotated. 4,096 floats = 2,048 rotation operations per token.
It helps to separate what changed from what did not. The token still represents is. We did not fetch a new embedding row. We altered how its coordinates line up with other coordinates when later dot products happen. RoPE does not add a separate position column; it rewrites the vector so position becomes inseparable from content during similarity checks.
Placement detail: RoPE is applied to the attention projections, not to the embedding directly. We are previewing the math here so that when those projections appear a few paragraphs from now, the rotation has somewhere concrete to land.
Critically, RoPE is applied to the attention projections (Projections #1 and #2), not to the embedding directly. We'll see this in a moment.
Step 4: Layer 1 โ The first of 32 passes
The working buffer now enters the execution core. This section repeats 32 times with different weights. Same operator sequence. Different learned matrices. Trace one layer carefully and the rest of the stack stops looking like mysticism; it becomes a loop.
We will keep one eye on the whole five-token buffer and one eye on a single lane: position 4, the token " is". That is the final prompt position, so after 32 layers it will be the row that gets projected against the vocabulary to choose the next token.
4a. RMSNorm
Before anything else, normalise. Compute the root-mean-square of the 4,096 values, divide each by it, multiply by a learned scale.
x = [0.4, -0.8, 0.2] RMS = โ(mean(0.4ยฒ + 0.8ยฒ + 0.2ยฒ)) = โ(mean(0.16 + 0.64 + 0.04)) = โ0.28 = 0.529 normalised = [0.4/0.529, -0.8/0.529, 0.2/0.529] = [0.756, -1.512, 0.378] ร learned scale ฮณ = [1.1, 0.9, 1.0] output = [0.831, -1.361, 0.378]
This keeps the numbers in a stable range. Without it, values would drift as they pass through 32 layers.
RMSNorm is cheap relative to the matrix multiplies that follow, but cheap is not the same as optional. It keeps the dynamic range of activations near the regime the model was trained on. In CPU terms, treat it like a calibration stage between large arithmetic blocks. In database terms, treat it like statistics maintenance that keeps later operators well-conditioned.
4b. Three projections
Focus on one token โ position 4, " is" โ and one projection.
Projection #1: "What information am I looking for?"
Input: x = [4096 numbers] Weight: Wโ = [4096 ร 4096 matrix] Output: pโ = x ร Wโ = [4096 numbers]
What just happened? Each of the 4,096 output numbers is a weighted sum of all 4,096 input numbers. The weights were learned during training. We just performed 4,096 ร 4,096 = 16,777,216 multiply-adds.
Nothing mystical happened. A vector went in. A matrix multiplied it. A new vector came out.
If you want a hardware mental model, each output coordinate is an accumulator register. It starts at zero. Then 4,096 products stream through it. Do that for 4,096 output coordinates and you have the full projected row. The model is not hunting through a symbolic graph here. It is executing dense linear algebra on a fixed-width lane.
Projection #2: "What information do I contain?"
Same operation. Different weights. Another 16.7 million multiply-adds.
Projection #3: "What information should I return if selected?"
Same operation. Different weights. Another 16.7 million multiply-adds.
Three projections. 50 million multiply-adds. For ONE token in ONE layer. And we have 5 tokens ร 32 layers.
The reveal: these three projections are called Q, K, and V in the literature โ Query, Key, Value. Now you know what they actually do.
The letters matter less than the roles. One vector asks what the token wants. One vector advertises what the token offers. One vector carries the payload that will be blended if the token gets selected. Once you internalize that, attention stops feeling like jargon and starts feeling like a dataflow graph.
4c. RoPE on projections #1 and #2
Now RoPE is applied โ but only to Projections #1 and #2. Not #3. Why? Because the rotation makes dot products between #1 and #2 depend on relative position. #3 is just the payload โ it doesn't participate in the relevance computation.
This is where the earlier 2D rotation becomes operational. We are not rotating for aesthetics. We are modifying the searchable and matchable representations so that when a token compares itself to earlier tokens, the score depends not just on content but on where those tokens sit relative to one another.
4d. Attention โ actually compute one
Now every token's Projection #1 is compared against every previous token's Projection #2.
Token 0 ("The"): P#1 = [0.6, 0.2] P#2 = [0.8, 0.1] P#3 = [1.0, 0.5]
Token 1 (" capital"): P#1 = [0.3, 0.9] P#2 = [0.5, 0.4] P#3 = [0.2, 0.8]
Token 2 (" of"): P#1 = [0.7, 0.1] P#2 = [0.3, 0.7] P#3 = [0.6, 0.3]Token 2 wants to know: who is relevant to me?
Compare Token 2's P#1 against every token's P#2:
Token 2 P#1 = [0.7, 0.1] vs Token 0 P#2 = [0.8, 0.1]: dot = 0.7ร0.8 + 0.1ร0.1 = 0.57 vs Token 1 P#2 = [0.5, 0.4]: dot = 0.7ร0.5 + 0.1ร0.4 = 0.39 vs Token 2 P#2 = [0.3, 0.7]: dot = 0.7ร0.3 + 0.1ร0.7 = 0.28 Raw scores: [0.57, 0.39, 0.28]
Scale by โd (to prevent large values from saturating softmax):
โ2 โ 1.414 Scaled: [0.403, 0.276, 0.198]
Softmax โ convert to weights that sum to 1:
exp(0.403) = 1.497
exp(0.276) = 1.318
exp(0.198) = 1.219
Sum = 4.034
Weights: [1.497/4.034, 1.318/4.034, 1.219/4.034]
= [0.371, 0.327, 0.302]Now blend the P#3 vectors using those weights:
0.371 ร [1.0, 0.5] = [0.371, 0.186]
0.327 ร [0.2, 0.8] = [0.065, 0.262]
0.302 ร [0.6, 0.3] = [0.181, 0.091]
โโโโโโโโโโโโโโโโโ
Sum = [0.617, 0.539]That's Token 2's new representation after attention. It's a weighted blend of what every other token offered to contribute, with weights determined by relevance.
This is one attention head. Real models run 32 heads in parallel, each with different learned weights, each learning different notions of "relevance."
At full model scale, these score calculations happen across much larger vectors and across all valid prior positions. For the last token in our five-token prompt, that means comparing against the representations for The, capital, of, France, and itself. Causal masking prevents any future position from participating because there is no future position yet. The model can look left, not right.
4e. Output projection
The 32 heads' outputs are concatenated and projected back to 4096 dimensions with yet another matrix multiply.
Concatenation here is bookkeeping, not synthesis. Each head produced a slice. The output projection mixes those slices back into one 4,096-wide row the residual stream can carry forward. If one head learned local syntax and another learned entity continuation pressure, this projection is where those separate channels get recombined into one active state.
4f. Residual connection
We don't throw away what we already knew. We ADD the attention output back to the input:
output = original_input + attention_result
This is critical. Without residuals, deep networks forget their earlier state. The residual is a highway lane โ information can skip over layers it doesn't need.
Residuals are why the whole stack behaves like iterative refinement instead of total replacement. Every layer gets a chance to adjust the current row, not rewrite history from scratch. That makes depth workable. You can think of each block as proposing a delta against the running state.
4g. Feed-forward network
After attention, each token passes through a feed-forward network. Individually. No token-to-token interaction here.
gate = x ร W_gate [4096 โ 11008] โ controls information flow up = x ร W_up [4096 โ 11008] โ expands the representation combined = SiLU(gate) ร up โ elementwise gating down = combined ร W_down [11008 โ 4096] โ compress back
Three more matrix multiplies. The middle dimension is 11,008 โ nearly 3ร wider than the model width. This is where the network has room to compute.
Add another residual.
Why widen first? Because a narrow 4,096-wide row only gives the layer so much room to express combinations of features. Expanding to 11,008 creates temporary scratch space. The gate decides which lanes should open. The up projection supplies candidate features. The down projection compresses the useful result back into the residual stream so the next layer sees the same fixed width again.
4h. Assembly checklist after Layer 1
Layer 1 complete. Operations performed (per token): โ RMSNorm ~8,192 multiply-adds โ Projection #1 (Q) 4096 ร 4096 = 16,777,216 multiply-adds โ Projection #2 (K) 4096 ร 4096 = 16,777,216 multiply-adds โ Projection #3 (V) 4096 ร 4096 = 16,777,216 multiply-adds โ Attention scores + softmax + blend ~262,144 multiply-adds โ Output projection 4096 ร 4096 = 16,777,216 multiply-adds โ RMSNorm ~8,192 multiply-adds โ Gate projection 4096 ร 11008 = 45,088,768 multiply-adds โ Up projection 4096 ร 11008 = 45,088,768 multiply-adds โ Down projection 11008 ร 4096 = 45,088,768 multiply-adds Total for one token, one layer: โ 202 million multiply-adds
This happens for all 5 tokens. And then the whole thing repeats for layers 2 through 32.
5 tokens ร 32 layers ร 202M = roughly 32 billion multiply-adds.
That's where "7 billion parameters" meets reality.
Parameter count is a storage fact. This operation count is a runtime fact. The two are related but not identical. Seven billion parameters tell you how many learned numbers the model owns. Billions of multiply-adds tell you what it costs to use those numbers for one forward pass. This chapter is about the second quantity.
Step 5: Layers 2โ32
Same operations. Different weights. Each layer refines the representations.
By layer 16, the hidden state for position 4 no longer represents just "is". It represents 'is, following "The capital of France", in a context where a geographical answer is expected.'
By layer 32, the model has compressed the entire prompt's meaning into the last token's hidden state.
The key word there is compressed. The model does not produce a side database saying subject=capital and country=France. It keeps revising one dense row of 4,096 floats until that row is positioned in activation space so that the correct next-token rows score highly. That is why interpretability is hard. The information is present, but it is distributed and entangled across coordinates, heads, and layers.
Step 6: Final RMSNorm
One more normalisation pass before the final projection.
This is the last cleanup stage before we compare the final hidden state against the entire vocabulary. After 32 layers of additions, projections, gates, and residual merges, we stabilise the magnitude one final time so the output scores are computed in the numeric regime the output head expects.
Step 7: lm_head โ the full table scan
Now comes the most expensive single operation.
The hidden state is 4,096 numbers. The vocabulary has 131,072 entries. Each vocabulary entry is also 4,096 numbers.
We compute one dot product per vocabulary row.
Hidden state h = [0.34, -0.12, 0.88, ...] (4096 numbers)
Row 0 (""): dot(h, row_0) = -3.14
Row 1 ("the"): dot(h, row_1) = 2.81
Row 2 ("a"): dot(h, row_2) = 1.73
...
Row 3681 ("Paris"): dot(h, row_3681) = 18.42 โ highest
...
Row 8892 ("France"): dot(h, row_8892) = 15.77
...
Row 12041 ("London"): dot(h, row_12041) = 11.09
...
Row 131071 ("โ
"): dot(h, row_131071) = -8.22131,072 dot products. Each one multiplies 4,096 pairs and sums the results. Total: 131,072 ร 4,096 = 537 million multiply-adds.
The result: 131,072 raw scores (logits). One per possible next token.
ORDER BY score DESC: Rank Token Score โโโโโ โโโโโโโโโ โโโโโโ 1 " Paris" 18.42 2 " France" 15.77 3 " French" 13.91 4 " Lyon" 11.72 5 " London" 11.09 ... 131072 "โ " -8.22
This is the cleanest place to apply database language. The output head is a full table scan followed by an ORDER BY score DESC. The highest-scoring row is not guaranteed to be emitted, because sampling policy still has a say, but the model has now finished its actual inference work. From here onward we are shaping and selecting from a scored result set.
Read this table literally: the model is not โthinking of Paris.โ It has produced a hidden state that aligns most strongly with the vocabulary row for " Paris" when all 131,072 rows are scanned.
Step 8: Sampling
Temperature scales the scores before softmax:
ฯ = 0.7 Scaled: 18.42/0.7 = 26.31, 15.77/0.7 = 22.53, 13.91/0.7 = 19.87, ...
Softmax converts to probabilities:
P(" Paris") = 0.891
P(" France") = 0.067
P(" French") = 0.019
P(" Lyon") = 0.008
...Top-P (nucleus) with p=0.95: keep tokens until cumulative probability reaches 0.95:
" Paris" 0.891 โ cumulative: 0.891 " France" 0.067 โ cumulative: 0.958 โ stop here
Sample from {Paris: 0.930, France: 0.070} (renormalised).
Random number says: Paris.
Token emitted: 3681
Decode it back to text: ' Paris'
The leading space matters because the tokenizer stored the token that way. The model did not emit the abstract word Paris. It emitted one vocabulary entry whose decoded surface form is a space followed by Paris. That is the piece that gets appended to the running text buffer.
Step 9: The next token โ and why it's different
Now the prompt is: The capital of France is Paris
We need the next token. But we don't rerun the entire 32-layer computation for all 6 tokens.
During the first pass (called prefill), we already computed Projection #2 and Projection #3 for every token at every layer. We stored those results. That's the KV cache.
For the new token ' Paris', we only need to:
1. Look up its embedding 2. Apply RoPE for position 5 3. Compute Projections #1, #2, #3 for THIS token only 4. Compare its #1 against ALL cached #2s (from the prefill) 5. Blend using cached #3s 6. Run the feed-forward network for this token only 7. Repeat for all 32 layers 8. Project against vocabulary 9. Sample
That's the decode phase. One new token per step. Each step reuses the cached state.
Prefill is the all-at-once pass over the original prompt. Every prompt token gets embedded, projected, attended, normalized, and pushed through all 32 layers. This phase is compute-heavy and parallel-friendly because the whole prompt exists up front.
Decode adds one token at a time. The new token still traverses all 32 layers, but only its fresh row needs to be computed. That shifts the bottleneck away from raw arithmetic throughput and toward latency, cache reuse, and memory movement.
KV cache is exactly what it sounds like: stored key and value projections from earlier tokens at every layer. Queries are new each step. Keys and values from history are reused.
Prefill: all tokens in parallel, compute-heavy, GPU-friendly.
Decode: one token at a time, bandwidth-heavy, latency-sensitive.
This is why long responses feel slow even on powerful hardware. Every single token requires a separate pass through 32 layers.
The qualitative experience of chatting with a model comes directly from this split. The first token often takes longer because prefill had to process the whole prompt. Later tokens arrive in a steadier cadence because the model is now just extending the cached context one step at a time. When the cadence slows on long outputs, you are feeling the cost of repeated decode passes.
Closing: The whole machine
Let's count what just happened to produce one token:
Operations to produce " Paris": Tokeniser: 5 table lookups Embeddings: 5 ร 4,096 = 20,480 floats copied RoPE: 5 ร 2,048 = 10,240 rotations 32 layers: ~32 billion multiply-adds Final RMSNorm: ~8,192 multiply-adds lm_head: ~537 million multiply-adds Sampling: 131,072 exponentials + sort Total: โ 32.5 billion multiply-adds for one token.
Nothing in this list is mysterious. Every step is a lookup, a projection, a dot product, a reduction, a normalisation, or a sampling decision.
The model doesn't understand France. It doesn't understand capitals. It executes a fixed program โ the same program for every prompt โ and the learned weights steer that program toward outputs that, statistically, complete the sequence well.
The rest of this book zooms into each stage. But keep this execution trace in your head. It's the whole machine.
Once you can narrate this path from memory, the later chapters become local deep-dives instead of separate mysteries. Tokenization is just the front-end compiler pass. Embeddings are just the table lookup. Attention is just the relevance-weighted join. The feed-forward network is just the per-row rewrite. The output head is just the final table scan. The model is large, but the control flow is short enough to hold in one head.