What you type becomes numbers
“The capital of France is”
state: raw text, 26 characters
โ
Tokenizer (BPE / SentencePiece)Split the sentence into small reusable pieces, then swap each piece for a dictionary number.
๐ข how it splits
Scans left-to-right, finds the longest match in a fixed merge table. "capital" is one entry. "un"+"believ"+"able" would be three. The table was built during training โ frozen at inference.
state: ["The", " capital", " of", " France", " is"] โ [450, 3007, 310, 3444, 338]
5 numbers. The text is gone โ only these IDs remain.
โ
Embedding Lookup (Embedding Matrix E)Use each ID as a row number into a giant table. Copy out that row โ 4,096 numbers per token.
๐ข the lookup
How it works: the model has a table with 128,000 rows and 4,096 columns. Each row represents one token from the vocabulary.
Given a token ID (say, 3007 for " capital"), we simply go to row 3007 and copy out all 4,096 numbers. No multiplication. No transformation. Just a table read โ like looking up a row in a spreadsheet by row number.
We do this once per token in the input. Five tokens = five row copies = five strips of 4,096 numbers stacked up.
state: 5 rows ร 4,096 numbers each = 20,480 floats
row 0: [-0.22, 0.74, 0.04, -0.16, โฆ] โ "The"
row 1: [ 0.52, -0.10, 0.27, 0.04, โฆ] โ " capital"
row 2: [-0.09, 0.41, -0.33, 0.21, โฆ] โ " of"
row 3: [ 0.67, -0.04, 0.18, -0.30, โฆ] โ " France"
row 4: [ 0.11, 0.56, -0.09, 0.38, โฆ] โ " is"
โ
Position Stamp (RoPE โ Rotary Position Embedding)Rotate pairs of numbers based on position so the model knows word order.
๐ข the rotation
Take two adjacent numbers, rotate them by an angle that depends on position:
new_a = aยทcos(ฮธ) โ bยทsin(ฮธ)
new_b = aยทsin(ฮธ) + bยทcos(ฮธ)
ฮธ increases with position. Different dimension pairs rotate at different speeds.
Why this matters: "The capital of France is" and "Is the capital of France" contain the same tokens โ but the model must treat them differently. Position stamping means "capital" at position 1 and "capital" at position 3 produce different attention scores, so word order changes the output. Same tokens, different positions, different result.
The deeper reason: RoPE makes the dot product between any two tokens depend on their distance โ not just their content. So the model can learn things like "when a noun is roughly two positions before a verb, that's probably the subject." The weights don't have to invent a notion of distance. RoPE provides one. The weights learn to exploit it.
How the pieces divide responsibility:
โข Embeddings say: "I'll convert tokens into vectors."
โข RoPE says: "I'll represent sequence order as geometry."
โข Attention says: "I'll compare vectors."
โข Training says: "Given those rules, I'll learn weights that minimise prediction error."
Each part does one job. RoPE's job is to make distance visible in the geometry so that attention and the learned weights can use it without having to rediscover it from scratch.
state: same 5 ร 4,096 grid, but each row now has a unique rotational fingerprint for positions 0โ4
The thinking loop
ร 32 Layers
โ
Normalise (RMSNorm)Rescale numbers so nothing drifts too large or too small between layers.
๐ข the formula
How it works, step by step:
1. Square every number in the row
2. Take the average of all those squares
3. Take the square root of that average โ this is the "RMS" (root mean square)
4. Add epsilon (ฮต) before the square root โ epsilon is a tiny safety number (like 0.00001) that stops you dividing by zero if all values happen to be near zero
5. Divide every original number by this RMS value โ now the row has a stable overall size
6. Multiply each number by a learned scale (ฮณ) โ one scale per position in the row, trained alongside the model weights, so the network can choose which dimensions to amplify or suppress
Where ฮณ comes from: it's stored in the model weight file, one value per dimension (4,096 of them). It was learned during training by gradient descent โ the same way all other weights are found.
state: each row rescaled to stable magnitude (RMS = 1.0)
โ
Attention (Multi-Head Self-Attention)Each token looks back at all earlier tokens and blends in whatever seems relevant.
๐ข the three projections
Each token gets transformed three different ways:
Q (Query) โ "what am I looking for?"
Multiply the token's row by a weight table called W_Q. The result is a new row of numbers that represents what this token needs from context. Think of it as the question this token is asking.
K (Key) โ "what do I contain?"
Multiply the same token's row by a different weight table called W_K. The result represents what this token offers to others. Think of it as the label on this token's filing cabinet drawer.
V (Value) โ "what should I contribute if selected?"
Multiply by yet another weight table W_V. The result is the actual information this token will hand over if attention decides it's relevant. Think of it as the contents of the drawer.
Why three? Because "what I'm looking for", "what I advertise", and "what I actually hand over" are three different things. A token might advertise itself as relevant (high K match) but contribute different information (V) than what it advertises.
How they combine:
1. Compare every Q against every K โ this is a dot product (multiply matching pairs, add them up). High result = "these two are relevant to each other."
2. Scale down by โd โ the scores get large when rows are wide (4,096 numbers). Dividing by โ4096 = 64 keeps them manageable.
3. Softmax โ turn the raw scores into weights between 0 and 1 that add up to 1. The highest score gets the most weight, low scores get almost nothing. (It uses e^x to exaggerate differences, then divides by the total.)
4. Blend the V rows using those weights โ tokens that scored highest contribute most to the output.
Kแต just means "K, flipped on its side" (transposed) โ so the dot products line up correctly when multiplying Q against K.
state (token 4, " is"):
looked back โ "France" scored 0.42, "capital" scored 0.31, "The" scored 0.08 โฆ
blended result: a new row that mixes in context from the whole sentence
โ
+ Add Back (Residual Connection)Don't throw away what you already knew. Add the new update on top of the old state.
๐ข the operation
output = input + update
Add each number in the update to the matching number in the original, position by position. The original signal survives โ the layer can only add new information on top, never erase what was already there. This is why transformers can be deep (32+ layers) without losing the thread.
โ
Normalise (RMSNorm)Rescale again before the next transformation.
โ
Feed-Forward (SwiGLU FFN)Each row processed individually: expand, filter, compress back. Patterns become decisions.
๐ข the expansion
What happens, in plain steps:
1. Gate projection โ multiply the 4,096-number row by a weight table to produce 11,008 numbers. This "gate" will control what gets through.
2. Up projection โ multiply the same original row by a different weight table, also producing 11,008 numbers. This is the raw expanded signal.
3. Gating โ run the gate values through SiLU (a smooth on/off curve: negative numbers get squashed toward zero, positive numbers mostly pass through). Then multiply the gate result with the up result, number by number. This lets the network selectively keep or suppress parts of the expanded signal.
4. Down projection โ multiply the 11,008 filtered numbers by a third weight table to compress back down to 4,096.
Why expand to 11,008? The wider middle layer gives the network more room to rearrange information before squeezing it back. Think of it as spreading the work across a wider desk, then filing the result back into the original drawer.
What is SiLU? A smooth activation curve. Numbers above zero pass through roughly unchanged. Numbers below zero get pushed toward zero. It's what makes the network nonlinear โ without it, stacking layers would just be one big multiplication and couldn't learn complex patterns.
state: 4,096 โ expand to 11,008 โ filter โ back to 4,096 (sharper, more refined)
โ
+ Add Back (Residual Connection)Merge update into running state. Hand the richer rows to the next layer.
after all 32 layers: token 4 (" is") has been refined 32 times.
It no longer means "is" โ it means "the-word-after-'The-capital-of-France-is' in a context expecting a city name."
Choosing the next token
โ
Final Normalise (RMSNorm)One last rescale before scoring.
โ
Score Every Token in the Dictionary (lm_head โ Vocabulary Projection)Compare the finished row against all ~128,000 known tokens. Each gets one score.
๐ข the scoring
How it works:
The model has a vocabulary table โ one row per token it knows (about 128,000 rows, each 4,096 numbers wide). The finished hidden state is also 4,096 numbers.
For each vocabulary row, we compute a dot product: multiply matching positions together and add them all up. The result is one number โ the "score" (also called a "logit") for that token.
High score = "this token fits well as the next output."
Low score = "this token doesn't fit here."
We do this for all 128,000 rows. That's 128,000 ร 4,096 = 524 million multiply-adds. It's a full table scan โ checking every candidate in the dictionary.
What's a logit? Just the raw score before it gets turned into a probability. The name comes from "log-odds" but you can think of it as: bigger number = more likely candidate.
scores (top 5 of 128,000):
" Paris" โ 18.4
" France" โ 15.8
" French" โ 13.9
" Lyon" โ 11.7
" London" โ 11.1
โฆ 127,995 more rows scored lower
โ
Pick One (Sampling โ Softmax + Top-K/Top-P)Turn scores into probabilities. Apply temperature, top-k, top-p. Sample one winner.
๐ข softmax + sampling
Turning scores into a choice:
1. Temperature (ฯ) โ divide every score by ฯ before anything else. Lower ฯ (like 0.3) makes the differences between scores more extreme โ the top candidate dominates. Higher ฯ (like 1.5) flattens things out โ more candidates have a chance. ฯ=1.0 means "use scores as-is."
2. Softmax โ convert the scores into probabilities that add up to 1. How: take e (โ2.718) raised to the power of each score, then divide each result by the sum of all results. Big scores become big fractions, small scores become near-zero.
3. Top-K โ only keep the K tokens with the highest probability. Throw the rest away. This stops the model from ever picking something deeply unlikely.
4. Top-P (nucleus) โ sort tokens by probability highest-first. Keep adding tokens until the running total reaches P (e.g. 0.95 = 95%). Discard the rest. This adapts: when the model is confident, only 1โ2 tokens survive; when it's unsure, more remain.
5. Sample โ pick one token randomly from what's left, weighted by probability. Or if temperature is 0, just take the top one (greedy).
probabilities after softmax:
" Paris" โ 89.1% | " France" โ 6.7% | " French" โ 1.9% | โฆ
sampled: " Paris" โ
โ
" Paris"
Token emitted. One token done.
โ
โฉ Feed " Paris" back in โ repeat for the next token
A decoder-only transformer (GPT / Llama / Claude style). For the full deep-dive with every multiply shown, start with Chapter 1.