Inference: The Vocabulary of Running AI

The Building Blocks

When you run an AI model — inference — you're executing a computational graph. Every concept below is something that graph is made of. None of them are magical. All of them are just numbers being multiplied and added together, at extraordinary scale.

Decoder vs Encoder: Two Families of Transformer

The original 2017 Transformer had two halves — an encoder and a decoder. The field split them apart and built entirely different model families from each half. Understanding which half you're using explains most of the differences between models.

The Key Difference: Causal Mask vs Full Attention

This single difference — whether attention is causal (forward-only) or bidirectional (full) — determines what the model can do. Decoders can generate because they're trained to predict the future without seeing it. Encoders can't generate, but they understand context more deeply because they see the entire input at once.

Why BERT can't write a story: It was trained to fill in blanks, not to continue sequences. It has no concept of "what comes next" — only "what fits here, given everything around it."
Why GPT can't do BERT's job as well: When GPT classifies a sentence, each token only sees what came before it. BERT sees the whole sentence. For understanding tasks, bidirectional context wins.

Encoder-Decoder (The Original Architecture)

Some models keep both halves. The encoder processes the full input bidirectionally, then the decoder generates output autoregressively while attending to the encoder's representations.

Models: T5, FLAN-T5, mBART, Whisper (for speech)
Good at: Translation, summarisation, speech-to-text — tasks where the input and output are different sequences with different structures

Size and Scale

The model families also differ dramatically in scale, which explains their different roles in practice:

BERT-class models are small because they don't need to generate — they just need to understand. A 110M parameter BERT can power a semantic search system that handles millions of queries. A 70B decoder model can reason about novel problems but costs 600× more compute per input.

The Vocabulary

Parameter

A parameter is any number in the model that was learned during training. When someone says "a 70 billion parameter model," they mean the model contains 70 billion individually tuned numbers that collectively encode everything the model knows.

Weights — the numbers inside the matrices (W_Q, W_K, W_V, FFN matrices, embeddings). The vast majority of parameters.
Biases — offset values added after a matrix multiply (the +b in Wx + b). Some architectures omit them entirely.
Scale factors — learned per-element multipliers in normalisation layers (γ in RMSNorm).

Parameters are fixed after training — they don't change during inference. The number of parameters determines the model's capacity (how much it can learn), its memory footprint (how much RAM/VRAM it needs), and its compute cost (how many operations per token). More parameters generally means more capable, but also more expensive to run.

Not all numbers in a running model are parameters. Activations (intermediate values), the KV cache, and optimizer states are all computed at runtime — they're not part of the model's parameter count.

Inference

Inference is the act of running a trained model to produce an output. Training teaches the model. Inference uses what it learned.

Everything on this page — matmuls, attention, softmax, quantisation — is about making inference faster, cheaper, and more accessible. Training is a one-off cost. Inference is the ongoing cost that scales with every user, every query, every token.

Token & Tokenisation

Models don't process text as characters or words — they process tokens. A tokeniser splits text into subword chunks that the model was trained on. Each token maps to an integer ID, and each ID maps to a vector in the embedding table.

Common words get a single token: "the", "Hello", "function"
Rare words get split into pieces: "quantisation" → "quant", "isation"
Vocabulary size is typically 32K–128K tokens — a fixed dictionary the model was trained with

Tokens are the atomic unit of everything. Context length is measured in tokens. Cost is measured per token. The model generates one token at a time. When people say a model has a "128K context window," they mean it can process ~128,000 tokens in a single pass — roughly 100,000 words.

Embedding

An embedding is a dense vector of numbers that represents a token (or any discrete item) as a point in continuous space. The embedding table is a matrix with one row per vocabulary token — look up a token ID, get its vector.

Words with similar meanings end up near each other in embedding space. "King" is near "queen." "Python" is near both "snake" and "programming" — in different directions. The geometry of the embedding space is the model's understanding of meaning.

The embedding table is the first and last thing in the model. Input tokens are looked up in it to enter the model. At the output, the final hidden state is projected back against it (or a separate output matrix) to produce logits over the vocabulary.

Context Window (Sequence Length)

The maximum number of tokens a model can process in a single pass. Everything — your prompt, the conversation history, the system instructions, and the model's response — must fit within this window.

GPT-2 (2019): 1,024 tokens
GPT-3 (2020): 2,048 tokens
GPT-4 (2023): 8K–128K tokens
Modern models (2024+): 128K–1M+ tokens

Attention is O(N²) in sequence length — doubling the context quadruples the attention compute. FlashAttention and other optimisations reduce the memory cost, but the fundamental compute scaling remains. Long context is expensive.

KV Cache

During autoregressive generation, the model produces one token at a time. Without caching, it would need to recompute the Key and Value vectors for every previous token at each step — an enormous waste of compute.

The KV cache stores the K and V tensors from all previous tokens so they only need to be computed once:

Step 1 (prefill): Process the entire prompt. Compute and cache K, V for all input tokens across all layers.
Step 2+ (decode): For each new token, compute its Q, K, V. Append its K, V to the cache. Attend to the full cached K, V. Generate the next token.

The KV cache is often the dominant memory consumer during long-context inference. For a 70B model with 128K context in BF16, the KV cache alone can exceed 40 GB. This is why quantised KV caches (INT8, FP8) and techniques like GQA (grouped-query attention) matter — they reduce the cache size without meaningful quality loss.

The distinction between "prefill" (processing the prompt) and "decode" (generating tokens) is fundamental to inference performance. Prefill is compute-bound (large MatMuls). Decode is memory-bound (reading the KV cache for a single token's attention).

Tensor

A tensor is a multi-dimensional array of numbers. That's it. A single number is a 0D tensor (scalar). A list of numbers is 1D (vector). A grid of numbers is 2D (matrix). Stack grids and you get 3D, 4D, and beyond.

Scalar — a single number: 3.14
Vector — a list: [0.2, −0.5, 0.8]
Matrix — a grid: 512 rows × 512 columns
Tensor — any of the above, or higher: [batch × sequence × hidden_dim]

Every weight, every activation, every input to an AI model is stored as a tensor. When people say "a 7B parameter model," they mean 7 billion numbers stored across hundreds of tensors.

Weight

A weight is a single learned number inside the model. During training, the model adjusts billions of these weights so that the outputs become useful. After training, the weights are fixed — they are the model. The weight tensors (W_Q, W_K, W_V, the feed-forward matrices, the embeddings) hold everything the model has ever learned.

The model file you download is just the weight tensors serialised to disk. Load them into GPU memory, and you have a model.

Layer

A layer is one step in the model's processing pipeline. In a Transformer, each layer contains a self-attention block and a feed-forward block. The input enters, gets transformed, and exits as a new representation. Stack 32, 80, or 128 of these and you have a deep model — "deep" literally means "many layers."

Layer 1 — learns surface patterns (syntax, common phrases)
Layer N/2 — learns intermediate structure (relationships, grammar)
Layer N — learns abstract meaning (intent, reasoning, context)

Each layer refines the representation. Early layers are concrete; later layers are abstract. The "depth" is what gives these models their power.

Feed-Forward Network (FFN / MLP)

Inside every Transformer layer, after attention has decided what to look at, the feed-forward network decides what to do with it. It's a simple two-layer neural network applied independently to each token:

Step 1: Project the token into a wider space (W₁ expands, e.g. 4096 → 11008 dimensions)
Step 2: Apply a nonlinear activation function (GeLU, SiLU) — this is where the network gains its expressive power
Step 3: Project back down to the original size (W₂ compresses, e.g. 11008 → 4096)

The expand-then-compress pattern gives the network a "wide bottleneck" where it can mix features freely before squeezing them back down. The FFN accounts for roughly two-thirds of the total parameters in a Transformer — and two-thirds of the compute.

Think of attention as routing — it chooses which information to combine. The FFN is processing — it transforms that information into something new. Every layer does both: route, then process.

Up Projection & Down Projection

The two matrix multiplications inside the feed-forward network have specific names. They describe what happens to the dimensionality:

Up projection (W_up) — expands the hidden state into a wider intermediate space. This is where the network gets room to represent complex feature combinations. The expansion ratio is typically 2.7–4× the model dimension.
Down projection (W_down) — compresses back to the original dimension. The information that survived the activation function gets squeezed back into the residual stream.

In gated architectures (LLaMA, Mistral), there's also a gate projection — a second up projection whose output element-wise-multiplies the first, acting as a learned gate that controls which features pass through. This is the SwiGLU variant:

The up and down projections are the largest weight matrices in the model. In a 7B model, each layer's FFN has three matrices of ~4096×11008 — that's ~135M parameters per layer, just in the FFN.

Activation Function

An activation function is a simple nonlinear operation applied element-by-element to a vector. Without it, stacking layers of matrix multiplications would collapse into a single matrix multiplication — the network would be linear, incapable of learning anything complex.

The word "activation" has a dual meaning in AI. The activation function is the nonlinearity (ReLU, GeLU, etc.). An activation (noun) is the output value after applying that function — the intermediate numbers flowing through the network during inference. When people talk about "activation memory," they mean the memory used to store all these intermediate values.

RMSNorm — Root Mean Square Normalisation

Normalisation keeps numbers from exploding or vanishing as they pass through dozens of layers. RMSNorm is the version most modern Transformers use (LLaMA, Mistral, Gemma). It's simpler and faster than the original LayerNorm.

Step 1: Square every element, take the mean, take the square root — that's the RMS (root mean square)
Step 2: Divide each element by the RMS — now the vector has unit scale
Step 3: Multiply by a learned scale factor γ (per element) — the model learns the right magnitude

The difference from LayerNorm: RMSNorm skips the mean-subtraction step. It doesn't re-centre the values, only re-scales them. This removes one reduction operation per normalisation — which matters when you're doing it billions of times.

ε (epsilon) is a tiny constant (e.g. 10⁻⁶) added to prevent division by zero. Every normalisation formula has one.

MatVec (Matrix-Vector Multiply)

The fundamental operation of neural network inference. Take a matrix (the weights) and multiply it by a vector (the input). Every row of the matrix produces one output number — a weighted sum of the input.

A model with 7 billion parameters does billions of these multiply-and-add operations per token. This is why GPUs matter — they're built to do this in bulk. The entire cost of inference is dominated by matrix multiplications.

MatMul (Matrix-Matrix Multiply)

MatVec multiplies a matrix by a single vector. MatMul multiplies a matrix by another matrix — which is equivalent to doing many MatVecs in parallel, one per column of the second matrix.

In attention: Q · K^T is a MatMul — every query against every key, producing the [seq_len × seq_len] attention score matrix
In batched inference: processing multiple tokens at once turns every MatVec into a MatMul — the batch dimension becomes columns
In the FFN: the up and down projections are MatMuls when processing a full sequence

MatMul is what GPUs are truly optimised for. Tensor Cores on NVIDIA GPUs, AMX on Intel, and the matrix engines on TPUs are all specialised MatMul accelerators. The arithmetic intensity of MatMul (O(N³) compute for O(N²) data) means the hardware can stay busy — the ratio of compute to memory access is high enough to saturate the silicon.

Dot Product

The simplest form of similarity measurement. Take two vectors, multiply them element-by-element, and sum the results. If the vectors point in the same direction, the dot product is large. If they're orthogonal, it's zero. If opposite, it's negative.

In attention, Q · K^T computes the dot product between every query and every key — this is how the model measures "how relevant is token A to token B?" Every attention score starts as a dot product.

Cosine Similarity

The dot product measures similarity, but it's affected by vector magnitude — longer vectors give bigger dot products even if they point the same direction. Cosine similarity fixes this by normalising:

+1 — vectors point in the same direction (identical meaning)
0 — vectors are orthogonal (unrelated)
−1 — vectors point in opposite directions (opposite meaning)

Cosine similarity is the standard metric for comparing embeddings — sentence similarity, RAG retrieval, semantic search. It measures the angle between two vectors, ignoring their length. Two documents can have very different word counts but similar meaning — cosine catches that.

Softmax

Takes a list of raw numbers and converts them into a probability distribution — all positive, all summing to 1. Large values get amplified; small values get crushed toward zero.

In attention — converts raw attention scores into weights that sum to 1
At the output — converts logits into a probability distribution over the full vocabulary

Softmax is the collapse mechanism. It takes a field of possibilities and forces a decision.

Logits

Logits are the raw, unnormalised scores that a model outputs before softmax is applied. The final layer of a Transformer produces one number per vocabulary token — typically 32,000–128,000 numbers. These are the logits.

Before softmax: logits can be any real number — positive, negative, large, small. They have no probabilistic meaning yet.
After softmax: they become a probability distribution — all positive, summing to 1. Now you can sample from them.

Logits are the model's raw opinion. Temperature scales them. Softmax normalises them. Sampling picks from them. Every text generation pipeline works with logits as the intermediate currency between the model's final layer and the token it actually outputs.

The name comes from "log-odds" — in logistic regression, the logit function is the inverse of the sigmoid. In modern usage, it just means "the pre-softmax output scores."

Q, K, V — Query, Key, Value

The three projections that make attention work. For each token, the model computes three different vectors from the same embedding:

Q · K^T computes relevance. Softmax normalises it. The result weights the V vectors. Each token's output is a weighted blend of every other token's value, based on learned relevance.

Attention — The Full Mechanism

Attention is the core operation that makes Transformers work. It's how the model decides which tokens are relevant to which other tokens — and blends their information accordingly.

1. Compute Q · K^T — a matrix of similarity scores between every pair of tokens
2. Divide by √d_k — prevents scores from growing too large, which would push softmax into near-one-hot outputs
3. Apply softmax per row — each token now has a probability distribution over all other tokens
4. Multiply by V — each token's output is a weighted sum of all value vectors, where the weights are the attention scores

In practice, models use multi-head attention — they run 32–128 independent attention operations in parallel, each with its own Q, K, V projections. One head might track syntax, another co-reference, another semantic similarity. The outputs are concatenated and projected back down.

SDPA — Scaled Dot-Product Attention

SDPA is the formal name for the attention formula above — "scaled dot-product attention." The "scaled" part is the division by √d_k. In code, this is the function you actually call:

PyTorch's SDPA implementation automatically selects the fastest available kernel for your hardware:

FlashAttention — fuses the entire attention computation into a single GPU kernel, avoiding materialising the full N×N attention matrix. Reduces memory from O(N²) to O(N). The reason modern models can handle 128K+ context.
Memory-efficient attention — similar tiling strategy for GPUs that don't support FlashAttention
Math fallback — the naive implementation when nothing better is available

Before FlashAttention, a 32K context window needed ~4 GB just for the attention matrix of one layer. FlashAttention made long-context models practical by never storing that matrix at all — it computes attention in tiles, streaming the result.

Output Projection

After multi-head attention concatenates the outputs from all heads, the result passes through one more linear layer — the output projection:

W_O is a learned matrix that mixes the outputs from all attention heads back into a single representation of the original dimension. Without it, the heads would contribute independently — the output projection lets the model learn how to combine what different heads discovered.

There's also an output projection at the very end of the model — the language model head — which maps the final hidden state into logits over the vocabulary. This is the last matrix multiply before softmax and sampling.

RoPE — Rotary Position Embeddings

Transformers process all tokens in parallel — they have no inherent sense of order. Positional encoding injects sequence position into the model. RoPE is the method most modern LLMs use (LLaMA, Mistral, Qwen, Gemma).

The idea: encode position by rotating the query and key vectors in pairs of dimensions. Token at position m gets its Q and K vectors rotated by an angle proportional to m:

Relative by construction: when Q and K are both rotated, the dot product Q·K^T naturally depends on the difference between their positions — not the absolute position. The model learns relative distance for free.
Extrapolation: because it's based on rotation angles, RoPE can extend to sequence lengths longer than training — the basis of most context-length extension techniques (NTK-aware scaling, YaRN).
No learned parameters: the rotation frequencies are fixed (like sinusoidal encodings), so RoPE adds zero extra parameters to the model.

The different dimension pairs rotate at different frequencies — low dimensions rotate slowly (capturing long-range position), high dimensions rotate fast (capturing local position). It's the same idea as Fourier features: encode position at multiple scales simultaneously.

Temperature

A single number that controls how decisive the model is when sampling. It scales the logits before softmax:

T → 0 — softmax becomes argmax. The model always picks the most likely token. Deterministic, repetitive.
T = 1 — raw probabilities. The model samples naturally from its distribution.
T > 1 — distribution flattens. Less likely tokens get a real chance. Creative, chaotic.

Temperature doesn't change what the model "thinks." It changes how aggressively it commits to its strongest prediction. Low temperature = sharp collapse. High temperature = broad exploration.

Latent / Latent Space

A latent is a hidden representation — a vector of numbers that the model uses internally but that doesn't correspond directly to anything a human would recognise. The "latent space" is the high-dimensional coordinate system these vectors live in.

Why it matters: The model never works with raw inputs. It projects everything into latent space first — a compressed, structured representation where the geometry encodes meaning. Nearby points mean similar things. Directions encode concepts. The model reasons in this space, and the output is decoded back into tokens, pixels, or audio.

When someone says "latent diffusion," they mean "diffusion that happens in a compressed representation space, not in raw pixel space." When someone says "the model's latent representations," they mean the internal vectors the model computes between input and output.

Number Formats: Why Precision Matters

Every weight, every activation, every intermediate result is a number. How you represent that number — how many bits, what format — changes everything about cost, speed, and quality.

FP32 — 32-bit Floating Point

The gold standard. 1 sign bit, 8 exponent bits, 23 mantissa bits. Can represent numbers from ≈10⁻³⁸ to ≈10³⁸ with ~7 decimal digits of precision.

Pro: Maximum precision, no numerical issues
Con: 4 bytes per weight. A 70B model = 280 GB. Needs multiple high-end GPUs just to load.

BF16 — Brain Float 16

Google's format. 1 sign bit, 8 exponent bits (same range as FP32), but only 7 mantissa bits (~2 decimal digits of precision). Half the memory of FP32, same dynamic range.

Pro: Halves memory. Training-safe — the wide exponent range prevents overflow/underflow
Con: Less precise than FP16 in the mantissa. Fine for neural nets; terrible for scientific computing.

Most modern models train in BF16. It's the default precision for LLaMA, Mistral, and most open-weight models.

Quantisation: Shrinking the Numbers

Quantisation converts floating-point weights into lower-precision integers. Instead of storing each weight as a 16-bit or 32-bit float, you map the values into a smaller set of discrete levels. The idea: neural network weights are statistically well-behaved — most values cluster around zero, with tails that decay rapidly. You don't need 32 bits of precision to represent them.

INT8 — 8-bit Integer

Maps each weight to one of 256 levels. A scale factor and zero-point convert between the quantised integers and the original float values.

Memory: 1 byte per weight. 70B model = ~70 GB. Fits on a single large GPU.
Quality: Negligible degradation for most models. The standard for production inference.

INT4 — 4-bit Integer

Maps each weight to one of 16 levels. Half a byte per weight. This is where things get aggressive — you're representing the entirety of a weight's learned knowledge with just 4 bits.

Memory: 0.5 bytes per weight. 70B model = ~35 GB. Fits on consumer hardware.
Quality: Measurable but often acceptable degradation. GPTQ, AWQ, and GGUF all support this.
Technique: Group quantisation — quantise blocks of 32–128 weights together with a shared scale factor, preserving local distributions.

INT4 is the sweet spot for running large models on consumer GPUs. Most "local LLM" setups use 4-bit quantised models.

BitNet & Ternary Weights: {−1, 0, +1}

The extreme end of quantisation. Microsoft's BitNet b1.58 constrains every weight to one of three values: −1, 0, or +1. That's 1.58 bits per weight (log₂(3)).

When every weight is −1, 0, or +1, the matrix-vector multiply becomes pure addition and subtraction. The hardware implications are massive:

No multiplier circuits needed — adders are smaller, faster, and use less energy
~1.58 bits per weight — a 70B model fits in ~13 GB
Custom silicon — purpose-built chips can be radically simpler and more efficient
Microcontroller inference — models can potentially run on devices with no floating-point unit at all

The catch: you can't just quantise a trained model to ternary after the fact. BitNet models must be trained from scratch with ternary constraints. The model learns to distribute its knowledge across three values instead of continuous floats. The research shows that at sufficient scale, ternary models match the quality of full-precision models.

Trits — The Ternary Digit

A bit holds two states: 0 or 1. A trit holds three: −1, 0, or +1. It's the fundamental unit of ternary computing, and it's what BitNet weights are measured in.

Each trit encodes a simple instruction for the matrix multiply: +1 means add this input, −1 means subtract it, 0 means skip it entirely. No multiplication needed — just routing and addition.

Density: You can pack 5 trits into a single byte. A 70B-parameter ternary model stores its weights in ~17.5 billion bytes — about 16 GB including overhead.
Hardware: Ternary logic maps naturally to simple circuits — a trit can be encoded as two bits, or packed more efficiently. Custom silicon for trit-based models can skip the entire floating-point unit.
History: Ternary computing isn't new. The Soviet Setun computer (1958) was built on balanced ternary. BitNet brings the idea back, applied to neural network weights.

A ternary weight isn't a degraded float. It's a different primitive — one that trades precision for radical simplicity. When the model is trained to work in trits from the start, it learns to encode its knowledge in patterns of add, subtract, and skip.

The Precision-Cost Tradeoff

Why This Matters

The entire trajectory of making AI accessible is a story of precision reduction. FP32 models need a server room. BF16 models need a workstation. INT4 models run on a gaming laptop. BitNet models could run on a phone — or a microcontroller.

Every halving of precision roughly halves memory, doubles throughput, and halves energy consumption. The model quality barely changes until you push below 4 bits — and with BitNet, even that boundary is dissolving.

The future of AI isn't just better models. It's the same models, running everywhere, because the numbers got small enough.