Positional Encoding

Take the running prompt What is the capital of France? and imagine stripping away every clue about order. You still have the tokens What, is, the, capital, of, France, and ?, but you no longer know which one came first, which one modified which, or which token should be the current query position during generation. A transformer without positional information has exactly that problem. Attention can compare content, but content alone does not tell it whether France came before the question mark, or whether capital was adjacent to of. Positional encoding exists because sequence models need a representation of order every bit as much as they need a representation of token identity.

The subtlety is that bare self-attention is permutation-equivariant. If you permute the rows of the input matrix, the attention output permutes in the same way. That is useful for set-like processing, but language is not a set. In our example, France capital what is the of ? contains the same tokens as the real prompt and yet means almost nothing. Positional encoding is the mechanism that breaks that symmetry. It injects enough sequence structure that the model can distinguish “token 5 attends to token 2” from “token 2 attends to token 5”, even when the token identities are otherwise identical.

Why this stage exists

The attention core later computes a compatibility score between queries and keys. If those vectors carry only semantic identity, then two identical words at two different positions look identical to the scorer. That is unacceptable for autoregressive inference because position determines causality, locality, phrase structure, and often meaning. The system needs to know that the token at position 6 is the one deciding what comes next, and that position 5 contains France, which is highly relevant when answering a question about capitals. Positional encoding solves the missing-coordinate problem. It gives the model a way to represent order without abandoning the parallelism that makes attention attractive.

Attention(X) = softmax((XW_Q)(XW_K)^T / √d_k) (XW_V)

In English: take the input matrix X, project it into queries, keys, and values, score each query against every key, normalize those scores with softmax, and use the result to blend the values. Nothing in that equation says “first token”, “third token”, or “distance of four”. If you never add positional structure, the scorer has no direct representation of order. That is the core problem this chapter fixes.

For the running example, a useful mental model is a small numbered tape. Suppose tokenization yields seven prompt positions: 0=What, 1=is, 2=the, 3=capital, 4=of, 5=France, 6=?. During the first generation step, the query state for position 6 should notice that position 5 is close, that position 3 carries the concept capital, and that all of those tokens are in the causal past. Positional encoding makes those distinctions cheap and learnable.

Sinusoidal encoding: the original fixed scheme

The original transformer paper injected position by adding a deterministic vector to each token embedding. Every pair of dimensions behaves like a clock at a different frequency. Low-frequency dimensions change slowly across long spans; high-frequency dimensions change quickly across short spans. The beauty of the method is that the encoding is fixed, cheap to generate, and does not require learned parameters. You can produce positional vectors for any sequence length, at least numerically, without extending a learned table. That is why sinusoidal encoding still matters pedagogically even though large modern language models more often use RoPE or relative bias variants.

PE(pos, 2i)   = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))

In English: for dimension pair (2i, 2i+1), compute a sine and cosine using the token position pos and a frequency that depends on the pair index i. As i increases, the denominator changes, so different dimension pairs oscillate at different rates. Position zero starts with a predictable phase, nearby positions produce nearby phases, and faraway positions drift apart in a structured way. The result is a vector of length d_model that can be added directly to the token embedding at that position.

What the formula means operationally is that a position is not stored as one big counter value. It is spread across many periodic signals. That makes the representation smooth. Moving from position 5 to position 6 nudges every frequency pair a little, not in the same amount but in a consistent way. Because each pair has a different wavelength, the combined vector is rich enough that positions do not collapse onto one another over the range the model cares about. For implementation, you either precompute a [max_seq_len, d_model] table once or generate a row on demand and add it to the embedding row for each token.

x'_pos = x_pos + PE(pos)

In English: take the token vector already looked up from the embedding table and add the position vector elementwise. The model then sees a blended representation that carries both token identity and absolute position. This is why sinusoidal encoding is conceptually simple: the positional stage is just an additive bias in model space.

Why sine and cosine specifically? Because fixed offsets can be expressed as linear transformations of these phase pairs. The trigonometric identities sin(a+b) and cos(a+b) mean that if the network has learned how to interpret one position, it can also learn to relate it to another position by manipulating the paired signals. You do not need to memorize one independent embedding per offset. Relative displacement is already latent in the phase relationships. That idea becomes much more explicit in RoPE.

Learned positional embeddings: GPT-2 style

Learned positional embeddings keep the same absolute-position idea but replace the fixed sinusoid with another lookup table. Instead of computing trigonometric values, you allocate P ∈ R[n_ctx, d_model] and learn one vector per position during training. Inference then does two lookups for each token: one into the token embedding table and one into the position embedding table. The vectors are added together before entering the first block. Mechanically, it is hard to beat for simplicity. If you already have fast row-gather code for token embeddings, the positional path is almost free to implement.

x'_pos = E[token_id] + P[pos]

In English: take the token vector for the current token ID and add the learned vector for the current position index. The model is free to learn whatever absolute-position representation helps training. The downside appears when you ask for longer contexts than the model saw during training. Position 4097 does not exist if the table was trained only to 4096. You can extend or interpolate the table, but extrapolation quality is usually worse than methods designed to generalize distance more naturally.

From a systems perspective, learned absolute positions are the most boring option in the best sense: lookup plus add, both memory-bound, both easy to quantize and batch. From a modeling perspective, they are limited by explicit table length. That trade-off is why they show up in earlier GPT-style models and smaller implementations, but are less common in modern long-context LLMs that need stronger extrapolation behavior.

RoPE: rotary position embeddings

Rotary position embeddings are the positional mechanism you should understand first for modern open-weight LLMs. Llama-family models, Mistral, Qwen, Phi, and many derivatives use RoPE or a close extension of it. The core idea is simple: instead of adding a position vector into the hidden state, rotate each two-dimensional subspace of the query and key vectors by a position-dependent angle. Position becomes phase. Because attention scores are built from dot products between queries and keys, those rotations make the score depend on relative offset in a clean, mathematically useful way.

This is the important conceptual shift. In sinusoidal or learned absolute embeddings, you modify the token representation and hope later layers discover how to interpret position. In RoPE, position is wired directly into the attention mechanism. The vector used to ask the question and the vector used to answer the question are both rotated according to where they sit in the sequence. The attention score therefore changes with distance automatically, before any learned matrix tries to recover that information.

RoPE([x_2i, x_2i+1], pos) =
[ x_2i cos(pos·θ_i) - x_2i+1 sin(pos·θ_i),
  x_2i sin(pos·θ_i) + x_2i+1 cos(pos·θ_i) ]

In English: take each consecutive pair of dimensions, treat it as a tiny 2D vector, and rotate it by an angle equal to pos × θ_i. Each pair gets its own base frequency θ_i, so some dimension pairs rotate slowly and others quickly. Apply that transformation to every query pair and every key pair. The rotated vectors keep the same magnitude; only their orientation changes.

R_m^T R_n = R_(n-m)
⟨R_m q, R_n k⟩ = ⟨q, R_(n-m) k⟩

In English: the dot product between a query rotated by position m and a key rotated by position n depends on the difference n-m, not separately on the two absolute positions. That is the reason RoPE behaves like a relative-position method inside attention even though the rotations are applied using absolute indices. When the query at position 6 in our running example compares itself to the key at position 5, the score naturally reflects an offset of one step. Comparing to position 3 reflects an offset of three steps.

That relative-offset property is why RoPE ages so well under scale. Language dependencies are often local or at least distance-sensitive. A modifier is usually close to the noun it modifies. A just-opened parenthesis wants a nearby closing partner. The query at the sequence frontier mostly cares how far away a candidate memory is, not whether that memory lived at absolute index 173 or 174. RoPE feeds that inductive bias directly into the similarity calculation.

Why rotate Q and K, but not V

The attention score is computed from QK^T. Values V are not part of the relevance computation; they are payloads blended after the weights have already been decided. RoPE therefore rotates queries and keys because those are the vectors whose mutual geometry should encode positional distance. Rotating values would change the content being transported without helping the score calculation. In most implementations that would merely inject noise into the payload path and make learning harder. The clean rule is: queries ask, keys advertise, values carry data. Position belongs in the first two for RoPE.

In code, that means the Q and K projection outputs go through the rotation kernel, while V flows through untouched into the weighted sum. This detail matters because incorrect tutorials sometimes rotate all three or speak vaguely about rotating the hidden state itself. For modern inference kernels, the correct place is after the linear projection into head space and before the QK dot product.

Implementation: precomputed cos/sin and the complex trick

You never want to evaluate transcendental functions inside the hot attention loop if you can avoid it. Practical engines precompute cosine and sine tables for every sequence position and every half-dimension frequency, usually once per model or once per maximum context setting. llama.cpp popularized the freqs_cis style naming because the rotation can be viewed as complex multiplication by e^{j·pos·θ}. Pair two real dimensions into one complex number, multiply by a unit-magnitude complex phase, then unpack back to real form if needed. Whether you expose that as true complex arithmetic or explicit fused multiply-adds is an implementation detail.

struct RopeTables {
    float* cos_table;   // [max_seq, head_dim/2]
    float* sin_table;   // [max_seq, head_dim/2]
};

void apply_rope(float* q, float* k,
                const RopeTables* tbl,
                int pos, int n_heads, int head_dim)
{
    const float* c = tbl->cos_table + pos * (head_dim / 2);
    const float* s = tbl->sin_table + pos * (head_dim / 2);

    for (int h = 0; h < n_heads; ++h) {
        float* qh = q + h * head_dim;
        float* kh = k + h * head_dim;
        for (int i = 0; i < head_dim; i += 2) {
            float qc0 = qh[i];
            float qc1 = qh[i + 1];
            float kc0 = kh[i];
            float kc1 = kh[i + 1];
            float co = c[i / 2];
            float si = s[i / 2];

            qh[i]     = qc0 * co - qc1 * si;
            qh[i + 1] = qc0 * si + qc1 * co;
            kh[i]     = kc0 * co - kc1 * si;
            kh[i + 1] = kc0 * si + kc1 * co;
        }
    }
}

In English: fetch the cosine and sine row for the current token position, then rotate each adjacent dimension pair of every query and key head. The kernel is small, regular, and easy to vectorize. On CPUs you typically load two floats, apply a couple of FMAs, and write back. On GPUs you usually fuse RoPE into the Q/K preparation path so the data stays in registers as long as possible.

ALiBi: bias the score instead of the vectors

ALiBi, short for Attention with Linear Biases, takes a different position: do not encode position in the vectors at all. Leave Q, K, and V untouched, and instead add a distance-dependent penalty directly to the attention score matrix. Each head gets a slope. When a query looks further back in the sequence, the bias becomes more negative. Nearby tokens therefore receive a structural preference even before softmax. The mechanism is simple enough to explain in one line and often generalizes to longer contexts better than learned absolute tables.

score_h(m, n) = (q_m · k_n) / √d_k - slope_h · (m - n)

In English: compute the normal scaled dot-product score, then subtract a head-specific penalty proportional to the distance between the query position m and the key position n. Larger distance means larger penalty. Different heads use different slopes so some heads remain local and others tolerate longer spans.

ALiBi is attractive because it adds almost no extra memory and barely changes the kernel structure. The risk is that it gives you less expressive geometry than RoPE. It encodes a monotonic preference for nearness rather than a richer phase relation across dimensions. In practice, some model families like MPT used ALiBi successfully, especially when context extrapolation mattered more than exact rotary behavior.

YaRN and long-context RoPE scaling

Once RoPE became standard, the next practical problem was extending context length beyond the original training horizon. Naively stretching positions sounds easy: map a long context into the range the model already knows by interpolation, or simply increase the maximum index and trust the rotations. Both approaches degrade. High-frequency dimensions rotate too fast or lose the relationships the model trained on. Quality drops show up as confused retrieval, repetition, and weakened long-range coherence. YaRN, short for Yet another RoPE extensioN, is one family of fixes that adjusts the frequency schedule in an NTK-aware way so long contexts preserve more of the geometry the model learned at shorter lengths.

The practical lesson is that positional schemes are part of the model's learned geometry. You cannot casually stretch them without consequences. NTK-aware scaling methods modify the base frequencies or interpolate them nonuniformly so short-range behavior remains familiar while long-range positions become representable. The runtime cost of YaRN-like methods is basically the same as RoPE because the hot kernel still performs pairwise rotations; the change is in how you build the phase tables.

Tensor shapes, costs, and what can be cached

In terms of runtime cost, sinusoidal addition and learned positional addition are both O(T · d_model) and are usually memory-bound. RoPE is O(T · H · d_head) for the vectors you rotate, again usually bandwidth-bound rather than compute-bound because the arithmetic per element is small. ALiBi is cheapest if you generate the bias on the fly inside the attention score loop; it becomes expensive only if you foolishly materialize a dense [H, T, T] bias tensor. YaRN inherits RoPE's hot-loop cost almost exactly.

RoPE work per token ≈ 2 × H × (d_head / 2) small 2D rotations

In English: for each token, for each head, rotate every adjacent dimension pair in both the query and the key. That sounds large, but compared with the matrix multiplications around it, the arithmetic intensity is modest. The main performance question is whether the implementation keeps the phase tables and vector fragments close to the execution units.

Operation type and bound type classification

Hardware behavior: CPU, GPU, FPGA

On CPUs, learned tables and precomputed sinusoidal tables are straightforward streaming operations. RoPE also maps cleanly to SIMD because each pair rotation is a fixed pattern of multiplies and adds. The main mistakes on CPU are failing to precompute tables, using scalar transcendental calls per token, or laying out head data so badly that vector loads straddle cache lines. For short decode steps, instruction overhead and cache behavior matter more than raw FLOP count.

On GPUs, the positional stage is almost never the dominant FLOP consumer, but it can still hurt throughput if it triggers separate kernels or unnecessary global-memory round-trips. Good implementations fuse RoPE into Q/K packing, keep the phase row in shared memory or registers when possible, and avoid materializing temporary rotated copies larger than needed. ALiBi is especially friendly to fusion because the score bias can be added where the dot-product accumulator already lives. The larger sequence gets, the more important it is that positional logic disappears into the attention kernel rather than standing beside it as a separate pass.

On FPGAs, positional schemes split into easy and slightly-less-easy. Learned lookup tables and ALiBi are simple control-plus-memory patterns. RoPE is still tractable because the rotation is fixed-structure arithmetic and the cos/sin tables can live in block RAM or LUT-backed ROMs. If you want deterministic low-latency inference on a streaming design, RoPE is usually manageable. The main architectural question is how much context you provision phase storage for and whether you support multiple scaling modes such as YaRN without re-synthesizing the design.

Variants in modern models

Database analogy and systems programming analogy

The database analogy is that token embeddings are the rows and positional encoding is the clustered order metadata that lets the execution engine distinguish one row's place in the scan from another's. RoPE is more specific than that: it is like baking the row offset into the comparison function of a merge or lookup operator, so relevance depends partly on how far apart the records are in the ordered stream. ALiBi is like adding a distance penalty to the query planner's cost model: remote rows are still eligible, but they are slightly more expensive by default.

The systems-programming analogy is a streaming pipeline over an array of structs or a struct-of-arrays layout. Token embedding gives you payload fields. Positional encoding injects the index or transforms fields as a function of that index before the comparison-heavy stage runs. RoPE is especially like rotating registers based on loop index immediately before a SIMD compare, so later arithmetic sees both payload and relative offset without storing a separate metadata object per comparison.

Common implementation mistakes

The most common errors are mundane but destructive: off-by-one position indices between prefill and decode, applying RoPE to V, precomputing tables with d_model instead of d_head, forgetting that grouped-query attention may use a different number of KV heads, and extending context with naive interpolation that was never validated on retrieval-heavy prompts. Another frequent mistake is storing full ALiBi bias matrices when the bias could be generated on demand inside the score loop. That wastes memory bandwidth for no modeling gain.

There are also software-architecture mistakes. Engineers sometimes hide positional logic behind a high-level tensor abstraction and accidentally force extra allocations or layout conversions every token. Others quantize weights aggressively but leave phase tables in awkward formats that cost more to unpack than the rotation itself. In distributed inference, a mismatch in RoPE scaling configuration across workers can be catastrophic because everything still has the right shapes while the attention geometry silently disagrees.

If you were implementing this yourself...

If you were implementing this yourself, start by choosing the positional scheme your target model actually uses and then refuse to hand-wave the geometry. For a GPT-2 style model, write the position table lookup and addition path and validate that positions above the training window fail exactly the way you expect. For a Llama-style model, implement RoPE after Q/K projection, precompute phase tables for the supported context, and unit-test the rotation on tiny known vectors before you integrate it into attention. Then test the running example at the sequence boundary: verify that the query for the final token sees the correct positional offset to France and capital, and verify that decode-time position increments match prefill-time indexing. If positional encoding is wrong, every later chapter inherits the bug.