The Transformer Layer

A large language model is mostly repetition. Once tokenization, embedding lookup, and positional handling are done, the model applies the same kind of block again and again: normalize, project, attend, project back, add the residual, normalize again, run the feed-forward network, project back, add the residual again. Chapter diagrams often make this look grander than it is. Mechanically, one transformer layer is a carefully arranged set of matrix multiplies around a shared residual stream. The reason the stack becomes powerful is not that each layer is exotic, but that depth lets later layers refine and reuse what earlier layers produced.

For the running example What is the capital of France?, the embedding sequence enters layer 0 as a batch of context vectors with positions already injected. Layer 0 does not answer the question outright. It slightly improves the representation: perhaps strengthening the relation between capital and France, or sharpening the fact that the final token is a question boundary. Layer 1 receives that improved sequence and refines it further. After dozens of such passes, the hidden state at the last position becomes informative enough that the output head can score a token like Paris very highly.

The modern execution order

For most current LLMs, especially Llama-style models, the block is pre-norm and roughly follows this order:

RMSNorm → QKV projections → Attention → Wo projection → Residual add
→ RMSNorm → Gate projection → Up projection → Activation → Down projection → Residual add

That compact line hides almost all of inference. The attention half lets tokens read from one another. The feed-forward half applies a wider per-token transformation that can expand, gate, and compress information without cross-token interaction. The two residual additions keep the whole process anchored to the incoming state. If you can implement this block correctly and efficiently, you can implement most of a decoder-only transformer.

The pre-norm versus post-norm distinction matters. Older transformer variants often used post-norm: do the sublayer, add the residual, then normalize. Modern LLMs usually use pre-norm: normalize the stream first, feed the normalized copy through attention or FFN, and then add the result back into the unnormalized residual stream. Pre-norm makes optimization easier at large depth and also gives inference kernels a stable numerical distribution entering each projection. That is why the block order in production models is not arbitrary formatting; it is a design choice with convergence and stability consequences.

One block written as equations

n₁ = RMSNorm(x)
q = n₁ W_Q
k = n₁ W_K
v = n₁ W_V
a = Attention(q, k, v)
x₁ = x + a W_O

In English: start with the incoming residual stream x. Normalize it into n₁, project that normalized stream into queries, keys, and values, run attention, project the concatenated head output through W_O, and add the result back to the original x. The first half of the layer is now done. The important thing to notice is that the residual path bypassed the expensive attention stack and remained available for the add.

n₂ = RMSNorm(x₁)
g = n₂ W_gate
u = n₂ W_up
m = φ(g) ⊙ u
x₂ = x₁ + m W_down

In English: normalize the updated stream again, make two wide projections, apply an activation φ to the gate branch, multiply that gated branch elementwise with the up branch, then project back down to model width and add the result to the running residual stream. In Llama-style blocks, φ is usually SiLU and the pattern is often called SwiGLU. This second half does not mix tokens with one another. It operates independently on each position, but with a much wider hidden dimension that gives the layer expressive capacity.

Why residual connections exist

Residual connections solve several problems at once. During training they improve gradient flow by giving the optimizer a short path through depth. During inference they preserve information. If a given layer learns nothing useful for one token at one moment, the residual path means the token's prior representation still survives. Depth therefore becomes additive rather than destructive. Each block proposes a correction to the stream instead of replacing the stream wholesale. That is exactly the right bias for a 32- or 80-layer system where no single layer should be allowed to erase the entire accumulated state.

The phrase residual stream is useful because it emphasizes continuity. The model is not creating a totally new representation at every layer and discarding the old one. It is accumulating edits in a shared buffer. For the final token in our France example, early layers may mark question structure, middle layers may relate capital to country entities, and later layers may sharpen factual recall. All of those contributions live in the same width-d_model vector that keeps getting updated in place by residual additions.

Every projection is fundamentally the same operation

Engineers often overcomplicate the number of named matrices in a transformer. W_Q, W_K, W_V, W_O, W_gate, W_up, W_down, and even the lm_head at the end of the whole model are all the same class of primitive: take an input vector or matrix and multiply by a weight matrix. The semantic labels differ because the outputs are used differently, but the kernel primitive is identical.

output = input × weight_matrix

In English: a projection is just a linear map from one feature space to another. If the input is a whole sequence, you apply the same matrix to every token row. If the width changes, the matrix changes shape, but the operation class stays the same. This matters because implementation strategy follows the primitive, not the label. Quantization, tensor parallel sharding, GEMM tiling, cache reuse, and memory layout choices all care that these are matrix multiplies, not that one of them happens to be called W_Q.

Once you see that, a lot of transformer implementation becomes a scheduling problem around GEMMs. The layer is not seven mysterious neural gadgets; it is a small family of large dense multiplies plus a handful of cheap elementwise operators and reductions. That observation is why optimized inference libraries spend so much time on weight packing, fused kernels, and device-specific matrix engines. The math is old. The difficulty is moving the bytes efficiently.

Concrete shapes: Llama-7B style dimensions

Use a concrete model so the shapes stay honest. Suppose d_model = 4096, n_heads = 32, d_head = 128, and d_ff = 11008. For the prompt What is the capital of France?, imagine a sequence length T = 7 before the first generated token. Then the residual stream entering a layer has shape [T, 4096]. If the attention uses 32 full heads with no separate KV compression, W_Q, W_K, W_V, and W_O are each effectively [4096, 4096].

Grouped-query attention changes the K and V shapes, and attention details vary across model families, but the block skeleton stays the same. The point of the table is not to freeze one model forever. It is to make clear that every tensor has a known shape and every projection either preserves width or expands and contracts it in a predictable way.

Parameter count for one layer

With the Llama-style dimensions above and no biases, the attention projections contribute four square matrices: 4 × 4096 × 4096. The feed-forward section contributes three rectangular matrices: 3 × 4096 × 11008. Add two RMSNorm gain vectors of length 4096. That is already almost the entire parameter count of the block.

Attention params = 4 × 4096 × 4096 = 67,108,864
FFN params       = 3 × 4096 × 11008 = 135,266,304
Norm params      = 2 × 4096         = 8,192
Total per layer  = 202,383,360

In English: one dense layer block in this configuration has about 202.4 million parameters. Most of them live in the feed-forward network, not attention. That surprises people who focus only on the fame of attention. At short contexts, the FFN often accounts for a large fraction of the FLOPs as well.

Multiply that per-layer total by 32 layers and you get roughly 6.48 billion parameters inside the repeating blocks alone. Add token embeddings, the final normalization, and possibly an untied output head, and you land in the familiar marketed range of a 7B-class model. If the input embedding table and output projection share weights, total parameters are lower. If they are separate, add another full vocabulary-by-model-width matrix.

Embedding params = vocab_size × d_model
Example: 32,000 × 4096 = 131,072,000

In English: a 32k vocabulary with width 4096 needs another 131 million parameters for the token embedding table. If the lm_head is untied, add the same amount again for output scoring. At FP16, each parameter is two bytes, so 6.6 to 6.7 billion parameters means roughly 13.2 to 13.5 GB of raw weight storage before runtime overhead, alignment, KV cache, or temporary activations are counted.

Layer specialization and real model depth

Real decoder stacks commonly use 32, 40, 48, 70, or 80-plus layers depending on model class. Empirically, early layers often behave more syntactically, middle layers more relationally, and later layers more semantically or task-specifically, although those are tendencies rather than clean module boundaries. For the France example, early layers may mostly sharpen token identity and local phrase structure, while later layers are where factual associations and answer-token preference become more obvious. The important engineering point is that the residual stream lets those specializations accumulate without forcing a rigid handoff protocol between layers.

The stack depth also changes what dominates runtime. With very short sequences, deeper models pay mostly for repeated weight reads and GEMMs. With long prefills, attention's sequence-length scaling becomes more visible. With single-token decode, the layer count often dominates latency because every generated token must traverse every block in order. That is why seemingly small per-layer inefficiencies become painful at interactive generation time.

Prefill and decode are different workloads

It is worth pausing on one inference-specific point: the same transformer layer behaves differently during prefill and during decode. When the model first processes What is the capital of France?, all seven prompt tokens enter the block together. The QKV projections are true matrix-matrix multiplies over a small batch of rows, attention forms scores across the seven-token window, and the FFN processes seven rows at once. That is the regime where GPUs look happiest, because the block exposes enough parallel work to keep large matrix engines busy. If you benchmark only prefill, the layer looks pleasantly compute-heavy and often scales well with batch size.

Once the model emits Paris and moves to the next token decision, the workload changes. Old tokens are not fully recomputed through attention; their keys and values are already stored. The new token still traverses every layer, but now the projections look more like matrix-vector multiplies, and the attention read path walks the accumulated cache. Arithmetic intensity drops, weight reads repeat every step, and memory traffic starts to dominate. That is why single-token latency can disappoint even when prefill throughput looks strong.

The implementation consequence is simple: profile the block in both modes and optimize them differently. Prefill wants large fused kernels and batch-friendly GEMMs. Decode wants ruthless control over memory movement, cache layout, launch overhead, and synchronization. Many systems that look fast in synthetic throughput charts feel sluggish in a chat interface because they were tuned for prefill but not for the one-token-at-a-time reality that users actually experience.

Pseudocode for one block

void transformer_block(float* x,            // [T, d_model]
                       KVCache* cache,
                       const Weights* w,
                       int T,
                       int layer_idx,
                       int pos_base)
{
    float* n1 = rmsnorm(x, w->attn_norm);                 // [T, d_model]
    float* q  = matmul(n1, w->Wq);                        // [T, H*d_head]
    float* k  = matmul(n1, w->Wk);                        // [T, H_kv*d_head]
    float* v  = matmul(n1, w->Wv);                        // [T, H_kv*d_head]

    apply_rope(q, k, w->rope_tables, pos_base, T);
    cache_append(cache, layer_idx, k, v, T);

    float* attn = masked_attention(q,
                                   cache_keys(cache, layer_idx),
                                   cache_vals(cache, layer_idx),
                                   T);
    float* attn_out = matmul(attn, w->Wo);                // [T, d_model]
    add_inplace(x, attn_out);                             // residual add

    float* n2 = rmsnorm(x, w->ffn_norm);                  // [T, d_model]
    float* g  = matmul(n2, w->Wgate);                     // [T, d_ff]
    float* u  = matmul(n2, w->Wup);                       // [T, d_ff]
    silu_inplace(g);
    mul_inplace(g, u);                                    // SwiGLU gate
    float* ffn_out = matmul(g, w->Wdown);                 // [T, d_model]
    add_inplace(x, ffn_out);                              // residual add
}

In English: one block is an attention sandwich followed by an FFN sandwich, both wrapped around the same residual buffer. Temporary tensors appear and disappear, but the persistent state that matters is the updated x and the per-layer KV cache entries used for future decode steps.

Computational cost, memory cost, and what can be cached

The weight matrices dominate model storage. The KV cache dominates growing runtime state during autoregressive decode. Within one block, attention cost scales with sequence length because each query must compare against available keys, while the FFN cost scales mainly with width and is independent of context length for a fixed token batch. That means short-context inference often spends a large fraction of total FLOPs in the FFN, whereas long-context prefill increasingly exposes attention's quadratic score work.

Approx dense FLOPs per layer, per token:
QKV + Wo     ≈ 4 × d_model × d_model
Gate/Up/Down ≈ 3 × d_model × d_ff
Attention score/value terms add sequence-dependent cost

In English: ignoring constants and kernel details, the block pays four square projections for attention and three tall-and-wide projections for the FFN. The attention sublayer then adds extra work proportional to how much context each token can see. For decode with a single new token, the projection cost is fixed but the query still scans all cached keys and values from prior positions.

What can be cached? Static weights are obviously loaded once and reused. RoPE tables or ALiBi slopes are static auxiliary state. The big dynamic cache is keys and values per layer and per past token. You usually do not cache RMSNorm outputs, FFN hidden states, or attention probabilities across decode steps because they are specific to one token's forward pass and would cost too much memory relative to the benefit. Good implementations reuse scratch buffers aggressively instead of preserving them.

Operation type and bound type classification

Hardware behavior

On GPUs, prefill is dominated by large GEMMs where tensor cores shine. Decode looks different: one token or a tiny batch must still read huge weight matrices every layer, so arithmetic intensity drops and memory bandwidth becomes more painful. Fusing RMSNorm, RoPE, biasless projections, and activation steps around GEMMs can eliminate extra trips to HBM. Attention kernels must also read the growing KV cache efficiently, which is why layout choices and paged-cache designs matter so much.

On CPUs, transformer blocks live or die by cache locality, vectorization, and quantized weight formats. Matrices that fit poorly into LLC or require scattered dequant paths will crush throughput. Decode on CPU is especially unforgiving because the batch is tiny and the weight read cost is repeated for every token. That is why good CPU inference engines obsess over weight packing, grouped matvec kernels, and keeping the residual stream in contiguous, SIMD-friendly layouts.

On FPGAs, a transformer block is attractive because the execution order is rigid and repetitive. RMSNorm, projection pipelines, RoPE, and activation functions can be deeply pipelined, while on-chip memory holds slices of weights or cache metadata. The downside is capacity. Large dense weights and long KV caches quickly exceed on-chip storage, so practical FPGA systems often combine streamed external memory with aggressive quantization and careful partitioning of layers across fabric regions.

Variants used by modern models

Tensor parallelism and pipeline parallelism

Because almost every expensive substep is a projection, transformer layers shard naturally. In tensor parallelism, you split large weight matrices across devices. Column-parallel sharding is common for W_Q, W_K, W_V, W_gate, and W_up: each device computes a slice of the output features. Row-parallel sharding often fits W_O and W_down: each device contributes a partial sum that is reduced across devices. The residual stream remains logically one tensor, but physically its transforms are distributed.

Pipeline parallelism instead cuts the model along depth. GPU 0 may own layers 0 through 7, GPU 1 layers 8 through 15, and so on. During prefill or batched serving, microbatches can occupy different stages concurrently. During single-stream decode, pipeline bubbles are harder to hide because each token must march through the stages in order. That is why pipeline parallelism helps capacity but does not automatically solve interactive latency.

Database analogy and systems programming analogy

A transformer layer is like a physical query operator stack over an in-memory record batch. Attention is the read-across operator: each row can consult other rows subject to causal rules. The FFN is a per-row projection and enrichment stage. The residual stream is the working rowset that persists across operators, with each stage appending or refining columns in an implicit compressed form rather than replacing the table outright. From that view, a deep transformer is a long execution plan over the same evolving batch.

The systems analogy is a hot loop over a resident buffer. You have a main array x, a set of packed weight blobs, a per-layer KV cache, and ephemeral scratch buffers sized to the largest intermediate. Efficient implementations minimize copies, fuse passes, and reuse scratch pages. Inefficient ones materialize every intermediate, bounce between layouts, and turn a clean operator pipeline into a memory-allocation benchmark.

Common implementation mistakes

Typical bugs cluster around shape discipline and cache discipline. Examples: normalizing the wrong tensor in pre-norm, forgetting the residual add after W_O, applying activation to the wrong FFN branch, transposing a weight matrix incorrectly, mishandling grouped-query head counts, or writing KV cache entries with the wrong position offset during decode. These bugs often preserve tensor shapes and therefore survive basic smoke tests while silently corrupting model behavior.

Performance bugs are just as common. Launching separate kernels for every small elementwise step, reallocating scratch buffers every token, dequantizing into large temporary matrices instead of fused use, or materializing attention probabilities unnecessarily will wreck throughput. In distributed setups, mismatched sharding assumptions between W_O and the residual reduction path can produce either wrong answers or hidden synchronization costs that erase the benefit of parallelism.

If you were implementing this yourself...

If you were implementing this yourself, begin with one block and make it numerically boring before making it fast. Hard-code a tiny model width and sequence length, print every tensor shape, and verify the exact execution order: pre-norm attention, residual add, pre-norm FFN, residual add. Then swap in realistic widths and confirm that every named projection is just a matrix multiply with the expected shape. After correctness, attack memory traffic: fuse RMSNorm where practical, avoid unnecessary temporaries, and make the KV cache layout explicit. Finally, profile prefill and single-token decode separately. They stress the same block in different ways. If the one-block implementation is right, the rest of the model is mostly repetition plus routing.