Feed-Forward Networks

The feed-forward network, usually shortened to FFN or MLP, is the part of a transformer layer that does heavy per-token transformation without mixing information across sequence positions. Attention lets one token read other tokens. The FFN takes the resulting token representation and says, now that this token has context, reshape it through a large nonlinear circuit. If attention is the communication stage, the FFN is the per-token compute stage. It is where the model expands a hidden state into a much wider intermediate space, applies a nonlinearity, and compresses it back to the model width.

That expansion-compute-compression pattern is why the FFN feels like a private workshop attached to every token. The hidden state for the last token in "What is the capital of France?" arrives carrying context gathered by attention. The FFN does not ask any new questions about earlier tokens. Instead, it transforms the token's current feature mixture into a new one, often making some features stronger, damping others, and generating interactions that a purely linear stack could never express.

This is why FFNs matter so much. Without nonlinear per-token transformation, a transformer would collapse into a stack of largely linear mixing operations wrapped around residual paths. You could still move information around, but the model's ability to build rich conditional features would be much weaker. The activation function is the break in linearity, and the wide hidden layer is the room in which that nonlinearity has space to operate.

Why this stage exists

Suppose attention has already made the final token strongly aware that the phrase "capital of France" is pointing toward the concept of Paris. That awareness is still just a vector of numbers. The FFN gives the model a way to build higher-level internal features from that vector. Some neurons or channels may behave like detectors for geography, answer-form, capitalization style, or the likelihood that a city token should follow. The exact semantics are distributed and messy, but the structural point is simple: the FFN makes context-sensitive features sharper and more compositional.

In systems language, the FFN is the large nonlinear projector that sits between communication phases. It gives each token a deep transformation budget that is independent of sequence length. Whether the prompt has 7 tokens or 7,000, the FFN cost for one token depends mainly on d_model and d_ff, not on how many other positions exist. That is why the FFN remains important even when attention receives most of the conceptual attention.

In many modern models, the FFN is also the single biggest source of floating-point work per layer. People often assume attention dominates everything because it is the famous mechanism. But once you optimize attention with caches and smart kernels, the repeated large dense projections inside the FFN become impossible to ignore. On decode workloads in particular, the FFN is often one of the central compute costs for every generated token.

The standard and gated formulas

Standard FFN(x) = W_down(φ(W_up · x))
Gated FFN(x)   = W_down(σ(W_gate · x) ⊙ (W_up · x))

The standard form is two matrix multiplies with an activation in the middle. First project from d_model up to a larger width d_ff. Then apply an activation function φ. Then project back down to d_model. The gated form, common in current LLMs, adds a second up-projection called the gate. One wide projection produces content, another wide projection produces gate values, and the two are multiplied elementwise before the down projection.

Read the gated equation in English. Take the input vector x. Compute W_up · x and W_gate · x. Pass the gate path through an activation such as SiLU. Multiply the activated gate elementwise with the up path. Then send the result through W_down. The gate acts like a learned valve across the wide intermediate space: some channels pass strongly, some weakly, some almost not at all.

The elementwise multiply is cheap compared with the wide projections, but conceptually important. It turns the FFN from a simple nonlinear map into a gated nonlinear map, which often improves model quality. Architectures like LLaMA use SwiGLU-style gating, and that design choice is one reason you often see an apparently odd intermediate width such as 11008 for a model width of 4096. The gate and up paths both need large projections, but the chosen width is tuned so parameter count and compute stay in a useful budget.

Tensor shapes

That ratio, d_ff ≈ 2.67 × d_model, is a common modern choice for gated designs. With d_model = 4096, a typical intermediate width is 11008. Earlier non-gated transformers often used 4 × d_model with a simpler activation. The exact multiplier varies by architecture, but the pattern is stable: widen aggressively, do nonlinear work in the wide space, then compress back down.

Why widen so much?

The expansion is not arbitrary decoration. A wider intermediate space gives the model room to represent many more candidate features than fit in the base residual stream. The input hidden state may be 4096 numbers wide, but once you project it to 11008 or more, the model can temporarily express a much richer set of detectors, mixtures, gates, and interactions. Most of those intermediate channels never survive intact. They exist only long enough for the activation and down projection to decide what should flow back into the narrow residual path.

That is the same reason a compiler may use a generous intermediate representation or a database engine may build a larger temporary structure than the final answer requires. The wide space is working memory. It is not the stable schema of the model; it is the scratch zone where the model can test many possibilities before compressing them into the next hidden state.

The down projection is therefore not just a cleanup step. It is the stage that decides which wide-space computations deserve to re-enter the main residual stream. If the up projection creates the candidate feature cloud, the down projection is the bottleneck that distills that cloud back into a reusable token state.

Activation functions

The activation function is where the FFN stops being a linear sandwich. Without an activation, two stacked linear projections collapse into one equivalent linear projection, which would waste the entire wide hidden space. The activation introduces curvature: some values pass unchanged, some are damped, some are gated smoothly, and the output depends on magnitude in a nonlinear way.

The reason current LLMs gravitated toward gated smooth variants is empirical rather than theological. Hard nonlinearities like ReLU are simple, but smooth gates often preserve richer gradient behavior during training and richer feature modulation during inference. Once a model family proves that a gated design yields better quality for the same deployment class, that design quickly becomes the new default.

Weight matrices and weight types

A weight matrix is just a large grid of learned numbers that defines a linear transformation. In the FFN, those grids are unusually large because the hidden width expands so aggressively. For a 4096-to-11008 projection, each matrix contains over 45 million parameters. Multiply that by multiple matrices per layer and dozens of layers per model, and you can see why FFN weights occupy so much of the total parameter budget.

The important deployment distinction is this: total parameters and active parameters are not the same thing. In a dense FFN, every token uses the whole block. In an MoE FFN, every token sees only the routed experts. That can buy more representational capacity without multiplying per-token FLOPs by the full number of experts. But it also introduces routing overhead, expert imbalance problems, and more complicated serving systems.

Mixture of Experts routing in detail

router_scores = softmax(W_router · x)
experts = top_k(router_scores)
FFN_MoE(x) = Σ(selected expert i) p_i · Expert_i(x)

In an MoE layer, the router computes scores over experts for the current token. The system then selects the top-k experts, often one or two, and runs only those expert FFNs. Their outputs are weighted by the router probabilities and summed. Conceptually, this is conditional computation: the model has many specialist sub-networks available, but each token only pays for a few.

Load balancing matters because without it, the router may overuse a small subset of experts. During training, models often add auxiliary balancing losses or capacity constraints so experts receive work more evenly. During inference, imbalance still matters operationally because a hot expert can become a bottleneck. If expert 3 receives half the traffic while expert 11 is idle, latency and device utilization suffer even though the math on paper looks fine.

This is the crucial inference trade-off. MoE reduces active compute but not total model footprint. All experts must still be stored, paged, or otherwise made available. Serving an MoE model is therefore partly a memory placement problem. You save arithmetic per token, but you may create new complexity in routing, expert locality, and all-to-all communication if experts span devices.

void gated_ffn(float* y, const float* x,
               const Matrix W_gate,
               const Matrix W_up,
               const Matrix W_down) {
    float gate[D_FF];
    float up[D_FF];
    float mix[D_FF];

    matvec(gate, W_gate, x);   // [d_ff]
    matvec(up,   W_up,   x);   // [d_ff]

    for (int i = 0; i < D_FF; ++i) {
        float s = gate[i] / (1.0f + expf(-gate[i])); // SiLU
        mix[i] = s * up[i];
    }

    matvec(y, W_down, mix);    // [d_model]
}

The pseudocode hides the true performance story, because production implementations do not literally allocate stack arrays like this for every token. They tile, batch, fuse, and vectorize. But structurally it is right: two wide projections, one elementwise gate, one down projection. If you understand that loop nest, you understand the logic of dense gated FFNs.

Why FFNs are such large compute consumers

For a dense non-gated FFN, the token pays for two large matrix-vector products during decode: up and down. For a gated FFN, it pays for three: gate, up, and down. Each one touches tens of millions of weights in a modern layer. By comparison, the activation function and elementwise multiply are tiny. This asymmetry explains why engineers talk so much about GEMM and GEMV performance when optimizing transformer inference: the projections dominate.

During prefill, these may be matrix-matrix multiplies over many tokens and many batch items, which GPUs love. During single-token decode, they shrink into matrix-vector style workloads, which are more bandwidth-sensitive and often harder to saturate. The same operator family therefore behaves differently depending on serving mode. Prefill is throughput-oriented. Decode is latency-oriented.

That is also why FFN quantization is so attractive. If you can halve or quarter the bytes read for those huge matrices while keeping acceptable quality, the wins multiply across every layer and every generated token. The arithmetic may still be significant, but on many real systems the first enemy is weight bandwidth.

Residual reintegration: why the FFN does not get the last word

After the FFN produces its output, that output is added back to the residual stream rather than replacing it wholesale. This design matters. It means the FFN can contribute a learned correction or enrichment while the old token state still remains available downstream. From a systems perspective, the FFN behaves more like an update function than a complete overwrite. That makes very deep stacks easier to optimize and harder to destabilize.

It also clarifies why the FFN can be extremely expressive without needing to preserve every intermediate feature explicitly. The wide channels are disposable. Their only long-term job is to influence the smaller residual vector that survives to the next layer. The architecture is saying: compute extravagantly in the middle if you want, but compress your final message back into the shared layer-to-layer format.

That residual contract is one reason FFN mistakes can be subtle. A broken FFN might not cause immediate catastrophic outputs if the residual path still carries reasonable information. Instead it may quietly degrade quality, making the model feel vague, repetitive, or less discriminating. The residual path keeps the system alive; the FFN determines how much sharper it becomes.

Hardware behavior: CPU, GPU, FPGA

On CPUs, FFNs are typically dominated by GEMV or small-batch GEMM kernels. Cache blocking, NUMA placement, quantized matmul kernels, and thread scheduling matter enormously. Dense FFNs can run well on modern CPUs, but they are usually bandwidth-sensitive during decode because each token requires touching large weight matrices that do not fit in low-level cache.

On GPUs, FFNs are often the comfortable part of the model because large dense matrix multiplies map naturally to tensor cores and highly tuned libraries. During prefill, the GPU can process wide batches efficiently. During decode, efficiency drops because you have fewer live tokens per step, but the hardware ecosystem is still optimized around this pattern. Quantization support, weight packing, and fused activation kernels can produce major gains.

On FPGAs or custom inference ASICs, FFNs are attractive precisely because the structure is regular. If the weights are streamed or tiled effectively, the datapath is mostly repeated multiply-accumulate work with small elementwise glue around it. The difficult part is not understanding the operator; it is storing or supplying the enormous weight volume at a rate that keeps the compute array busy.

Common optimisations

Kernel fusion is the first obvious optimization. You may fuse biasless linear outputs with activation, fuse gate activation with elementwise multiplication, or pack weight layouts so the up and gate projections are read in a way that matches the device's preferred memory pattern. Production systems also batch tokens across requests during decode to turn more GEMVs into GEMMs, trading some queueing latency for higher device utilization.

Quantization is the second major optimization family. INT8 and INT4 FFNs reduce memory movement and can improve throughput dramatically when kernels are mature. AWQ and GPTQ are popular because they try to preserve accuracy in aggressively quantized weights. The best choice depends on your model family, tolerance for quality loss, and hardware support for low-bit matrix multiplies.

For MoE systems, optimization expands into routing and placement. Keep frequently co-activated experts near each other. Avoid unnecessary cross-device traffic. Consider expert parallelism only if the communication cost does not erase the reduced compute cost. MoE serving is where model architecture and cluster architecture start negotiating directly.

One more practical optimization is serving-policy aware batching. If you can batch several requests so their tokens hit the FFN together, the kernels move from skinny matrix-vector work toward healthier matrix-matrix work. That often improves throughput dramatically, though it introduces queueing trade-offs. A real inference engine is therefore optimizing not just the FFN kernel, but the arrival pattern of work sent into that kernel.

Database and systems analogies

The FFN is a bit like a per-row user-defined function in a database engine, except massively vectorized and learned rather than hand-written. Attention is the join-like phase that lets rows consult other rows in the sequence. The FFN is the heavy row-local transformation that takes the enriched row and derives new columns from it. Every token goes through it, and the engine expects the same schema on the way out as on the way in: [d_model] in, [d_model] out.

The wide hidden layer is like a very large scratchpad of candidate derived features. Most of those features are transient. They exist only long enough to decide how the token should re-enter the residual stream. That is why the FFN can be so computationally heavy even though it leaves behind only one same-sized output vector.

What can be cached

The answer is: almost none of the token-local activations. The persistent state is the weights. The intermediate gate values, up values, activation outputs, and elementwise products are all ephemeral. Once the token leaves the FFN, those arrays can be discarded. This is very different from attention, where keys and values must be preserved across future decoding steps.

You can of course rely on ordinary hardware caches, reuse packed weights across requests, and keep expert weights resident on the device. But there is no semantic cache analogous to the KV cache for dense FFN activations. The same prompt token in a different context produces a different hidden state, so memoizing FFN outputs is generally not useful.

Operation and bound classification

Common implementation mistakes

One classic mistake is getting the matrix orientation wrong. Depending on the framework, the stored weight shape may be [out, in] or [in, out], and blindly multiplying without checking conventions produces silent nonsense. Another is forgetting that the gated form needs two distinct wide projections, not one projection reused twice.

Quantized implementations often fail on scale handling rather than on the matmul itself. If per-channel scales, zero-points, or dequantization order are wrong, the FFN output drifts badly. MoE implementations introduce their own failures: top-k selection bugs, incorrect probability weighting, dropped experts at capacity boundaries, or routing tensors accidentally sorted out of sync with token order.

A less obvious mistake is benchmarking the FFN only in prefill mode and assuming decode will look similar. Many serving problems emerge only under small-batch, single-token latency conditions, where kernel launch overheads, weight bandwidth, and cross-request batching policy suddenly matter much more.

If you were implementing this yourself

Start dense, not clever. Implement Up → Activation → Down first, verify against a reference checkpoint, then extend to gated variants. Print shapes, compare a few output elements, and make sure residual wiring is correct around the block. Once that works, add the second gate projection and confirm that your SwiGLU or GeGLU outputs match a trusted implementation exactly enough for the chosen precision.

Only after correctness should you optimize. Decide whether your target workload is prefill-heavy, decode-heavy, or both. That decision changes almost everything about kernel selection and batching strategy. If you expect low-latency interactive decode, focus on weight layout, quantization, and request batching. If you expect offline batch inference, focus on large GEMM efficiency and expert placement if MoE is involved.

Most importantly, keep the conceptual model straight. The FFN is not sequence mixing. It is per-token nonlinear transformation. If you remember that one sentence, the shapes, costs, optimizations, and failure modes all make more sense.

And if performance work ever becomes confusing, return to the three dominant questions: how many big projections are being executed, how many bytes of weight data must move to execute them, and how effectively the hardware is being kept busy while they run. Most FFN engineering decisions are just refinements of those three questions.