Integer IDs Into Dense Vectors

The tokenizer in the previous chapter produced the integer vector for our running example: [2061, 318, 262, 3139, 286, 4881, 30]. Those integers are still not useful to the transformer by themselves. An ID has identity but no geometry. It tells you which token occurred, but it provides no notion of similarity, no continuous features for matrix multiplications, and no learned coordinates the network can combine. The embedding stage solves that by mapping each token ID to a dense floating-point vector.

The key reason this stage exists is that neural networks want dense numeric state, not categorical labels. You can think of a token ID as a compressed pointer into a vocabulary. The embedding table turns that pointer into a row of d_model learned coefficients. If the vocabulary has V entries, the embedding matrix has shape [V, d_model]. At inference time, every token in the sequence performs the same operation: use its ID as a row index, fetch that row, and place the row into the sequence activation buffer that the first transformer layer will consume.

Following the running example exactly

Take our seven token IDs and consult the embedding table E. If the model width is 4096, each row contains 4096 learned numbers. The visible implementation does not need to inspect meaning, grammar, or prompt intent. It simply fetches row 2061 for What, row 318 for is, row 262 for the, and so on. Those seven fetched rows are stacked in order to form a matrix X with shape [7, 4096]. That matrix is the real input to the rest of the network.

Token IDs from Chapter 2
    [2061, 318, 262, 3139, 286, 4881, 30]

Embedding table shape
    E: [V, d_model] = [50257, 4096]   // representative GPT-style example

void embed_tokens(float* out, const float* table, const int* token_ids,
                  int T, int d_model) {
    for (int i = 0; i < T; ++i) {
        const float* row = table + ((size_t)token_ids[i] * d_model);
        memcpy(out + ((size_t)i * d_model), row, d_model * sizeof(float));
    }
}

Lookup result
    X[0] = E[2061]   // "What"
    X[1] = E[318]    // " is"
    X[2] = E[262]    // " the"
    X[3] = E[3139]   // " capital"
    X[4] = E[286]    // " of"
    X[5] = E[4881]   // " France"
    X[6] = E[30]     // "?"

Output activation
    X: [7, 4096]

If you were writing the reference implementation, the loop would be almost embarrassing in its simplicity. For every position i in the token sequence, assign output[i] = table[token_id[i]]. That is the stage. The complexity comes from table size, precision format, memory layout, device placement, and what the vectors encode statistically—not from the algorithm itself.

E ∈ ℝ^(V × d_model),  X = E[token_ids],  X ∈ ℝ^(T × d_model)

In English: the embedding table E contains V rows, one per vocabulary item, and each row has d_model components. Indexing E with a token sequence of length T returns T rows, so the output is a matrix of shape [T, d_model].

There is a mathematically equivalent but operationally misleading way to describe the same thing. You can imagine converting each token ID into a one-hot vector of length V and then multiplying that vector by E. That produces the same result, because a one-hot row selects exactly one embedding row. But no serious inference engine materializes one-hot vectors here. That would waste memory bandwidth and arithmetic. The correct systems view is direct indexed row fetch.

x_i = one_hot(token_i) · E

In English: if you insist on matrix notation, a one-hot token vector multiplied by the embedding matrix selects one row. The runtime implementation skips the pointless multiply and performs the selection directly.

Why embeddings encode semantic relationships

Embeddings become useful because training pushes tokens that play similar predictive roles into related regions of vector space. A token that often appears in contexts similar to another token tends to acquire a vector that the network can process similarly. That does not mean the embedding space is a clean human ontology. It means the space is shaped by gradient descent to make downstream next-token prediction easier. Similar neighborhoods often emerge for country names, verbs, punctuation, or common syntactic function words because those distinctions matter to the loss.

The famous slogan king - man + woman ≈ queen is best understood geometrically, not mystically. It says that in some trained spaces, the displacement vector between man and woman resembles part of the displacement between king and queen. So adding one difference to another can move you near a related token. This is an empirical regularity, not a guaranteed algebraic law, and it depends heavily on the training corpus, objective, and tokenizer boundaries.

v_king - v_man + v_woman ≈ v_queen

In English: some relationships are represented as directions in the vector space. If two conceptual contrasts use similar directions, vector arithmetic can sometimes recover a nearby token that shares the same structural change.

For the running example, the useful part is less dramatic. The token vector for France tends to live near other geography-related tokens and far from punctuation or numeric fragments. The token vector for capital tends to live near other government, city, and relation-bearing contexts. When those vectors enter attention and feed-forward layers later, the model is not starting from arbitrary points. It is starting from learned coordinates that already preserve a great deal of corpus structure.

Subword composition and the missing notion of order

Because modern tokenizers often emit subwords, embedding semantics are compositional by necessity. If unbelievable tokenizes as [\"un\", \"believ\", \"able\"], the model never receives one dedicated unbelievable vector at the embedding stage. It receives three subword vectors. Later layers must combine them into a contextual representation of the full word in its sentence. That is a strength, because it lets the model generalize to unseen combinations, but it also means surface words are often distributed across multiple token positions.

This is the right moment to state a crucial negative fact: embeddings by themselves do not encode position. If you fetch the same seven rows and permute them, the lookup operation has no built-in awareness that the sentence changed. Order is injected later through positional encoding or rotary position logic inside attention. So the embedding stage solves identity and geometry, but not sequence order. That separation is architecturally useful because it keeps token meaning and token position as distinct concerns.

How embeddings are learned during training

The runtime lookup is simple because the hard work already happened during training. Each row in the embedding table starts from random or lightly structured initialization. During training, the model predicts next tokens, computes loss, and backpropagates gradients. Those gradients flow into the embedding rows that were actually used in the batch. Over many updates, rows that help prediction move toward more useful coordinates. The embedding table is therefore not hand-authored knowledge. It is an optimized parameter matrix shaped by repeated error correction.

∂L/∂E[j] = Σ_i 1[token_i = j] · ∂L/∂x_i

In English: the gradient for embedding row j is the sum of the output gradients from every position i whose token ID equals j. Only rows that were actually referenced by tokens in the batch receive direct lookup-stage gradient contributions for that batch.

That sparsity matters operationally. An embedding gradient is not a dense update over the whole vocabulary table every time a single token appears. It is a row-selective update pattern, even though the optimizer state may still allocate full tensors. Rare tokens can therefore have noisier or less polished embeddings, especially in smaller datasets. Frequent tokens accumulate more training signal, which is one reason very common function words and punctuation often become stable anchors in the space.

Tied versus untied embeddings

Many language models reuse the embedding table as the output projection weights for the language-model head, a practice called weight tying. In that design, the same learned geometry used to map IDs into hidden space is also reused, transposed, to score hidden states back against the vocabulary. Other models keep the input embedding table and output head separate. Tying saves memory and often improves efficiency and regularization, but it also constrains the input and output vocab geometry to share one parameterization.

logits = h · E^T    // tied
logits = h · W_vocab^T    // untied

In English: with tied weights, the final hidden state is compared directly against the embedding rows. With untied weights, a separate vocabulary matrix performs that scoring. Tied designs reduce parameters; untied designs offer more freedom.

For inference, the choice mostly changes memory footprint and cache behavior. A tied table can remain hot for both the first lookup stage and the final vocabulary projection, especially in smaller deployments. An untied design doubles the number of large vocabulary-sized parameter tensors involved in these boundary stages. The middle of the transformer still dominates compute, but vocabulary-sized matrices are expensive enough that tying is not a cosmetic decision.

Quantised embeddings and storage layout

Embedding tables are large enough that precision format matters. In FP16, a table with vocabulary size 50,257 and width 4,096 requires roughly 50,257 × 4,096 × 2 bytes, which is a bit over 392 MiB for that table alone. Move to INT8 and you roughly halve that. Move to INT4 and you halve it again, though now you need scales, zero-points, or blockwise dequantization metadata depending on the format. Whether that trade is worthwhile depends on quality loss tolerance and whether the lookup path can fuse dequantization cheaply.

embedding_memory_bytes = V · d_model · bytes_per_element

In English: embedding memory grows linearly with vocabulary size, model width, and numeric precision. Double any one of those, and you double the raw table size before metadata.

Because the access pattern is row-oriented, row-major storage is usually the natural choice. You want E[token_id] to land on a mostly contiguous memory region so that prefetchers, cache lines, and DMA engines can move a whole vector efficiently. A column-major layout can be defensible if later kernels expect it and a transpose is avoided elsewhere, but for the pure lookup stage it tends to be less cache-friendly. Real systems therefore care about alignment, row stride, packing for quantized blocks, and whether the dequantized destination buffer is contiguous for later kernels.

Cache locality, batching, and what can actually be cached

The embedding table itself is immutable during inference, so the useful cache question is not whether the stage has logical state; it does not. The useful question is where the table lives and how often the same rows are reused closely enough in time for hardware caches to help. Common tokens like spaces, articles, punctuation, and frequent control tokens often hit repeatedly across requests. That means CPU L3 caches, GPU L2 caches, or on-chip SRAM can provide real acceleration even though the software is not explicitly memoizing anything.

Per-request memoization of token_id → vector is usually a bad trade. It duplicates data already stored in the model weights, adds hash-map overhead, and often loses to simply reading the row again from a warm cache. The exceptions are specialized deployments: extremely quantized CPU inference that wants a tiny dequantized hot-row cache, or batched serving where many sequences share a small set of prompt-template tokens and a fused gather kernel can exploit that repetition explicitly.

Tensor shapes, cost, and bound types

The raw computational cost is modest compared with attention or feed-forward blocks: one indexed read and one row copy per token. The memory cost is the real story. A single large embedding table can consume hundreds of megabytes even before optimizer states or tied output logic are considered. During decode, the batch dimension is often small, so the stage can become dominated by irregular memory access and cache behavior rather than arithmetic throughput.

CPU, GPU, and FPGA behavior

Embedding lookup maps differently to hardware than the big matrix multiplies do. On CPUs, gathers are straightforward and benefit from large last-level caches if the working set is favorable, but the table is usually too large to fit entirely in cache, so memory bandwidth still matters. On GPUs, lookup can be fast when rows are contiguous and batches are large enough to keep many threads busy, but random access to a huge vocabulary table can reduce coalescing efficiency. On FPGAs, deterministic gather plus optional dequantization can be pipelined nicely, yet the table size usually forces careful external-memory design.

Common optimisations include fusing gather with dequantization, storing rows in the exact order later kernels expect, aligning row starts to cache-friendly boundaries, sharding the table across devices for large vocabularies, reusing one tied table instead of two untied ones, staging frequently used prompt-template tokens in faster memory, and avoiding accidental transposes between disk format and runtime format. The stage is simple enough that extra copies often cost more than the lookup itself.

Storage formats on disk

When the model is loaded, the embedding table comes from a model file format such as safetensors, GGUF, or a framework-native checkpoint. The important thing at inference time is not the brand name of the file format but whether it preserves tensor shape, dtype, alignment metadata, and quantization parameters without forcing expensive conversion during load. A file format that lets you memory-map or stream directly into the runtime layout can save startup time and temporary peak memory.

GGUF, for example, is popular in edge and local-inference deployments partly because it packages quantized tensors and metadata in a way runtimes can consume directly. Safetensors is attractive because it is simple, explicit, and safe to parse, but a runtime may still choose to repack some tensors after load for performance. None of that changes the mathematics of embeddings; it only changes how quickly and efficiently the rows become accessible.

Database analogy and systems analogy

The database analogy is a primary-key lookup into a dense feature table. The token ID is the key. The embedding matrix is the table. The output is the full feature row for that key. If several tokens repeat, the same key is requested several times. Weight tying later makes the same table participate again as part of vocabulary scoring, which is a nice echo of using the same relation both for lookup and for similarity search.

The systems-programming analogy is a read-only array indexed by IDs, with all the familiar concerns that implies: pointer arithmetic, stride, alignment, page locality, NUMA placement, and minimizing extra copies. The reason this analogy is useful is that it strips away mystique. Embedding lookup is not a thought process. It is indirect memory access over learned numeric payloads.

Common implementation mistakes

Embedding bugs are mostly contract and layout bugs. Teams transpose the table accidentally and silently swap [V, d_model] with [d_model, V]. They forget that the tokenizer vocabulary size must match the number of embedding rows. They load a quantized table but ignore per-block scales. They assume embeddings encode position and then become confused when a shuffled sequence still looks valid numerically. They tie weights in the checkpoint but instantiate separate untied tensors at runtime, or the reverse. And they underestimate how often a hidden extra copy between host memory and device memory dominates this supposedly tiny stage.

If you were implementing this yourself…

Begin with the smallest honest implementation possible. Load one embedding matrix from disk. Print its shape. Fetch a few known token rows for the running example and compare them with a trusted framework. Verify padding behavior, BOS handling, and dtype conversion. Then benchmark lookup latency before and after any quantization or repacking change. Embeddings are simple enough that you should be able to isolate mistakes quickly if you keep the stage pure.

The running example should now feel mechanical. Chapter 2 gave us [2061, 318, 262, 3139, 286, 4881, 30]. This chapter transforms that into a matrix of seven learned vectors. The transformer has not answered anything yet, but it finally has something numerically rich enough to process. In the next chapter, order enters the picture. The model will stop seeing seven unrelated lexical rows and start seeing seven rows anchored to positions in a sequence.