Determinism, Seeds & Sampling

Seeds: The Starting Point of Randomness

Both Transformers and diffusion models use random numbers during generation. A seed is the starting value for the random number generator (RNG). Same seed → same sequence of random numbers → same output (in theory).

Source 1: Sampling — Intentional Randomness

Sampling is the deliberate source of variation. The model produces probabilities; sampling decides what to do with them.

Transformer Sampling

After softmax, you have a probability distribution over the vocabulary. Several strategies exist:

Greedy (temperature = 0) — always pick the highest-probability token. Deterministic. Same input = same output. No randomness at all.
Temperature sampling — divide the logits by temperature before softmax. Low temperature (0.1–0.5) → sharpens the distribution → more predictable. High temperature (1.0–2.0) → flattens it → more creative/random.
Top-k — only consider the top k most likely tokens, zero out the rest, then sample. Prevents the model picking wildly unlikely tokens.
Top-p (nucleus) — only consider the smallest set of tokens whose probabilities sum to p (e.g. 0.9). Adapts the candidate pool size based on confidence.

Temperature literally controls collapse sharpness. At 0, the wavefunction collapses deterministically to the peak. At high values, it explores the full field of possibilities.

Diffusion Sampling

Each denoising step adds a small amount of fresh Gaussian noise (except the final step). This keeps the process stochastic — the model doesn't just converge to the mathematically closest clean image, it explores the space of plausible outputs.

DDPM (original) — full stochastic schedule. Each step adds fresh noise. More variation, slower (1000 steps).
DDIM — deterministic variant. No added noise between steps. Same seed = exactly same image. Much faster (20–50 steps).
Euler / DPM-Solver — treat denoising as solving a differential equation. Faster convergence, tuneable stochasticity.

The scheduler is the collapse trajectory. DDPM explores more of the possibility space. DDIM takes the most direct path. Same destination, different routes through the noise.

Source 2: Floating-Point Non-Determinism — Accidental Randomness

Even with the same seed, same prompt, and greedy/deterministic sampling, you can still get different outputs. This is the one that surprises people.

The maths is deterministic. The hardware isn't.

Floating-point numbers (float16, float32) can't represent most real numbers exactly. They round. And the order in which you do additions affects the rounding:

This matters because GPUs are massively parallel — thousands of threads adding numbers together simultaneously. The order those threads finish is non-deterministic. Different execution order → different rounding → different results.

Different GPU model — different FP units, different rounding behaviour
Different GPU count — different parallelisation splits, different addition order
Different driver version — different kernel implementations, different thread scheduling
Same GPU, same driver — can still vary if the CUDA runtime doesn't enforce deterministic kernels
float16 vs float32 — fewer precision bits → larger rounding errors → bigger divergence

The difference starts tiny — maybe the 7th decimal place. But in an autoregressive model, one slightly different token changes every subsequent token. In diffusion, one slightly different denoising step shifts the trajectory through noise space. Small rounding error → amplified through thousands of sequential operations → completely different output.

The Two Sources, Summarised

True determinism requires: same seed + same model weights + same precision + same hardware + same driver + deterministic CUDA kernels + same batch size. In practice, most systems settle for "reproducible enough" — same seed gives visually similar results, not bit-identical ones.