Why Computers Aren't Deterministic

The idea that computers are deterministic is one of the most persistent and dangerous myths in engineering. A single CPU executing a single integer instruction is deterministic. Everything built on top of that — is not.

Floating Point: The Numbers Lie

Floating-point numbers approximate real numbers using a fixed number of bits. Most real numbers can't be represented exactly — they get rounded to the nearest representable value.

This is not a bug. This is the specification. But it means:

(a + b) + c ≠ a + (b + c) — addition is not associative. Rearranging the order of operations changes the result.
Different compilers — may reorder FP operations for performance. -ffast-math in GCC explicitly permits this. Same source code, different binary, different results.
Different hardware — x87 uses 80-bit internal precision. SSE uses 64-bit. ARM uses 64-bit with different rounding modes. Same algorithm, different CPU, different answer.
FMA (fused multiply-add) — a*b+c computed with one rounding step instead of two. Whether the compiler uses FMA changes the result at the last bit.

Every neural network, every physics simulation, every financial calculation is built on floating point. None of them are bit-for-bit reproducible across hardware without extreme care.

Thread Scheduling: The OS Decides

Your program has four threads. The OS decides which one runs on which core, for how long, and when to interrupt it. This decision depends on:

What else is running on the machine
Thermal throttling state
Power management decisions
Interrupt arrival timing
Memory pressure and page fault handling

Two threads writing to the same variable without a lock: the result depends on which thread's write lands last. This changes every run. Every microsecond of timing difference in the OS scheduler produces a different interleaving of instructions, and a potentially different outcome.

The word "sometimes" is the signature of non-determinism. The bug only appears under specific timing. It passes every test, then fails in production. This is a heisenbug — observing it (adding logging, running a debugger) changes the timing and makes it disappear.

Clocks: Time Is a Lie

Every call to "get current time" returns a different value. Any logic that depends on it is non-deterministic by definition.

Timeouts are non-deterministic — they depend on system load.
Cache TTLs are non-deterministic — expiry timing varies.
Rate limiters are non-deterministic — request timing varies.
Pre-emption pauses are non-deterministic — OS scheduler decides, not you.
Hypervisor steal time is non-deterministic — you can't even detect it from inside the VM without checking /proc/stat.
NTP corrections are non-deterministic — clock discipline happens when the daemon decides.
Any test that uses sleep() or Date.now() is a non-deterministic test.

Memory: The Invisible State

The program sees virtual addresses. The CPU sees physical pages, cache lines, TLB entries, and prefetch queues. None of this is visible to your code, but it affects behaviour:

Cache hits vs misses — same algorithm, 100× speed difference depending on what was accessed recently
ASLR — Address Space Layout Randomization. Memory addresses differ every execution. Pointer values differ. Any code that depends on pointer ordering is non-deterministic
Page faults — first access to a memory page triggers a kernel trap. When this happens depends on access patterns and OS paging decisions
NUMA topology — which CPU socket owns which memory bank. Cross-socket access is slower. Thread placement affects performance non-deterministically

GPUs: Massively Parallel Non-Determinism

A GPU runs thousands of threads simultaneously. The order in which they complete is non-deterministic. For integer work, this doesn't matter — addition is commutative and associative. For floating point, it does:

1024 threads each compute a partial sum. The reduction tree adds them in pairs. Which pairs complete first varies → addition order varies → rounding varies → result varies.

atomicAdd on float — the order of accumulation is hardware-dependent. Same kernel, same data, different result each launch.

This is why the same neural network, with the same weights, same input, and same seed, can produce different outputs on different GPUs — or even on the same GPU on different runs. The model's forward pass involves millions of floating-point reductions, each with non-deterministic accumulation order.

I/O: The Outside World

The moment your program reads from disk, network, stdin, a sensor, or anything external, determinism is gone.

Disk reads — latency varies with physical head position (HDD), wear levelling (SSD), OS caching, and concurrent I/O from other processes
Network — latency, packet loss, reordering, retransmission. Two identical requests take different times. Responses arrive in different orders
User input — a human is the ultimate non-deterministic system. Keystroke timing, mouse position, decision-making
Sensors — temperature, accelerometer, GPS. Physical world measurements have noise, drift, and sampling jitter

Languages and Runtimes

Many non-determinism sources are baked into the tools you use:

What You Can Actually Do About It

Fix the seed — control PRNG state. Necessary but not sufficient.
Use deterministic kernels — CUDA has torch.use_deterministic_algorithms(True). Slower, but reproducible. Mostly.
Avoid FP accumulation order dependency — use Kahan summation, or sort before reducing
Lock everything — serialise concurrent access. Kills performance, guarantees ordering
Use logical clocks — Lamport timestamps, vector clocks. Don't depend on wall time
Make I/O a seam — inject deterministic test doubles for disk, network, time, and randomness
Accept it — design for non-determinism. Idempotent operations, CRDTs, eventual consistency, property-based tests instead of exact equality

The Truth

Determinism is the collapsed state. Non-determinism is the superposition. Most systems exist in superposition most of the time — we just pretend they've collapsed because the output usually looks the same.