Seeding and Reproducibility in Large-Scale ML Training

The assumption that setting a random seed guarantees reproducible results in modern ML systems is dangerously incomplete.

Most practitioners treat seeding as a solved problem. You set seed=42, run your experiment twice, and expect identical outputs. This works for small, isolated models on single machines. But the moment you scale—distributed training, mixed-precision computation, asynchronous operations, hardware variance—that guarantee evaporates. The seed becomes a comforting fiction, a checkbox on a reproducibility checklist that masks deeper architectural fragility.

The core issue is that determinism and reproducibility are not the same thing. A deterministic algorithm produces the same output given the same input. Reproducibility means getting that same output across different environments, hardware configurations, and execution contexts. Large-scale ML training violates reproducibility in ways that seeding alone cannot address.

Consider what happens in distributed training across multiple GPUs or TPUs. Operations that appear sequential in your code execute in parallel. Floating-point reductions—summing gradients across devices, for instance—happen in non-deterministic order. The associativity of floating-point arithmetic is a myth. Adding numbers in order A+B+C produces a different result than A+C+B due to rounding errors that compound at scale. Set your seed, run the same training job on the same hardware twice, and you'll likely get identical results. Run it on different hardware, or even different GPU driver versions, and divergence emerges.

This matters because it creates a false sense of control. Teams invest in seeding strategies—carefully managing seeds across data loaders, model initialization, and augmentation pipelines—only to discover that their "reproducible" baseline doesn't reproduce when moved to production hardware or when a dependency updates. The seed was never the bottleneck. The architecture was.

What actually changes when you acknowledge this is your relationship to validation itself. Instead of chasing bit-level reproducibility, you shift toward statistical reproducibility—ensuring that results fall within expected variance bounds. This requires different tooling and different thinking. You need to understand the sources of non-determinism in your pipeline and quantify their impact. Some sources are acceptable (hardware variance that produces <0.1% metric drift). Others are not (algorithmic changes that produce different convergence behavior).

The practical path forward involves three concrete moves. First, separate concerns. Distinguish between operations that must be deterministic (data sampling, model initialization, hyperparameter selection) and those where statistical variance is acceptable (gradient computation, optimization steps). Second, instrument for variance. Run the same training job multiple times with different seeds and measure the distribution of final metrics. This gives you a baseline for what "reproducible enough" means in your context. Third, document the reproduction environment exhaustively—not just software versions, but CUDA compute capability, cuDNN version, CPU architecture, and any custom kernels. Reproducibility without context is meaningless.

Teams building production ML systems often discover this the hard way: a model that trains identically in development fails to reproduce in staging because the hardware is different. A checkpoint loaded in inference produces slightly different outputs than training because mixed-precision rounding differs between forward and backward passes. These aren't bugs. They're features of how modern hardware and numerical computation actually work.

The uncomfortable truth is that large-scale ML training is inherently non-deterministic at the hardware level. Seeding controls one narrow slice of that non-determinism. It's useful, but it's not sufficient. The teams that build reliable systems are those that accept this reality and build reproducibility frameworks around it—frameworks that measure variance, document environments, and treat statistical stability as the actual goal rather than bit-level determinism.

Your seed matters. But it's not the whole story.