This repository demonstrates how floating-point non-associativity and reduction order can introduce subtle non-determinism in common LLM operations such as RMSNorm, matrix multiplication, and attention. The Jupyter notebook visualizes the effect of “split reductions,” simulating fused GPU kernels and showing how tiny rounding differences propagate through network layers.
.
├── Examples.ipynb # Jupyter notebook with simulations and heatmaps
├── pyproject.toml # Project and dependency configuration for UV
├── uv.lock # UV lockfile for reproducible environment
└── README.md # This file
- Clone the repository:
git clone https://github.com/psmgeelen/DeterministicLLMs.git
cd DeterministicLLMs- Install dependencies using UV:
uv syncUV reads
pyproject.tomlanduv.lockto install all required packages automatically.
- Launch Jupyter:
jupyter notebook-
Open
Examples.ipynband run the cells to simulate RMSNorm, Matmul, and Attention with both normal and split reductions. -
Observe the heatmaps, which visualize tiny differences caused by rounding and reduction order changes.
- Floating-point arithmetic is non-associative, so the order of operations affects results.
- Even deterministic kernels can produce different outputs if batch size, sequence length, or kernel implementation varies.
- Tiny differences caused by precision and rounding propagate through network layers, causing observable non-determinism.
- Controlling reduction strategies is critical for reproducibility, benchmarking, and deterministic inference.
This repository is released under the UNLICENSE – do whatever you want.