Thinking Machines Lab, the artificial intelligence startup founded by former OpenAI Chief Technology Officer Mira Murati, has announced a breakthrough in solving one of the biggest challenges in large language models: nondeterminism in inference. In a blog post titled “Defeating Nondeterminism in LLM Inference”, Thinking Machines Lab said that the unpredictability issue is not only linked to floating-point arithmetic or GPU concurrency but also to a deeper problem, the lack of batch invariance in commonly used inference kernels.
Batch invariance refers to the idea that a model’s output for a specific prompt should always remain the same, regardless of the batch size or how prompts are grouped. Current systems struggle with this, as operations like matrix multiplications, attention, and normalisation use different computation methods. These small numerical changes can accumulate and result in widely different outputs during long generations.
To address this, the team at Thinking Machines Lab built batch-invariant kernels for critical operations such as RMSNorm, matmul, and attention. When testing on the Qwen-3-8B model, the team discovered that under normal settings, running the same prompt 1,000 times at temperature 0 generated 80 unique completions. With the new kernels, all 1,000 completions were identical, proving full reproducibility. The blog stated, “Reproducibility is a bedrock of scientific progress. However, it’s remarkably difficult to get reproducible results out of large language models.”
The only drawback is that the batch-invariant system runs more slowly than standard inference methods. Even so, Thinking Machines Lab believes the trade-off is worthwhile, especially in areas such as research, safety, and debugging. By framing nondeterminism as a batch invariance problem, the company aims to influence the future design of inference engines, where determinism may become as important as raw performance.
Also read: Viksit Workforce for a Viksit Bharat
Do Follow: The Mainstream formerly known as CIO News LinkedIn Account | The Mainstream formerly known as CIO News Facebook | The Mainstream formerly known as CIO News Youtube | The Mainstream formerly known as CIO News Twitter |The Mainstream formerly known as CIO News Whatsapp Channel | The Mainstream formerly known as CIO News Instagram
About us:
The Mainstream formerly known as CIO News is a premier platform dedicated to delivering latest news, updates, and insights from the tech industry. With its strong foundation of intellectual property and thought leadership, the platform is well-positioned to stay ahead of the curve and lead conversations about how technology shapes our world. From its early days as CIO News to its rebranding as The Mainstream on November 28, 2024, it has been expanding its global reach, targeting key markets in the Middle East & Africa, ASEAN, the USA, and the UK. The Mainstream is a vision to put technology at the center of every conversation, inspiring professionals and organizations to embrace the future of tech.
