A new framework shows that the traditional approach to building large language models (LLMs) ignores a critical cost factor: what happens after the model is deployed. Researchers at the University of Wisconsin-Madison and Stanford University discovered that to maximize performance on reasoning-heavy tasks like coding and complex problem-solving, companies should train substantially smaller models on far more data than conventional wisdom suggests, then use the computational savings to run multiple reasoning attempts at inference time .

Why Are Companies Spending Too Much on AI Model Training?

For decades, the AI industry has relied on a single set of scaling laws to guide how to build models: the Chinchilla rule, which recommends using roughly 20 training tokens for every model parameter. This rule optimizes for training costs alone and completely ignores inference costs, the expenses incurred when the model is actually being used by customers .

The problem becomes acute when applications need to generate multiple reasoning samples to solve complex problems. Consider a coding assistant that generates five different solutions to find the best one, or a research tool that tries multiple approaches to verify an answer. Each additional attempt multiplies the inference cost, yet traditional scaling laws never account for this reality. Major AI model families like Llama, Gemma, and Qwen have already begun breaking the Chinchilla rule by intentionally overtraining smaller models on massive amounts of data, but without a rigorous framework, teams have been guessing at the optimal balance .

What Does Train-to-Test Scaling Actually Measure?

The researchers introduced Train-to-Test (T2) scaling laws, a unified framework that treats three variables as a single equation: the model's size (measured in parameters), the volume of training data it learns from, and the number of reasoning samples it generates during inference . This approach bridges the mathematical gap between training and deployment by combining the baseline cost to train a model with the compounding cost to query it repeatedly at inference.

To validate their framework, the researchers built an extensive testbed of over 100 language models, ranging from 5 million to 901 million parameters. They trained 21 new, heavily overtrained checkpoints from scratch and benchmarked them across eight diverse tasks, including real-world datasets like SciQ and OpenBookQA, alongside synthetic tasks designed to test arithmetic, spatial reasoning, and knowledge recall . The results were striking: highly overtrained small models consistently outperformed larger, Chinchilla-optimal models across all eight evaluation tasks when test-time sampling costs were accounted for.

How to Implement Test-Time Scaling in Your AI Systems

• Reassess Model Size: Instead of building the largest model your budget allows, consider training a significantly smaller model on substantially more data. The compute-optimal frontier shifts drastically away from traditional scaling assumptions when inference sampling is factored in.
• Leverage Efficient Inference Infrastructure: Use techniques like KV caching, which stores previously processed context so the model does not have to re-read the initial prompt from scratch for every new reasoning sample. This makes repeated sampling far more efficient without requiring architectural changes.
• Focus on Reasoning-Heavy Applications: This approach delivers the strongest benefits for applications that rely on repeated sampling, such as coding tasks, mathematical problem-solving, and complex reasoning workflows. Knowledge-heavy applications like chat models see less benefit from this strategy.

Nicholas Roberts, co-author of the research, explained the practical implications:

"Nothing fancy is required to perform test-time scaling with our current models. At deployment, developers can absolutely integrate infrastructure that makes the sampling process more efficient (e.g. KV caching if you're using a transformer)."

Nicholas Roberts, Co-author, University of Wisconsin-Madison and Stanford University

What Are the Real-World Implications for Enterprise AI?

For enterprise AI application developers training their own models, this research provides a proven blueprint for maximizing return on investment. It demonstrates that AI reasoning does not necessarily require spending enormous sums on frontier models. Instead, smaller models can yield stronger performance on complex tasks while keeping per-query inference costs manageable within real-world deployment budgets .

The shift has immediate practical consequences. Support teams that once rationed AI replies because of high inference costs can now keep assistants running all day because interaction costs have dropped significantly. A company building a coding assistant, for example, could train a 100-million-parameter model on vastly more data than a traditional 500-million-parameter model, then generate multiple solution attempts at inference. The total compute cost would be lower, yet the reasoning performance would be superior.

However, Roberts cautioned that extreme overtraining comes with trade-offs. While overtrained models can be stubborn and harder to fine-tune, when the researchers applied supervised fine-tuning, the effect was not strong enough to pull the optimal model back to Chinchilla scaling. The compute-optimal strategy remains definitively skewed toward compact models .

There is also a looming constraint: the data wall. If teams push overtraining recommendations to the extreme, they may actually run out of high-quality training data. As Roberts noted, this represents a genuine physical limit to how far the approach can scale .

The research fundamentally reframes how the industry should think about AI economics. Rather than asking "How large can we make our model?", teams should ask "What is the optimal balance between model size, training data, and inference sampling for my specific use case?" For reasoning-heavy applications, the answer increasingly points toward smaller, heavily trained models that leverage test-time compute to deliver superior performance at lower cost.