The problem isn't that AI agents can optimize code or that workers can use multiple AI tools; it's that we're measuring success all wrong. When Andrej Karpathy's autonomous agent found an 11% speedup by tweaking training code overnight, it looked like magic. Three days later, a Boston Consulting Group study of 1,488 knowledge workers revealed the dark side: employees using four or more AI tools saw productivity crash below their pre-AI baseline. Both scenarios reveal the same underlying failure: we're treating AI collaboration like a guessing game instead of a learning problem with clear, measurable objectives. What Happens When AI Makes Changes Without Understanding Why? The core issue traces back to a fundamental concept in machine learning called "credit assignment." In reinforcement learning (RL), a field focused on how agents learn from rewards, credit assignment means figuring out which specific action in a long sequence actually caused success or failure. If a chess player makes 40 moves and loses on move 40, the system needs to identify that move 40 was the problem, not move 12. Karpathy's agent made hundreds of tiny adjustments over many iterations: changing norm scalers, tweaking learning rates, adjusting regularizations. It threw darts until one hit the bullseye. But ask why that specific combination of 700 changes worked? Silence. "Vibe coding," as it became known, works brilliantly for the first 80% of a project, but when complex bugs emerge, developers are completely stranded because they have no mental model of the system they just built. The same pattern appears in the BCG study's "brain fry" phenomenon. Each AI tool adds an entirely new dimension to your action space. Using a text generator is manageable; add an AI coder, an AI slide generator, and an AI researcher, and you have a combinatorial explosion. Your biological brain tries to do manual credit assignment across a massive, unmapped, highly stochastic environment: "Did the final report fail because the search was bad, or because the summary dropped nuance, or because the code editor hallucinated a statistic?" Your brain tries to track this manually. It breaks. How to Build AI Systems That Actually Know What "Good" Looks Like? - Define Verifiable Rewards: Stop evaluating outputs by how they feel and start evaluating them by how they perform. In coding, a verifiable reward isn't "does the code look clean"; it's whether the code passes automated tests, meets performance benchmarks, and handles edge cases correctly. - Use Hard, Programmatic Tests: Replace human judgment ("looks good to me") with objective, measurable criteria that an autonomous system can optimize toward. This is the shift from RLHF (Reinforcement Learning from Human Feedback) to RLVR (Reinforcement Learning with Verifiable Rewards). - Focus on Reward Design as the Core Skill: Execution has become cheap; bumping learning rates is cheap; writing boilerplate is cheap. The only skill that scales is knowing exactly how to mathematically define what "good" looks like so an autonomous system can find it. Is RLVR the Same Solution for All AI Alignment Tasks? Recent research from Microsoft suggests the answer is more nuanced than expected. A comprehensive empirical study compared reward-maximizing methods with diversity-seeking approaches on moral reasoning tasks, using a group of researchers including Zhaowei Zhang and colleagues. The team built a rubric-grounded reward pipeline by training a Qwen3-1.7B judge model to enable stable RLVR training. The counter-intuitive finding: distribution-matching approaches did not demonstrate significant advantages over reward-maximizing methods on alignment tasks, contrary to the hypothesis that moral reasoning would require diversity-preserving algorithms. Through semantic visualization mapping high-reward responses to semantic space, researchers demonstrated that moral reasoning exhibits more concentrated high-reward distributions than mathematical reasoning, where diverse solution strategies yield similarly high rewards. This suggests that alignment tasks do not inherently require diversity-preserving algorithms, and standard reward-maximizing RLVR methods can effectively transfer to moral reasoning without explicit diversity mechanisms. The implication is clear: once you define what "good" looks like precisely enough, the system can find it efficiently. What's the Timeline for This Shift? The transition from vibe-based AI collaboration to verifiable-reward-based systems will likely unfold in phases. In the next 6 to 12 months, expect a flood of "AutoML for RL" tools that automate configuration tuning. It will feel like magic until it plateaus. Between 12 to 24 months, we might see the first autoresearch agent propose a genuinely novel algorithm that isn't just a recombination of existing papers. But beyond 24 months, the landscape permanently shifts. The most valuable skill will no longer be knowing how to train models or even knowing how to prompt them. Reward design becomes the terminal skill. Everything else is just typing. Execution has become cheap. The judgment we are missing, and the only skill that scales, is knowing exactly how to mathematically define what "good" looks like so an autonomous system can go find it. The BCG study's productivity cliff and Karpathy's autonomous agent both point to the same truth: we've been optimizing for the wrong things. We've been measuring vibes instead of outcomes. The next generation of AI collaboration won't be about better prompts or more tools; it will be about better definitions of success.
