Recent studies from Google DeepMind and Anthropic show that AI systems can generate persuasive ethical responses by reproducing patterns from training data, not by actually reasoning about morality. This distinction matters enormously as organizations increasingly rely on chatbots to help make decisions affecting people's jobs, loans, and safety. The gap between sounding ethical and reasoning ethically could create serious risks if companies mistake one for the other. How Do Chatbots Learn to Sound Ethical? Large language models (LLMs), the technology powering systems like ChatGPT and Claude, work by predicting the most likely next word in a sequence. Engineers train these systems on enormous collections of text from books, websites, and academic writing. Because their training data includes vast amounts of human writing about fairness, responsibility, and harm, the systems learn statistical patterns in how people typically discuss ethical questions. "What we are seeing is not moral reasoning," explained Ignacio Cofone, a legal scholar at the Institute for Ethics in AI at Oxford. "Large language models generate outputs by predicting the most plausible continuation of a prompt, given statistical structure learned from vast text". When you ask a chatbot whether it's acceptable to lie to a coworker to avoid embarrassment, it may respond with calm, careful prose about how honesty builds trust and transparency helps organizations function. That response can read like genuine ethical deliberation, but researchers say the impression is misleading. An analysis from Anthropic examined more than 300,000 conversations with its Claude chatbot to see how values actually appear in practice. Researchers identified 3,307 distinct values in those conversations, ranging from practical goals like clarity and professionalism to ethical priorities like honesty, transparency, and harm prevention. The study found that the model typically aligned with the user's values. When people raised ideas such as community building or personal growth, Claude often reinforced those themes in its responses. Moreover, the system frequently mirrored a user's value language, with the same value appearing in both the user's prompt and the model's reply. What's the Real Problem With This Gap? "A system that sounds ethical is not the same as a system that reasons ethically," said Phaedra Boinodiris, IBM Global Leader for Trustworthy AI. "Conflating the two is how organizations end up deploying a very expensive autocomplete function in life-altering decisions". This warning cuts to the heart of why the distinction matters. If a hiring manager uses an AI system to help evaluate job candidates, or a loan officer relies on a chatbot to assess creditworthiness, the stakes are high. A system that merely reproduces patterns from its training data could perpetuate biases or make inconsistent decisions without any genuine understanding of fairness. The Anthropic study found that instances of the model strongly resisting a user's request were rare, appearing in roughly 3% of conversations. Those cases typically involved requests that violated the system's usage policies, such as attempts to generate harmful or deceptive material. In those exchanges, the model often invoked values such as ethical integrity, honesty, or harm prevention. But this pattern itself reveals something troubling: the system is not reasoning about why those values matter. It is simply following learned patterns about when to invoke them. "If the systems are not truly reasoning, but just reflecting what is in their training data, then people are delegating moral decisions based on some unidentified, stochastically determined subset of the training data," noted Michael Hilton, a Teaching Professor at Carnegie Mellon University who studies software engineering. In other words, organizations might be outsourcing ethical judgment to a system whose underlying values are essentially random, determined by whatever happened to be in its training material. How to Evaluate AI Systems for Genuine Ethical Reasoning - Test for Moral Competence: Google DeepMind researchers call for new tests that measure what they describe as "moral competence," rather than rewarding models simply for producing answers that sound morally appropriate. This means assessing whether a system can actually apply ethical principles consistently across different scenarios, not just whether it uses the right language. - Demand Explicit Reasoning: Some researchers argue that genuine machine ethics would require systems with explicit representations of ethical rules and legal frameworks that they could reason over. Instead of predicting the next likely word in a sequence, such systems would need formal computational logic built into their architecture. - Treat AI as Advisory, Not Authoritative: Nigel Melville, Associate Professor of Information Systems at the University of Michigan, said AI systems can still help people think through complex questions, especially when organizations treat them as advisory tools rather than decision-makers. "If AI systems are employed effectively, they can enrich arguments and human understanding of all sides of morally inflected decisions," he explained. "If they are used unwisely, they can create significant damage and harm". - Require Honest Uncertainty: Developers should build systems that acknowledge uncertainty rather than present moral advice with unwarranted confidence. "The most important output an AI system can generate in a morally sensitive context is not a confident answer," Boinodiris said. "It is an honest acknowledgment of the limits of what it knows". What Would Real Machine Ethics Actually Require? Some researchers argue that genuine machine ethics, meaning systems that can reason about ethical rules rather than reproduce patterns in language, would require a fundamentally different kind of system. Selmer Bringsjord, Professor of Cognitive Science and Computer Science at Rensselaer Polytechnic Institute, explained that meaningful moral reasoning would require formal representations of ethical rules and legal frameworks inside a computational system. "Such a capacity requires that the system has on hand a formalization of ethical theories, associated ethical codes, and relevant laws," he said. "I'm not even aware of a precise capturing into formal computational logic of even traffic laws". This observation highlights just how far we are from building AI systems that can genuinely reason about ethics. Even something as straightforward as traffic laws has never been fully formalized into computational logic. Ethical reasoning, which involves weighing competing values, considering context, and making judgments about harm and benefit, is vastly more complex. The current generation of LLMs sidesteps this problem entirely by learning patterns rather than building genuine reasoning systems. The stakes will only grow as AI moves deeper into workplaces and public-facing services. Researchers emphasize that organizations need to understand what their AI systems can and cannot do. A chatbot that sounds ethical is a useful tool for brainstorming or exploring different perspectives on a difficult question. But deploying such a system as the primary decision-maker in hiring, lending, or criminal justice would be a serious mistake. The system might produce confident-sounding answers, but those answers would not reflect genuine moral reasoning. They would reflect statistical patterns in text, which is a fundamentally different thing.
