The AI folks don't seem to understand intelligence

Just three years after OpenAI said its models were good at grade-school math, the company announced they had solved the Navier-Stokes Millennium Prize problem in 88 hours, something that had stumped humanity’s greatest mathematical minds since the 1930s. Project that trend forward and you see why OpenAI Chief Executive Officer Sam Altman thinks AI will cure all diseases - and why people are afraid that a rogue AI might harm humanity.

Santa Fe Institute scientists Melanie Mitchell and David Krakauer described AI’s mantra as “Scale is all you need.” What that means is the bigger the training sets and the more computer you have, the better AI becomes. Altman’s vision of AI’s exponential growth curve, and the related hopes and fears, are similarly driven by an unspoken assumption that intelligence is all you need. If you’re smart enough, you can cure cancer - or destroy the world. But that’s not how power and intelligence work.

Anthropic PBC says Claude, its AI model, already writes more than 80% of the code its engineers merge. Push that to 100% and AI would be building its own successors. It could keep racing ahead until finally it understood everything and could control everything. That’s the fear and, for some, the hope. AI could create what Anthropic Chief Executive Officer Dario Amodei calls “a country of geniuses in a datacenter.”

The United States ran that experiment at Los Alamos. It put legendary physicists such as Niels Bohr, Richard Feynman and Robert Oppenheimer to work building the atomic bomb. But Alex Wellerstein, a historian of science and nuclear weapons at the Stevens Institute of Technology, calculated that Los Alamos accounted for just 4% of the Manhattan Project’s almost $2 billion cost. Eighty percent of the budget went to plants in Tennessee and Washington that produced uranium and plutonium. In his 1945 Congressional testimony, Oppenheimer said that without scientists there would have been no bomb, but “if there had been only scientists, there also would be no atomic bomb.”

AI has changed coding, mathematics, and weather forecasting. They all have a vast pool of available training data, rapid feedback, and clear right and wrong answers. These keep models moored to reality, not spiraling off to disaster as the result of compounding small errors.

Constant corrections are why we remember the Wright brothers. The best-funded attempt at flight in 1903 was made by Samuel Langley, secretary of the Smithsonian Institution, who had $50,000 in government funds and an engine four times as powerful as the one in the hands of the Wrights. Langley designed a plane that was inherently stable. But his brilliant design plunged into the Potomac River because the models he built didn’t scale. The Wright brothers were bicycle mechanics. They didn’t have his resources, but they knew that bikes were unstable machines kept upright by constant corrections from the rider. They used that insight, plus wind tunnel tests of as many as 200 wing shapes, to beat their better-funded rival.

Intelligence needs input. The faster and clearer the input, the more powerful intelligence becomes. Math, coding and weather forecasting? Check. Financial markets qualify, too. Hedge fund Two Sigma Investments boasts that its computing power would rank among the world’s top five supercomputer sites, and that isn’t an accident. Writing fails the tests, hence the deluge of AI slop on LinkedIn. So does curing cancer, because it requires experiments on living systems that take years to respond. Intelligence might improve your hit rate, but it can’t get you around that limit.

Intelligence gets used first in the places with the fastest and clearest feedback. What’s left are the problems where it’s worth less. That’s why its power is unlikely to keep climbing. The more you know, the harder it becomes to learn even more. We ran that experiment in theoretical physics. In the early days after the quantum mechanics revolution, progress leapt forward. Over time, despite better instruments and tools, foundational breakthroughs got harder. In a 1975 lecture, the Nobel Prize-winning physicist Paul Dirac said, “It was very easy in those days for any second-rate physicist to do first-rate work. There has not been such a glorious time since then. It is very difficult now for a first-rate physicist to do second-rate work.”

Research-heavy industries show the same pattern. In semiconductors, Stanford University economist Nicholas Bloom and three co-authors found that doubling chip density takes more than 18 times as many researchers now as in the early 1970s. Even Tom Davidson and Tom Houlden of the research group Forethought, whose model gives roughly 60% odds that fully automated AI research will compress three years of progress into one, assume the returns fall as software approaches its limits.

So, when Amodei predicts that powerful AI could compress 50 to 100 years of biological progress into five to 10, and that reductions of 95% or more in cancer deaths and new cases seem possible, skepticism is appropriate. The key question, one AI leader wrote, is how much progress each additional increment of intelligence buys, and which other inputs become the constraint when intelligence is plentiful. That leader was Amodei himself, in his 2024 essay “Machines of Loving Grace.” In the same essay, he noted that today’s particle physicists have no shortage of theories but lack the accelerator data to choose between them, and it’s not clear superintelligence would change that. Other Anthropic researchers noted that Amdahl’s Law dictates that speeding up one part of a process tends to move bottlenecks, not eliminate them. They even forecast that this would apply to a world dominated by self-improving AI.

That’s why we all need to take a breath before plunging into the utopian and apocalyptic forecasts about AI, and to watch the bottlenecks instead. An advantage in thinking isn’t an army, a power grid, or a supply chain, and it can’t build them overnight. Doing that will take even more time and investment than the roughly $760 billion in capital spending Google parent Alphabet Inc., Amazon.com Inc., Meta Platforms Inc. and Microsoft Corp. alone expect this year. Altman’s curve is real in mathematics. Maybe not far beyond it. Transforming coding is important. It’s not the first sign of a cybernetic God.

Gautam Mukunda writes about corporate management and innovation. He teaches leadership at the Yale School of Management and is the author of “Indispensable: When Leaders Really Matter.”