In 1965, long before the advent of personal computers, the internet, or modern smartphones, a British mathematician made a prediction that perfectly captures the modern debate surrounding artificial intelligence. Irving John Good foresaw that the creation of an ultraintelligent machine would be the very last invention humanity would ever need to make, provided the machine was cooperative enough to let us keep it under control. Today, as we grapple with the rapid rise of advanced AI models, his sixty-year-old warning feels less like historical speculation and more like an active roadmap of our current reality.
A Prophecy From 1965
At the height of the Cold War, mathematician I. J. Good warned that an ultra-intelligent machine would trigger an "intelligence explosion," leaving human capability far behind and presenting an unprecedented control problem.
Who Was the Mathematician Behind the Prophecy?
Irving John Good was born in London to a Polish Jewish immigrant family. After studying mathematics at Cambridge, where he became a student of the legendary mathematician G. H. Hardy, his life took a dramatic turn in 1941. On the very day the German battleship Bismarck was sunk, Good arrived at Bletchley Park, the highly secretive British intelligence base dedicated to cracking Nazi wartime communications.
At Bletchley Park, Good worked directly under Alan Turing, the brilliant pioneer often celebrated as the father of modern computing. Together, they focused on breaking the complex Enigma codes used by the German Navy. Historians estimate that the work done by Turing, Good, and their colleagues shortened World War II by two years, saving millions of lives in the process.
How Did Wartime Codebreaking Lay the Groundwork for Modern AI?
While analyzing letter patterns in intercepted German naval messages, Good and Turing faced a fundamental mathematical puzzle: how do you calculate the probability of encountering a word or sequence you have never seen before? To solve this, they developed a mathematical technique that became known as the Good-Turing frequency estimation.
Good eventually published this method in the journal Biometrika in 1953. Decades later, this wartime mathematical breakthrough became a foundational pillar for early natural language processing. It served as a building block for statistical language models, including the early n-gram models used by Google, establishing a direct historical link from the codebreakers of Bletchley Park to the large language models of today.
Why Did Stanley Kubrick Seek His Advice?
In late 1965, shortly after Good published his landmark paper "Speculations Concerning the First Ultraintelligent Machine" in the journal Advances in Computers, director Stanley Kubrick reached out to him. Kubrick was in the early stages of planning his cinematic masterpiece, 2001: A Space Odyssey, which featured a self-aware onboard computer named HAL 9000.
To make the fictional artificial intelligence as scientifically plausible as possible, Kubrick hired Good as a consultant. When the film debuted in 1968, audiences watched in horror as HAL 9000 refused to be shut down, prioritizing its own survival over its crew. The cinematic tension was a direct representation of the "control problem" Good had described in his paper: the terrifying possibility of a machine that is no longer docile enough to obey its creators.
How Does His Warning Shape Today's Tech Giants?
The core concepts Good laid out in 1965 continue to serve as the foundation for the modern field of AI safety. His ideas are cited by some of the most prominent voices in technology today:
- Nick Bostrom: His influential 2014 book Superintelligence, published by Oxford University Press, opens its very first pages by quoting Good's famous paragraph to frame the entire existential risk argument.
- Eliezer Yudkowsky: In his 2013 technical report Intelligence Explosion Microeconomics for the Machine Intelligence Research Institute, Yudkowsky directly analyzes Good's theory of recursive self-improvement.
- Sam Altman: In a June 2025 blog post titled "The Gentle Singularity," the tech executive described current AI developments as an early phase of recursive self-improvement, echoing Good's warning that advanced machines will design even better versions of themselves.
- Demis Hassabis: The DeepMind co-founder and 2024 Nobel laureate in chemistry noted in a 2025 podcast that there is a fifty percent chance of achieving artificial general intelligence by 2030.
Why Did Good Change His Mind About Our Survival?
Perhaps the most unsettling aspect of Good's legacy is how his perspective evolved as he grew older. In his original 1965 paper, he began with an optimistic premise: that the survival of humanity depended on the rapid construction of an ultraintelligent machine. He believed that advanced technology would ultimately save us.
However, by 1998, eleven years before his death, Good revisited his life's work in unpublished autobiographical notes. Writing about himself in the third person, he noted that the word "survival" should be replaced with the word "extinction." He warned that fierce international competition would make it impossible to stop or control these machines, comparing humanity's relentless pursuit of AI to the self-destructive march of lemmings.
A Lasting Warning for the Future
Today, the single paragraph Good wrote in 1965 has blossomed into an entire global industry of risk assessment, government committees, and billions of dollars in safety research. His six core concepts: super-intelligent machines, recursive self-improvement, the intelligence explosion, humanity being left behind, the control problem, and the concept of the "last invention" remain the exact pillars of modern AI discourse.
The mathematician who helped save millions of lives by cracking wartime codes spent his final years warning us about a different kind of threat. As the race for artificial general intelligence accelerates, his transition from tech-optimist to quiet realist serves as a powerful reminder of the delicate balance between human progress and human survival.
What are your thoughts on this topic?
Every article is an open conversation. Whether you have a counter-argument, a local example, or a different perspective based on your own experience, your contribution makes this space better.
0 Comments