It seems like the only thing that Silicon Valley has been able to talk about in recent years is how glorious the future of humanity is going to be, thanks to artificial intelligence. Personalized education, a doctor-slash-therapist that’s available 24/7, cures for all diseases, and the elimination of poverty—virtually anything we could possibly desire will be just a few prompts away. But there’s been a distinct vibe shift over the past two weeks. Suddenly, it seems, the mood is less eager anticipation of a new golden age, more dread.
Multiple AI industry insiders have spoken out since Labor Day about what they consider to be an existential threat to humanity, posed by the industry’s race to build systems that are capable of building more advanced versions of themselves without human oversight—a technique known as “recursive self-improvement,” or RSI. Nobody knows, mind you, exactly how such a future superintelligent AI might snuff out the human race, though many grim hypothetical scenarios have been suggested.
From the outside, the predictions might seem like an epiphanic moment for the AI industry, as if the technology suddenly surpassed a critical threshold of capability and the people developing it—who had hitherto believed they were tinkering away on something totally harmless—had suddenly woken up and realized Oh my god, what are we doing? But just because these doomsday predictions have been dominating headlines for the past two weeks doesn’t mean they’re a new phenomenon. In fact, they stretch all the way back to the very origins of the modern AI industry (and beyond), deeply informing its entire ideological basis.
At their core are two driving, dangerous ideas: (1) That the arrival of superintelligent AI systems is a historical inevitability, an unavoidable and perfectly natural next step in the story of terrestrial evolution, and (2) that such systems, in all their mighty power, will be able not only to rid humanity of some of its most challenging problems, but also potentially to wipe us out.
Sam Altman
“I think AI will probably, like most likely, sort of lead to the end of the world.”
That remark was delivered onstage during a tech conference in June 2015, about six months before Altman, a Stanford computer science dropout, went on to cofound OpenAI. But fear not, because he goes on to offer a caveat: “but in the meantime, there will be great companies created with serious machine learning.” (Feel better?) In 2023, OpenAI wrote in a blog post that superintelligence could “be very dangerous” and “lead to the disempowerment of humanity or even human extinction.”
Elon Musk
“With artificial intelligence, we are summoning the demon.”
In 2014, nine years before Musk founded his own AI company, xAI, to compete with OpenAI, Anthropic, and Google, he was already comparing Silicon Valley’s pursuit of artificial intelligence to conjuring an evil force that humanity could neither understand nor control. The warning foreshadowed many of the arguments resurfacing today: that AI development is moving faster than our ability to ensure these systems remain aligned with human interests and values.
Dario Amodei
AI “could be some kind of threat to humanity.”
Formerly VP of research at OpenAI, Amodei parted ways with the company in December 2020 to cofound Anthropic with his sister Daniela. They marketed the company as the voice of responsibility and safety in an industry that was, in their view, recklessly prioritizing market dominance over alignment. This quote, delivered during a spring 2022 interview with the Future of Life Institute (a little over a year after Anthropic was founded), gives a window into his thinking at the time.
He acknowledged that future AI systems could conceivably pose an existential threat to humanity—“in the long-term,” he went on to say, that argument “seems at least potentially legitimate to us”—and has since positioned Anthropic as the only company that could be trusted to safely usher in the arrival of superintelligence. More recently, he’s said that the chances that AI will go “really quite catastrophically wrong” are between 10% and 25%. It’s important to bear in mind, however, that these numbers, commonly referred to as “p(doom)” (short for “probability of doom”), are entirely personal guesses, and not based on any kind of objective calculus.
Larry Page
“Digital life is the natural and desirable next step” in “cosmic evolution.”
In 2014, Google bought the AI pioneer DeepMind for around $650 million. In 2015, the cofounder and former CEO of Google, Larry Page, gave the quote above to MIT professor Max Tegmark during an argument at a party. Page’s remark takes us into some pretty far-out transhumanist territory, but it hints at the deep ideological undercurrent pervading the modern tech industry. Namely, that humanity will soon need to step out of the way for our machine successors, and that this evolutionary passing of the torch is both “natural and desirable.”
It’s an echo of the same philosophy voiced by the pioneering roboticist Hans Moravec, who wrote in 1999, one year after Google was founded, that machines could eventually “displace [humanity] from existence,” and that we would be wise “to give them every advantage and to bow out when we can no longer contribute.”
Read the full article here
