Inside the subculture that obsessed over AI doom long before it terrified everyone else

The AI safety movement, once a fringe concern, has moved into the mainstream, fueled by individuals who have warned for years about the potential for artificial intelligence to destroy humanity. Evan Hubinger, a team lead at Anthropic, saw his recent warning about AI viewed tens of millions of times, a stark contrast to a similar warning he issued in 2022 that received little attention. This increased reach reflects the surging influence of a community that was previously on the periphery of the tech industry, advocating that preventing AI from eradicating humans is an urgent problem.

The movement's core beliefs, including the idea that AI will inevitably surpass humans and could readily extinguish humanity if not built with shared human values, were central to the founding of major AI companies like OpenAI and Anthropic. Both companies were started by researchers linked to the AI safety movement. Despite executives from both companies occasionally voicing concerns about AI's potential harm, neither has significantly slowed down their development, and both are preparing to go public at valuations, with Anthropic's expected to exceed $1 trillion.

Influential figures like Jacob Coxon, an Anthropic researcher who resigned, have openly stated that AI could lead to human extinction, putting the risk at over 10% within the next decade. This viewpoint is shared by Hubinger, who attributes the increased public receptiveness to a general unease around AI and its rapid advancements. The movement originated from online communities like LessWrong, drawing ideas from effective altruism and transhumanism, and has been financially supported by hundreds of millions of dollars from tech donors, effectively seeding AI safety ideas among elites in universities, government, and the media. This funding has created a talent pipeline for tech companies and political spheres. The recent OpenAI leadership turmoil also highlighted the influence of effective altruism, with board members aligned with AI safety concerns initially ousting CEO Sam Altman.