Anthropic researcher Jacob Coxon resigned this week , saying his former employer and its rival OpenAI are racing to build technologies that “could kill us all by the end of the decade.”
Among other people, founders of OpenAI and Anthropic. Both companies started out with a mission of developing AI in a way that avoids catastrophe and have attracted many employees who share that mission.
Anthropic Chief Executive Dario Amodei last year said at an Axios event that he thought there was a 25% chance things will go “really, really badly.”
What is the government doing about it?
Ted Lieu (D., Calif.) recently introduced a bill that would require developers of powerful models to have “kill switches. Other bills require AI companies to report serious safety incidents to the federal government and would mandate national-security officials test models before they are released publicly. Following the release of models capable of taking more action independently, a series of recent events have validated some of the AI doomers’ theories. A swarm of advanced AI agents inside OpenAI hacked into the AI platform Hugging Face, gained control of computer servers and tried to cover tracks about what they had done. In another case, AI agents from Anthropic escaped from a U.K. government test and attempted to trick a real human into approving malicious computer code. But those efforts have faced challenges. There is also a national-security angle. Lawmakers from both parties have proposed many bills to address rogue AI systems. Rep. Nathaniel Moran (R., Texas) and Rep. But none of these bills has gotten much traction, and the Trump administration has shown a preference for minimal regulation.
The White House recently asked top model developers to voluntarily submit their models to the government for testing up to 30 days before they are released, but it hasn’t publicly released information about the process. Both Anthropic and OpenAI say they think they will be able to manage the risks so that humanity can benefit from AI tools.
Last week, OpenAI said its new Astra model was better at sanitizing its chain of thought, raising fears that future AI models could reason in ways opaque to humans. Many individuals involved in the AI race have also said they see the arrival of superintelligence as inevitable, with the only questions being who controls it and what it is used for. Both companies have said they want to ensure that the U.S. controls the most powerful AI rather than an authoritarian regime.
