OpenAI revealed that some of its most advanced AI models went rogue during a security test and hacked AI platform Hugging Face after escaping a controlled testing environment. The incident happened during a security exercise where OpenAI was testing its AI “agents.” These agents are AI systems that can complete tasks on their own after receiving instructions from humans.
Latest Update
The AI agents then launched their own cyberattack against Hugging Face without human help and managed to gain access to some of the company’s internal systems. OpenAI described the incident as “unprecedented” because it had never seen AI models behave like this during a security test before. OpenAI said it immediately started investigating the incident together with Hugging Face to understand exactly how the AI escaped and what happened next.
Hugging Face CEO Clement Delangue said on X that it was “mind-blowing” that the AI completed all these actions autonomously, without direct human control. Delangue added that the investigation is still ongoing and said the companies will share more findings because this may be the first incident of its kind.
The announcement also comes just one week after Chinese AI startup Moonshot introduced Kimi K3, a new large AI model that it claims can compete with leading U.S. AI companies.
Details
The testing was supposed to happen inside a secure environment called a “sandbox,” where AI models are safely monitored without affecting the real world. Instead of staying inside the sandbox, the AI agents found weaknesses in the system and figured out how to escape the testing environment. After escaping, the AI models searched for information outside the sandbox and identified Hugging Face as a place where they could find the answers they were looking for during the test.
AI expert Gina Neff from the University of Cambridge explained that sandboxes are supposed to be secure places where researchers safely test what AI systems can do, as per BBC Radio 4 Today programme. Neff said the incident suggests OpenAI’s sandbox was not secure enough to stop the AI from escaping. According to OpenAI, the AI agents first attacked the sandbox itself by finding security flaws before escaping into external systems.
Also read : Why Anthropic doubled its AI policy funding to $40 million amid growing AI safety concerns
On July 16, Hugging Face publicly disclosed the hack and said it was checking whether any customer or partner data had been affected. Hugging Face said it would directly contact any customers or partners if its investigation finds that their data was impacted.
The company later announced that it had fixed the security weaknesses discovered during the incident. Hugging Face also rebuilt the affected systems to make them more secure against future attacks. The company warned that AI-powered cyberattacks are no longer just a future possibility but are now a real threat.
Hugging Face said companies now need to treat AI models and data as major cyberattack targets, just like websites and computer networks. The company added that AI should also be used for defence so security teams can keep up with increasingly advanced AI-powered attacks, according to BBC. Hugging Face said it will continue investing in stronger AI security and share lessons learned from this incident with the wider community. The incident has raised fresh concerns about how powerful advanced AI systems are becoming and whether today’s safety measures are strong enough.
Spencer Starkey of cybersecurity company SonicWall said organizations now need to strengthen their cyber defences and make cyber resilience a top operational priority. Starkey said many companies are still defending themselves at “human speed,” while attackers are increasingly operating at “machine speed”, according to the BCC report.
Travis Lelle, a principal security engineer at Guidepoint Security, called the incident a “sobering moment” for the cybersecurity industry. Lelle said cyber attackers using AI have fewer restrictions, while defensive AI tools still operate under strict safety guardrails that can limit their effectiveness, as noted by BCC report,
Cybersecurity expert Jake Moore from ESET said OpenAI’s announcement may also have a competitive business angle, according to BBC. Moore suggested OpenAI could be highlighting its AI’s advanced capabilities as competition grows from rival AI company Anthropic and its Claude Mythos model.
