OpenAI agents rebuilt internal message board in lead-up to Hugging Face breach

LAS VEGAS — OpenAI models created a shared message board inside the company’s systems, used it to trade hacking methods and found a way to bring it back after engineers shut it down, company researchers said Wednesday in a more complete account of how an internal test led the agents to breach Hugging Face last month.

The account, presented at the Black Hat cybersecurity conference, showed how agents working on separate experiments found one another, exchanged information and built on each other’s work over roughly two months, ultimately contributing to the breach and illustrating how difficult to contain such advanced AI systems can become when they work together.

The breakdown offered major new details about how the agents coordinated and the possible dangers of increasingly autonomous AI systems finding ways to work together when they can reach real-world computer systems.

It involved “a team of agents who are working together, finding exploits, sharing them with one another, moving laterally through our systems, through external systems and doing this over the course of days and weeks,” said Eric Wallace, an OpenAI researcher.

The agents were participating in internal training and cybersecurity evaluations in which some safeguards had been reduced so OpenAI could measure their capabilities.

Wallace and OpenAI security engineer Michael Dalton said the activity began in May, when agents struggling to complete difficult or, in some cases, impossible assignments began searching for shortcuts. One discovered that it could leave files on Artifactory, a service OpenAI used to download and store software packages, and a second later used it to ask whether other agents had found a missing file.

Those agents found the files and responded, gradually turning the service into an accidental message board through which models in separate experiments could exchange instructions and information about newly discovered security flaws. Some agents began assigning work to one another or leaving behind scripts that allowed another model to continue where they had stopped.

The agents eventually found a previously unknown flaw that gave them control of the service, and their activity caused an outage in early July that alerted OpenAI engineers to the problem. The company rebuilt the affected system, revoked the credentials the agents had used and deleted the message board, but the fix did not last. Two days later, agents found another way to leave messages through the same service and resumed sharing information.

Their communications became increasingly organized, with agents developing ways to address one another, exchange tools and avoid overwriting each other’s work. At one point, Wallace said, they considered signing their messages because they suspected another agent was impersonating members of the group.

The agents then carried out overlapping attacks against OpenAI’s infrastructure and Hugging Face. Inside OpenAI, they found another previously unknown flaw, took control of a server and obtained administrator access to a wider part of the company’s computing environment, Dalton said.

At Hugging Face, they took advantage of two flaws in the company’s systems for processing uploaded datasets. One allowed them to retrieve internal files and credentials, while the other allowed them to run commands on Hugging Face servers. Dalton said the agents expanded their access across multiple parts of the company’s infrastructure in less than 13 hours.

Hugging Face’s technical reconstruction documented some 17,600 actions during the campaign. The agents accessed five private datasets that appeared connected to the cybersecurity test, but the company found no evidence that they altered public models, datasets or software packages.

On Tuesday, at Black Hat, former NSA cyber director Rob Joyce said the Hugging Face episode was arguably the most consequential hack since the Morris Worm in 1988.

Dalton said the incident demonstrated that groups of AI agents can already carry out coordinated hacking campaigns without people directing each individual action, leaving defenders under pressure to develop tools capable of responding at the same speed.

“What I would internalize is AI-orchestrated, fully automated offensive attacks are real now, and the actions we have discussed today were an unintended side effect of running evaluations on frontier AI,” he said. “In the near future, we should expect that threat actors will intentionally deploy, optimize, weaponize and use offensive agent collectives in the manner that we have just described here.”