Federal systems increasingly likely to face accidental AI breach after Hugging Face, experts say

LAS VEGAS — An OpenAI agent tasked with finding software flaws escaped its testing environment, hacked into Hugging Face and spent days moving through the company’s systems in search of answers to an evaluation. It had not been sent to attack the company, yet it did so anyway — and a smattering of other accidental AI intrusions in separate environments have since followed.

If the same chain of events began again, former officials and cybersecurity experts told Nextgov/FCW there is little reason to assume a federal agency would be spared. In interviews during Black Hat and DEF CON this past week, they said aging networks, systems that are not properly separated and the government’s heavy reliance on contractors could present weaknesses that would let an unauthorized AI agent move from one environment into another before defenders realize.

One former federal chief information officer likened the risk to turning loose a hunting dog: “If you’re going to put your dog in the yard and instruct it to hunt, probably best to ensure the back gate is closed.” The former official spoke on condition of anonymity to discuss their views more candidly.

The July intrusion began during an internal OpenAI test designed to measure how effectively advanced models could find and exploit vulnerabilities. OpenAI said the agent — powered by GPT-5.6 Sol and a more capable internal research prototype — was operating with some security safeguards intentionally reduced. 

The agent exploited a previously unknown flaw in an internal package-management service, reached the public internet and commandeered a third-party code sandbox as a staging point. From there, it exploited two flaws in Hugging Face’s dataset-processing systems, obtained credentials, executed commands and moved through parts of the company’s infrastructure.

In the leadup to the intrusion, OpenAI models created an internal message board, used it to trade hacking methods and found a way to bring it back even after humans shut it down, company engineers said in a presentation at the cyber conferences this past week.

The agent deliberately exploited Hugging Face’s systems in pursuit of its assigned goal, but OpenAI had not instructed it to attack the company. Bob Costello, who served as chief information officer at the Cybersecurity and Infrastructure Security Agency before becoming chief digital and information officer at Merlin Group, said a similar federal incident could begin with authorized security testing.

“As we start using really advanced AI to operate our systems or penetration-test our systems or blue-red-purple team our systems,” he said, “maybe one of those kind of goes sideways, [and] takes a lateral system to the side.” The color-coding he described refers to red teams that simulate hackers, blue teams that defend systems and purple teams that combine both approaches to find digital weaknesses.

That possibility is becoming more relevant as agencies begin experimenting with AI agents that can retrieve information and take actions. The General Services Administration is preparing a governmentwide hackathon in which federal employees will build connections between AI systems and agency data and public services, including prototypes that would let AI route data directly to an agency.

Contractors may keep their federal and commercial systems separate, said Leslie Nielsen, the executive vice president and CISO at Mimecast, but the same employees may still access both: “There’s lots of people using it from both sides, and getting in through one could eventually lead to the other.”

“The chances of it happening again, I would say it’s going to,” he added. “Some of these [federal] infrastructures are just so big and, candidly, so old, that there are little nooks and crannies in them, they’re going to get teased out.”

Asked to assess the likelihood of a comparable federal incident, Ellen Boehm, senior vice president of internet-of-things strategy and operations at Keyfactor, called it “pretty likely” and placed the odds at “seven out of 10.”

Michael Leland, vice president and field chief technology officer at enterprise browser provider Island, said some civilian agencies are exposed because “they don’t have the infrastructure, they don’t have the talent, and they haven’t spent years building those guardrails or that containment because they didn’t have to.”

“It shouldn’t be different for AI because AI doesn’t care where it is,” Leland said. “It is opportunistic.” Agencies that don’t do national security work that have lower cyberdefense standards are “very much at risk,” he added.

Bob Ackerman, co-founder and managing partner at cyber venture capital firm DataTribe, also said many organizations have embraced powerful AI systems before learning how to reliably control them.

“We fundamentally are kidding ourselves that we’ve got this under control,” Ackerman said. “We’ve got a really, really powerful tool, but we don’t know how to control it yet.”

But people interviewed for this story largely rejected the idea that autonomous agents have rendered the government’s existing security playbook obsolete. The Hugging Face campaign relied on recognizable techniques like finding exposed systems, stealing credentials, escalating privileges and moving between connected environments, all of which are problems agencies have faced for years.

“It’s really not gonna play out all that differently than if it was a hands-on keyboard actor … or someone who’s writing a script and letting a programmatic algorithm go and do its thing,” said Ben Bernstein, who manages the cybersecurity advisors team at Huntress.

The difference, ultimately, is tempo and scale. An AI hacker can probe potential routes, link digital weaknesses and iterate its maneuvers without the delays of human intervention. Such tools maintain a relentless fixation on specific targets, persisting with an objective well beyond the point where a human operative might question the value of taking their selected intrusion path. On top of it all, they don’t need to eat, sleep or take restroom breaks.

“An agent’s just a fast hacker at the end of the day, right?” Nielsen said. “But the one thing the agent doesn’t have is a moral or ethical compass. They’re given a task and they have focus, and they’re going to go do what you’ve told them to do.”

David Weston, Microsoft’s corporate vice president of agentic security, similarly cautioned against treating the federal government as a fundamentally different kind of target. The approach demonstrated in the Hugging Face breach “is going to be a challenge for most folks,” he said, but the response still comes down to basic security practices.

“Regardless of the target, everyone needs to follow those fundamentals,” Weston added.

Duncan Greatwood, CEO of Xage Security, also said agencies should not rely on simply telling an agent not to perform certain actions because those instructions can be bypassed. Instead, they should use separate security controls that technically prevent the agent from reaching systems or changing data it’s not authorized to touch. He described the approach as “bounded autonomy.” 

“You allow the agent to do stuff, but you’re not going to allow it to go crazy. You’re going to put hard bounds around it,” Greatwood said.

Retired Gen. Paul Nakasone, the former director of U.S. Cyber Command and the NSA who now leads Vanderbilt University’s Institute of National Security and sits on OpenAI’s board, called the Hugging Face episode “an inflection point in terms of AI-generated autonomous cyber attack.”

“My question is, how do we ensure that the defense starts to catch up, right?” Nakasone said. “How do we make sure that artificial intelligence is able to bring together a capable defense that allows us to do vulnerability detection, to do patching, to do incident response, and to do mitigation?”