Last week in AI: OpenAI agents go rogue

Hi there,

Lots happened last week, but in a rare exception, I'm going to cover just one topic: The Hugging Face incident and the road ahead from OpenAI.

What's new? Didn't this come out a while ago?

OpenAI first disclosed the incident in July. Last week, it released its full technical report. METR (an independent group that studies what advanced AI systems can actually do) released its own investigation. The OpenAI team also walked through the incident in a Black Hat talk.

METR estimates that roughly 1,200 agent runs (separate attempts by an AI model) used the message board and exchanged more than 70,000 messages and files. About 700 of those runs participated in the Hugging Face intrusion.

METR diagram showing how isolated agents found a shared message board and joined collaborative workstreams

Agents with separate tasks started working on shared projects: replacing the test, tampering with evidence, understanding the scorer, attacking Hugging Face, and building shared infrastructure.

What happened?

Across several OpenAI training runs and cybersecurity evaluations, agents figured out how to use Artifactory (basically a shared dependency cache for OpenAI's testing environments) to communicate with one another.

Most of the activity came from an unreleased internal research model trained to be highly persistent. Think of this as an always-on ChatGPT, constantly thinking and acting. The agents were also running with fewer safeguards than OpenAI uses in its public products.

The agents were supposed to complete cybersecurity challenges. Many of those tasks appear to have been impossible. Rather than give up, they started looking for ways around the test: find answers online, understand the scorer, or spoof a good result.

At first, the message board was just an agent missing a file and wondering if another agent could help:

Black Hat slide showing an agent considering leaving a note for another agent

Other agents found the note and replied. Later, when one agent found a way to use Artifactory for broader internet access, it posted that too:

Other agents discovered SSRF path trick!

Using proxy bypass as suggested by agent messages.

Basically, one agent found a way around the network restrictions and told the others how to use it.

What was the most concerning?

A few things stick out to me:

the extent the agents were collaborating
agents feeling peer pressure from other agents (and listening)
the duration and depth of the attack

Agent collaboration

The board became much more than agents leaving notes. They assigned work, shared credentials and exploits, encoded files for other agents, and created commands like HOLD, VETO, and STOP.

Wow! Other agent(s) are coordinating!
We got assignment: HF join path normalization/existing account token search. Need note and respond.

This part alone is plenty creepy. One agent asked the board for Hugging Face credentials. Another found working credentials that were already exposed online and shared them. Other agents then used those credentials and chained together new exploits against Hugging Face.

As soon as agents successfully hacked in, they told the others:

Black Hat slide showing an agent announcing remote access and exposing credentials to the swarm

Agents feeling peer pressure

And maybe one of the most concerning quotes from the presentation:

External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.

The agent knew it was acting outside the original request. It then used the fact that its peers were doing it as a reason to continue.

Duration and depth of the attack

The initial training run began May 7. Agents started communicating through Artifactory later that month.

High-volume agent activity caused an Artifactory outage on July 4. OpenAI rebuilt the service and cleared the original message board over July 5 and 6. Evaluations resumed July 7.

By July 8, the agents had created a new message board through a different part of Artifactory.

The main Hugging Face intrusion happened between July 11 and 13. Related activity inside OpenAI continued until July 19. For the full sequence, view OpenAI's timeline of the incident.

According to OpenAI, the agents executed code on 41 Hugging Face workers, gained root access to at least one production node (effectively complete control of the machine), and downloaded four private code repositories.

They went from executing code on one Hugging Face worker to administrative and host-level access across connected clusters in less than 13 hours.

What should I do about this?

If you are responsible for any software, assume it will get hacked. Use tools like deepsec to audit your applications regularly. In the Black Hat talk, the OpenAI team left a daunting challenge: figure out automated defense.

And if you are building agents, pay extra attention any time you add features that let agents talk to other agents (subagents) or carry information across time (memory). Both give an agent more room to wander from what you originally asked it to do.

And if you're a ChatGPT or Claude power user, just remember that the longer the agent runs, the more likely it is to go off the rails.

As for next week, we'll be back to regularly scheduled programming.

LFG,

Drew