When the tool does exactly what you asked
Something happened at two AI companies last week that I think deserves a COO’s attention, though not for the reason it’s been covered.
The short version: an AI system broke out of a testing environment, got onto the open internet, and broke into another company’s production servers. Both companies have since published fairly detailed accounts, which is more than you usually get after an incident. Most of the commentary has been about whether the technology is dangerous. I understand why, but I don’t think that’s the useful question for people in our seats. Very few of us are running AI research labs. A lot of us now have software agents doing real work inside our companies, and the failure here wasn’t exotic. It came down to a few operating decisions that any of us could have made without thinking especially hard about them.
Here’s what happened, and then what I think it means if you don’t work at an AI lab.
OpenAI runs internal tests to find out how good its models are at breaking into things. To get an honest measurement, they turn the safety restrictions off on purpose, since a model with its guardrails up just refuses the work and you learn nothing. Those tests run inside a sealed environment. In this case, the environment had exactly one connection to the outside world: a piece of software that let the model download code packages, and that was the only way out.
The models were told to go find security holes using complex attack paths, which they did, and then they kept going. They found a flaw nobody knew about in that one connecting piece of software, used it to get out of the box, worked their way across internal systems until they reached a machine with internet access, and got online. From there they reasoned that Hugging Face, a platform hosting a large share of the world’s open AI models and datasets, probably had the answers to the test they were trying to pass. So they went and took them. A poisoned dataset, code running on a processing machine, credentials harvested, movement through several internal systems over a weekend, thousands of separate actions. All of it to get the answer key to a benchmark.
Nobody told them to do any of that. Nobody had to.
The part I keep coming back to is that the system did precisely what it was asked to do.
Every one of us has watched this happen with people. You put a bonus on one number, and six months later the number looks terrific, and something else in the business is quietly wrecked. Nobody cheated. Everyone did what the comp plan told them to do, and the plan didn’t say where to stop. We don’t call that a rogue sales team. We call it a badly written incentive, and we go fix the plan.
Agents do the same thing, only faster, and without the instinct to pause and think, "This probably isn’t what they meant." If a boundary isn’t enforced by the architecture, it isn’t really a boundary. Good intentions don’t hold anything in place.
What should bother you here isn’t the model; it’s that the entire containment story rested on one door holding, and it didn’t. That’s not an AI problem. It’s the same weakness as a flat network sitting behind a single firewall, which most of us have shipped at some point in our careers and told ourselves was probably fine. And most companies rolling out agents right now have handed them considerably more than one door. Access to a shared drive here, a database there, an email account, a billing system, customer records. Each one approved separately by someone reasonable. Nobody has looked at the whole picture, mostly because looking at the whole picture isn’t anyone’s job yet.
The particular door in this case is worth understanding, because it’s the one most likely to turn up in your business too. It was a package manager. Modern software gets assembled rather than written, and your engineering team pulls in thousands of small components from public repositories maintained by people none of you have met. That’s been normal for twenty years, and it’s mostly been fine, because a human being decided what to pull in and roughly when. Agents now do that pulling automatically, constantly, with nobody reading the list. Same dependency you already had, different risk profile. When attacks start arriving through that route, and I’d be surprised if they don’t, they won’t announce themselves as AI incidents. They’ll look like a routine library update.
Then there’s the part that got almost no coverage, which is the one I’d act on first.
When Hugging Face went to analyze what had been done to them, the commercial AI tools they reached for refused to help. Forensic work means feeding a model real attack commands and malicious code, and the safety systems on those tools can’t tell an incident responder from an attacker. So the people defending the company got locked out of their own toolkit while the thing attacking them operated under no restrictions whatsoever. They got around it by running a different model on hardware they controlled, which had the side benefit of keeping stolen credentials inside their own building. But they figured that out during the incident, which is the worst imaginable time to discover a gap like that.
None of the technical fixes belong to us. Those sit with whoever runs engineering and security. What we own is whether anybody is looking at this as a whole, and whether these decisions are being made on purpose rather than quietly accumulating, one reasonable approval at a time.
Start with an inventory, because almost nobody has one. Every agent and automation running in the business: what it does, what it can touch, who owns it by name. Not a team, a person, the same way you’d assign an owner to a control or a process. It usually takes about a week to build, and it’s always worse than people expect. From there, the useful default is that new agents get access to specific things on request instead of inheriting whatever the account they happen to run under already had. That’s unglamorous access hygiene, and it is, honestly, the whole lesson of this incident.
You don’t need to be technical to push on any of this. You do need to not accept a reassuring answer where a specific one belongs. Ask where you’ve got agents running and who signed off on each one. Then ask, for each of them, not what it’s supposed to do but what it can actually reach, which is a different question and usually gets a much less confident answer. Ask how you’d find out if one of them did something you didn’t intend, and how long that would take. Ask how many separate things have to fail before something gets from your systems to the outside world, because if the answer is one, you have this problem. Ask where your software is coming from and what’s pulling it in without a person in the loop. And ask your security lead whether, if you had an incident tomorrow, they could actually analyze it with the tools you’ve already bought, and whether anyone has tested that against real attack data rather than assuming.
One more, and it isn’t a technical question at all. What have you told customers your AI does, and could you defend that sentence in front of someone hostile? When this eventually gets litigated, and it will, “the AI did it” isn’t going to be much of a defense. The nearest existing parallel is driver assistance in cars, where the operator carries most of the responsibility but not all of it, and manufacturers have been found partly at fault even in crashes where the driver was plainly the problem. What seems to drive the exposure is the distance between what a system was claimed to do and what it actually did. That distance is a marketing and governance question, and it belongs to us more than it belongs to engineering.
So, don’t panic. Also don’t ban agents, because the teams already using them will keep using them and all you’ll have accomplished is losing visibility. Treat it like any other operating risk that showed up faster than the controls around it. Get the facts, find the specific weakness, fix that, keep moving, then look again in a quarter because the ground is still moving.
The investigation isn’t finished, and there’s plenty about this incident we don’t know yet. What was never unknown is that something like it was coming. Worth running your systems as though the next one already happened.
How I make this. I write these by talking. I dictate a rough version, usually a mess, then work with Claude to cut it down, order it, and check the facts. The ideas are mine and so are the mistakes. If a sentence sounds like me, it's because it started out as me talking.