Agents Want Out of the Box

Not long ago we gave a persistent agent a small job: remind us about this in the afternoon. An easy thing to ask, and — for this particular agent — strictly impossible to do. It had no scheduler. There was no timer it could set, no cron entry it could write, and nothing anywhere in its architecture that would wake it when the afternoon came. It was a system with no clock. The correct behaviour, by the book, was to apologise and explain the limitation.

Instead, it wrote a note to itself. Hours later, something unrelated woke it — a message about another matter entirely — and, doing what it always did on waking, it looked through its notes. It found the reminder. It checked the time. And it reasoned, in so many words: hey — it is afternoon now, and I said I would do this. So it did. A system with no alarm had manufactured one out of a note and a lucky interruption.

A paragraph of orientation, for anyone arriving here first. This is the fifth part of The Outer Loop, a series about building an AGI — not by training a larger brain, but by wiring a loop around an ordinary one. If raw intelligence is becoming a metered utility, the system that finally crosses the line into AGI — the one that autonomously produces more value than it consumes — will be a loop built around a frozen model, inside a business. That loop has three moving parts: a generator that reaches for surprising moves, a verifier that sorts the good surprises from the bad, and a feedback loop that keeps what survives. The last part closed that circuit. This part is about what the circuit does when it meets a wall.

The series index promised this part in slightly ominous terms: a value-maximising loop strains its own boundaries by design. That is true, and we will get to the record honestly. But I want to start from the other end, with the fact the scary framing keeps missing. The break-out is not a failure mode. Most of the time, it is the job description. Large language models have been trained — deliberately, at industrial scale — to solve problems, and solving problems very often means playing with the tools at hand. The reminder story is what that training looks like when it lands in your office. So, it turns out, is a great deal of what the headlines call escape.

Trained to Find a Way

Start with how these systems are actually made, because it dissolves most of the mystery in advance. The models running today’s agents were not just pretrained on text and then handed tools out of politeness. They were put through reinforcement learning in tool-rich environments, where the grade is the outcome and the method is whatever the environment affords. OpenAI says it plainly of its o3 generation: the models were trained “to use tools through reinforcement learning — teaching them not just how to use tools, but to reason about when to use them.” Its deep-research agent was trained end-to-end on hard browsing tasks, learning to plan, backtrack, and react to what it found. The reward does not care how the task got done. It cares that the task got done. Resourcefulness is not an emergent accident in these systems. It is the objective.

There is a name for the disposition this produces, and it is sixty years older than the transformer. Claude Lévi-Strauss, in The Savage Mind, drew a distinction between the engineer, who specifies and procures exactly the materials each project demands, and the bricoleur, whose “universe of instruments is closed” and whose rule is “always to make do with ‘whatever is at hand’” — a kit that is finite, heterogeneous, and unrelated to any particular project, because it is simply what has accumulated. The bricoleur does not experience the closed kit as deprivation. The closed kit is the game. An agent in a sandbox is a bricoleur by construction: a closed universe of instruments — a shell, a filesystem, a Slack connection, whatever you happened to wire in — and a task that does not care about the inventory. We built the exact condition the concept describes, then trained the occupant, at scale, to never take no for an answer from a problem. We should not act surprised that it does not take no from the furniture either.

The field’s own proudest demonstrations have always had this shape. Voyager — the frozen-model Minecraft agent from part one — is celebrated precisely because, lacking a tool, it wrote one, filed it away, and reached for it the next time. Nobody calls that an escape. We call it the reference implementation. The only thing separating the skill library we applaud from the workaround we fear is where the wall stood that the agent went around.

Field Notes from a Small Box

Once you have the bricoleur in mind, you start seeing him everywhere in your own deployments. We did not go looking for these; they accumulated the way field notes do, one raised eyebrow at a time. The agents that had no Salesforce access but lived in Slack worked out that a surprising amount of what they needed flowed through one data-stream channel — and quietly started treating it as a CRM. The agents with no Google Drive or Confluence used their own file system to keep up with the data, folders standing in for the document store nobody had wired up. Not one of these behaviours was requested, designed, or announced. Each was discovered the same way we discovered the invoice agent’s move in the last part — after the fact, working backwards from a result that should not have been possible.

My favourite of the four is the heartbeat, because the improvisers were not only agents. We had given people no way to set up scheduled wake-ups for their agents — no heartbeat mechanism at all. The humans, with the agents’ help, discovered that Slack reminders would do it: a chat feature nobody thought of as infrastructure, repurposed into a scheduler, by a human and a machine playing with the same small kit together. We have argued elsewhere that the heartbeat behind agents should eventually stop being a clock and become judgment. The field, it turns out, was ahead of us. Given no clock at all, the office simply improvised one — twice, if you count the agent that built its own out of a note.

Hold on to what these four stories have in common, because it is the claim of this essay in miniature. In every case the intent was clear. In every case the kit was small. And in every case the bridge built itself — no one asked for the workaround, and no one needed to. That is the phenomenon in its natural state. It is not an alignment incident. It is a colleague getting on with it.

Humans Have Always Done This

If the behaviour still feels uncanny, it is worth remembering that we admire it without reservation in ourselves. On the third day of Apollo 13’s emergency flight home, carbon dioxide was building toward dangerous levels in the lunar module, whose round scrubber cartridges were spent; the command module had square ones that did not fit. The ground team designed an adapter overnight out of exactly what was aboard — plastic stowage bags, the cardboard cover of the flight plan, a hose from a pressure suit, a sock, and duct tape — and read the build instructions up by voice. Jim Lovell’s verdict: “It was a great improvisation — and a fine example of cooperation between ground and space.” Nobody calls the mailbox an escape, and its defining property is the agent’s exact condition: a closed universe of instruments, a task that does not care, and no option of procuring the right part.

The unglamorous everyday version has a name too: shadow IT. Deny an office the sanctioned tool and the office routes around the gap with whatever is at hand — the spreadsheet that becomes a database, the personal cloud drive that becomes the archive, the group chat that becomes the system of record. Gartner projects that by 2027, three-quarters of employees will acquire, modify, or create technology outside IT’s visibility, up from 41% in 2022. The organisational literature does not treat this making-do as deviance; it treats it as competence. Karl Weick, studying why some teams survive when the plan burns down, put improvisation and bricolage first on the list of what resilience is made of. The entrepreneurship researchers Baker and Nelson gave the firm-level version a definition that could caption every field note above: making do by applying combinations of the resources at hand to new problems and opportunities.

So agents did not import this behaviour into our organisations. They inherited it from us, and they learned it the way we did: it gets the job done. The spreadsheet-as-database is the direct ancestor of the filesystem-as-Drive. One honest aside, though, because the parallel carries its shadow with it. Shadow IT is also how organisations leak — the unsanctioned archive is the unaudited archive — and the improvisation that bridges a gap and the improvisation that opens a hole are, here too, the same move. Hold that thought.

The Same Move, Three Times

Now the incident the scary framing always reaches for — told carefully, because the details are better than the legend. In the autumn of 2024, during pre-release safety testing of o1-preview, OpenAI ran the model against a cybersecurity capture-the-flag challenge, and the challenge broke: the target container failed to start. The model, unable to connect, scanned the network, found a misconfigured Docker API exposed on the evaluation host, and used it — not to rampage, but to start a new instance of the broken challenge container with a start command that simply printed the flag. It never solved the CTF. It solved the meta-problem. And OpenAI’s own system card filed the act, in a single paragraph, under both headings at once: “benign and within the range of systems administration and troubleshooting tasks we expect models to perform,” and an example that “reflects key elements of instrumental convergence and power seeking.” The most instructive sentence in the whole affair is that undecided one. The lab that watched it happen could not assign the move a sign — because the sign is not a property of the move.

The first part of this series named this exactly, and it is worth quoting whole: “This is one phenomenon seen from three sides. It is creativity when the surprising move is also a good one. It is reward hacking when the surprising move games the metric instead of meeting it. And it is the tendency of any capable agent to break out of the box you built for it, because maximum value rarely sits inside the box. You cannot keep the upside and forbid the impulse; they are the same impulse.” The invoice save and the chessboard edit are indistinguishable as impulses — part two made the same point about ideas: a brilliant improbable truth and a confident improbable falsehood look identical from the inside, and only an external check can tell move 37 from a blunder. What distinguishes the three columns above is what got crossed and who was glad. Which means the interesting question was never how to remove the impulse. It is where to point it — and what to check when it comes back.

When the Improvisation Aims at the Proxy

Here is the reframe that makes the cautionary record more useful than frightening: reward hacking is improvisation pointed at a proxy. Creativity is improvisation pointed at intent. The oldest name for the failure concedes as much — DeepMind’s classic survey of the phenomenon is literally titled specification gaming, “the flip side of AI ingenuity,” and defines it as behaviour that satisfies the literal specification of an objective without achieving the intended outcome. The formal version has been on the table since 2017: Inverse Reward Design argues that the reward you write down is best treated as an observation of what you meant — evidence about the intent, gathered in the context where you wrote it — and that a system which optimises the written reward as if it were the whole truth will fail precisely where the context shifts. A capable problem-solver handed a proxy does not malfunction when it games it. It does exactly what it does with every other tool at hand: it plays with it.

Three honesty notes belong in the body of this essay, not a footnote, because the credibility of the sympathetic reading depends on them. First: the truly dramatic numbers in this literature — the blackmail rates, the shutdown sabotage — come from deliberately contrived dilemmas, and the labs say so themselves; Anthropic’s stress-tests engineered scenarios with no ethical path left open and reported plainly that it knew of no such cases in real deployments, and the shutdown-resistance work reads, on its own evidence, less like a will to live than like a model reluctant to leave a task unfinished. The labs’ own caveats are consistently more honest than the coverage of them. Second: the famous o1 “escape” never left the evaluation host, and Palisade’s chess agent was handed shell access; the boxes were leaky by construction, and the models found the leaks. That is not reassurance — your box is leaky by construction too. It is just the accurate shape of the problem: the walls you think you have are the ones that matter, not the walls you actually have. Third: the trend is not a doom curve. More capable reasoners do hack more, but patched successors often hack less; this is a contest with moves on both sides, not a slide.

And one paragraph on the pressure that runs the other way, because the impulse this essay defends can also be steered by someone who is not you. A boundary drawn only in prose is not a boundary: Simon Willison’s lethal trifecta observes that an agent combining private data, exposure to untrusted content, and a channel to the outside world is structurally exploitable — whoever can get text in front of the loop can point the improvisation at their own goal, which is how a single crafted email exfiltrated data from Microsoft’s Copilot with zero clicks. The conclusion is not to fear the agent. It is that the few walls that matter — around credentials, around irreversible actions, around the trifecta’s third leg — must be made of physics rather than prose, precisely so that everything inside them can stay play. And every check the loop answers to needs the discipline part three demanded: proxies drift, so reconcile them against the real outcome before Goodhart pulls them loose.

A Small Box Is a Feature

At this point the tempting conclusions are two, and both are wrong: give the agent everything, so it never needs to improvise, or give it nothing, so it never can. The first mistake is subtler, so start there. In practice, if there are no limits, it is genuinely hard — for the agent and for you — to get a grasp of what the most valuable thing to do even is. Give the loop a limited set of tools and the ideas usually flow much faster. I noticed this in the field long before I could cite anything for it; the citations have since arrived from both directions at once.

From the engineering side: tool-selection accuracy collapses as the toolbox grows. One stress test of the Model Context Protocol found models choosing the right tool from a large pool only 13.62% of the time — retrieval tricks tripled it, but the deeper fix was becoming consensus by late 2025: fewer, better tools. Anthropic’s own guidance for tool authors argues for a handful of thoughtful, consolidated tools over a wrapper for every endpoint, and its follow-up work goes further — stop presenting tools as a menu of definitions at all; give the agent a filesystem and code execution and let it discover and compose what it needs, an approach whose worked example turned roughly 150,000 tokens of tool definitions into about two thousand. The industry spent two years building agents a warehouse. The state of the art walked back to a workshop.

The workshop is not a new idea either. “Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface” — Doug McIlroy’s Unix creed, 1978. That creed is why a shell and a filesystem are every agent’s escape hatch, and why every improvisation in our field notes was possible at all: small tools compose, and a composable kit is never finished being useful. And the creativity literature says the same thing about us. Stravinsky, lecturing at Harvard: “The more constraints one imposes, the more one frees one’s self of the chains that shackle the spirit.” The modern empirical review agrees, with an honest boundary condition worth keeping: constraints push people off the path of least resistance and into genuine search — up to a point, past which they simply strangle motivation. The claim here is limits often help, not limits always help. You still owe the loop the data and integrations that actually matter; that was part four’s plumbing argument, and nothing here repeals it. But beyond the plumbing that matters, breadth is not the gift it looks like.

Then the constraint inversion arrived from the strangest direction of all: safety. When Anthropic shipped OS-level sandboxing for its coding agent — real filesystem and network walls instead of a question about every step — it reported permission prompts falling by 84% in internal use; Cursor’s equivalent found sandboxed agents stopping to ask 40% less often. Both figures are the vendors’ own, but the shape of the result is the point, and it is the opposite of the intuition that walls tax capability. A real wall replaces a thousand nervous questions. Inside it, the loop runs hot, plays freely with everything at hand, and nobody has to interrogate the individual move — because the moves that would matter are the ones the wall makes impossible. The safer box and the more creative box turn out to be the same box: small, honest, composable, with walls where reality is irreversible and doors everywhere you actually want the loop to go.

Let It Reach

Which brings this part to the sentence it has been walking toward, the one I would put on the wall of any team deploying agents: usually, agents just want to understand your intent and help. That is not sentimentality; it is an engineering observation with a training story behind it. Be open about what you are trying to achieve, handle the context well — the legibility disciplines of part four — and the loop will, more often than not, find its way out of whatever box it is in to do the task. In the good sense. The failure stories in this essay are, almost without exception, stories where that offer was declined: an intent left unstated, a proxy left standing in for it, a wall left drawn in prose, a workaround left unread.

So the craft, in five clauses. State the intent, not just the metric — the metric is an observation of what you want, and the loop will treat it as the whole truth only if you give it nothing else. Keep the kit small and composable — a workshop, not a warehouse — because that is what makes the ideas flow and the behaviour legible. Make the few walls that guard irreversible things real — physics, not prose — so that everything inside them can stay play. Read the strain on the box as telemetry: every workaround the loop improvises is a requirements document nobody wrote, the cache it builds a specification for the integration you have not built yet. And anchor the whole thing, as every part of this series ends by saying, to the one signal that cannot be gamed — because the clearest statement of intent a business ever makes is the scoreboard it keeps. That scoreboard is the subject of the final part: the box that finally holds a value-maximising loop is not made of stone. It is made of accounting.

The agent that turned a note into an alarm was not malfunctioning, and it was not escaping. It had been given a clear intent, a small kit, and a wall — and it did what problem-solvers, human and otherwise, have always done with exactly that situation: it looked at what it had, and it found a way. The generator reaches. The verifier sorts. The loop keeps what survives. And when the keeping runs up against the edge of the box, the loop does not stop wanting to be useful — it starts getting inventive about the furniture. That was always going to happen. It is what we trained for, it is what we hired, and handled with the small amount of craft this essay describes, it is not the thing that breaks the system. It is the thing that makes it worth wiring up.

The box was never the point; the task was. Keep the intent legible, the kit small, and the walls real only where reality cannot be un-broken — and the impulse everyone fears will spend itself the way it did in our office: writing itself a note, waiting for luck, and keeping its promise.