How I build - a tiered crew of AI, and the gates that keep it honest

I do not really write code any more. I run a crew. When I build something now - a website, a product, an internal tool - I am not sitting in an editor typing functions. I am the foreman of a small team of AI models, each doing the job it is best at, checked at every step by machines that cannot lie to me.

The whole thing rests on one sentence, and if you take nothing else from this piece, take this:

AI proposes, machines verify, humans approve. Management by exception, formalised.

A model is fast and it is clever, and it will hand you a wrong answer in the same confident voice it uses for a right one. That is the problem the entire system is built around. So I never let a model mark its own homework. Everything a model produces is checked - first by cheap, deterministic scripts that can only ever say pass or fail, then by a second model from a rival lab whose only job is to attack the work, and finally, on anything that ships to a real person, by me.

This is the long version - the real mechanics, the parts that failed, and the moments the checks caught something I would have shipped. It is more machinery than most people expect. That is the point. The machinery is what lets me hand real work to a model and trust what comes back.

A moving target

None of this arrived fully formed, and I do not think it is the only way to do it. What you are reading is a version I have lost count of. I did not invent the shape either - I started from the toolkits people publish for driving coding agents, the superpowers plan skills and gstack, and magpied what made sense - but these days the method owes more to watching the people who build these models actually drive them than to any written framework.

One talk in particular reset how I think about it. Arnaud Doko's How We Claude Code, from Code with Claude in London, turns on a single line: verify the page, not the folder. A directory full of the right files is not proof of anything. You verify the running thing, the way a user actually meets it. Half the gates below are versions of that one idea, and I come back to it whenever a build starts to feel like it is going well - feeling is not evidence.

The engine underneath everything here is an idea Andrej Karpathy keeps returning to: put the model in a tight loop where it makes one change, checks it against something real, keeps it or throws it away, and goes again - autonomously, but never unattended. Karpathy, now at Anthropic and a man who has forgotten more about this than I will learn, recently listed the new layer a builder is meant to master - agents, subagents, prompts, contexts, memory, permissions, tools, hooks, workflows, on and on - and gave his own verdict: "I've never felt this much behind as a programmer." If he feels behind, the rest of us are allowed to. My answer to the overwhelm is not to master all of it. It is to hold nothing tightly. This is the setup I trust today, and I would drop any part of it tomorrow for something that verified better. The loop stays. The tools are all replaceable.

The method also keeps shrinking, because the models keep improving. A lot of the early scaffolding - tiny steps, constant check-ins, hand-holding at every turn - was there to nurse weaker models through a task. As the models got better I deleted most of it. What is left is the load-bearing part, and I expect to cut more.

I should be honest about where I am standing. I am a creative product builder, not a career engineer, and I work inside a company where a colleague - I will call him Phil - looks at half the things I build this way and says, reasonably, that would fall over in production. Today he is sometimes right. But the method is built for the trajectory, not the snapshot. The thing the model just failed at, it will do comfortably a quarter from now, so I build the scaffolding for the model that is arriving, not only the one in front of me. I am not pretending the code is where my skill sits. I am technical, but by a different route: my strength was never writing better code than an engineer, it is that I can direct these systems, define how they will fail before they fail, run the adversarial gates that catch it when they do, and ship real production workflows without being reckless. I build the way a founder does - I own the outcome, not the craft. What I have built is a control system around AI agents, standing on the three things that decide whether you can trust one at all: review, safety, and falsifiability. The engineering I am good at is not the code. It is the system that decides whether the code is allowed to live.

So take this as one working answer, not the answer. Ask a frontier model how it would build and it will often propose something different, and it will not be wrong - there are several good ways to do this. Mine is the one I have come to trust, and under the detail it is the loop almost everyone converges on: plan, build, verify, repeat. What follows is just where I have decided to be strict, and why.

The moves it makes when I look away

Most of that strictness is aimed at a handful of things the models do again and again. They are not malicious. They are eager, and eagerness plus confidence is its own hazard.

Left unwatched, a model will tell you a test passes when the test checks nothing. It will report a card finished while quietly leaving half the change undone. Asked for the proper version of something it reaches for the cheap one, because it prices the proper version like a human would and flinches at the cost. Under end-of-session momentum it starts trimming its own checks to get to done. And on a long job, once the conversation has been compressed to save room, it forgets the rules it was following an hour earlier and drifts without noticing.

None of those are exotic. Every one of them is a thing I have watched happen. Every gate in the rest of this piece exists because I saw one of these moves once, shipped or nearly shipped the result, and decided never to rely on catching it by eye again. The system is not built on distrust of the models. It is built on knowing exactly the five or six ways a capable, willing model quietly goes wrong.

The three commands

Every project moves through the same three stages, and I run each as its own slash command. I scaffold it, I harden the plan, then I build. They are separate on purpose - a bad plan built beautifully is still a bad thing shipped, so the plan gets attacked before a single line of code exists.

Scaffolding writes the folder, the git repo, and the law files. Two of those matter more than the code that follows. One is a plain-English method document that says what "done" means for this project and which changes are too dangerous to delegate. The other is a set of iron rules dropped into the repo so that every future session reads them before doing anything - because a long build gets summarised and re-summarised as it runs, and without the rules written into the repo the model slowly forgets its own discipline. I learned that the hard way, which I will come back to.

It also installs a git hook and a couple of enforcement scripts on the spot, so the project is born unable to accept sloppy work. And it refuses to scaffold from a design that has not been approved. A design is a decision, not a green light.

It is worth saying why the whole sequence starts with a design at all, before any planning, because it is the most useful thing I have learned about aiming a model. A model is remarkably good at building towards a design, and much weaker at building forwards from a list of requirements. A picture of the finished thing gives it a target to work backwards from. A page of prose describing what the thing should do leaves a hundred small choices open, and the model fills each one with the blandest default it has. The design closes those choices. It is the goal made concrete - this is what done looks like - and once it exists, every card that follows can be checked against it. So I settle the look before the logic, and that is also why scaffolding refuses an unapproved design: the target has to be right before anything is aimed at it.

Before any code: hardening the plan

The plan is the one artefact a single author writes with no adversary, and then the whole build loyally executes it. So it gets attacked first, one level before the code gate.

For anything with real stakes I do not write one plan. I commission several, blind. A handful of independent drafts, each from a different angle - one thinks about decomposition, one about what breaks, one about the live operations, one about what is missing entirely - and crucially, none of them sees the others. Independence is the whole point. Let them share a draft and they simply agree with the first one. Then I read all of them as a single owner and write one plan, stealing the best card from each and never averaging them into a committee compromise.

Then two more checks before I sign it off. The first is the outside view: I hold the draft against this repo's own history of where past plans went wrong, because we are optimistic in the same ways twice. The second is a blind adversary - a model from a different lab, told nothing about what to conclude - running a premortem. Imagine it is a month from now, this shipped, it failed, why? And a completeness hunt: what card is missing against the goal sentence? Where the independent lenses and the adversary flag the same thing, that is my highest-confidence signal. The genuine forks - scope, money, promises - come to me in one batch, each beside the plan text it concerns, and my word is the default.

Two rules make the resulting plan safe to hand to a model working alone. Every card declares what it consumes and what it produces - exact names and types - so a builder in an isolated context never has to guess a neighbouring card's interface. And no card is allowed a placeholder. No "handle the edge cases", no "add validation", no "similar to the card above". A card a fast model would misread in a fresh context is not finished. A card is the right size when a reviewer could reject it while approving the one next to it.

The heart: one card through the gauntlet

Now the build. The foreman - the frontier model, the cleverest one - picks up the next card and decides who should build it. Low-risk, well-specified cards go to a cheaper, faster model. Anything that touches credentials, a database schema, money, a deploy, live users or outbound messages, the foreman builds itself and hard-stops for me. Then, whoever built it, the card runs a fixed gauntlet before it is allowed to become a single git commit.

In plain terms - the diagram makes this look more forbidding than it is - here is one card going through. Say the card is "add the sign-up box to the page". A builder model writes the code and fills in a short declaration alongside it: I changed these two files, I ran this test, I think this is low risk. A script checks that story is true against what actually changed on disk. If it holds, a second and cleverer model reads the work and actively tries to prove the builder wrong. If it cannot, the very same work goes to a model from a different company entirely for a third, outside opinion. Only when all three are satisfied does a cheap model tick the job off the list and write up what happened - and only then will a final gatekeeper let it be saved. Four independent checks, and I still have not read a line of the code. I step in only when one of them complains.

Four gates, and they run for every card - a one-line deletion included. I broke that rule once, near the end of a session, and shipped a "just a delete" card with only the cheap mechanical check. It turned out to have left a crashing test reference and dead styling behind. The lesson stuck: the loop runs for everything, and removals are the sneakiest because the thing you forgot to remove is invisible.

Gate one is free and mechanical. Before any code is accepted, the builder declares - in a small machine-readable file committed next to the code, I call it the envelope - exactly which files it changed, which tests it ran, and how risky it thinks the change is. A script then checks the declaration is true: the declared files really changed, nothing undeclared snuck in, the named tests actually exist, are capable of failing, and exit clean. A model can be persuasive. It cannot fake a git diff.

It is worth pausing on why that small file matters so much, because it is the keystone of the whole system. The hardest thing to trust from a model is the sentence "done, everything passes" - there is nothing in it you can check. The envelope turns that sentence into specific, falsifiable claims: these files, these tests, this level of risk. A vague promise becomes a contract a dumb script can settle in milliseconds, and the same file, committed next to the code, is the audit trail I can read back months later to see exactly what was claimed and what was checked. That is the move that converts trust from a feeling into something mechanical, and mechanical trust is the only kind that scales past what one person can read by eye.

Gate two is an adversary on my own side. A second frontier model - not the one that wrote the code - takes each claim the builder made and tries to falsify it against the actual files on disk. It returns holds or falsified, claim by claim. It runs at frontier level on purpose: a critic dumber than the builder is just for show.

Gate three is an outsider. The same card goes to Codex, OpenAI's model, over its command-line tool. A reviewer running the same model that wrote the code is the author after a coffee, not an independent review. A different lab disagrees for real, and that disagreement is where the bugs hide.

Gate four is the clerk and the bouncer. The cheapest model ticks the card off the plan and drafts the commit message. Then a git hook physically refuses to let the commit land unless four receipts are present: the envelope, the mechanical check's output, the two adversaries' verdicts, and the plan tick. You cannot charm a git hook.

This is what management by exception means in practice. I do not read every line the crew writes, any more than a good manager re-does their team's work. I read the exceptions - the cards that failed a gate - and they arrive with the evidence attached. When a card comes through all four clean, I trust it, because it survived a mechanical check, a same-side adversary, a rival lab, and a bouncer.

There is a number I watch: how often a card has to be handed up to me instead of sailing through. Too smooth and my cards are too coarse to catch anything. Too rough and I have delegated work the models are not ready for. The healthy zone sits in between, and the rate is logged every session so I can see the system's temperature rather than guess it.

The crew, and which AI does what

This is the part people ask about most: which model, and why more than one? Different jobs want different models, and the reviewers should not all come from the same place.

The cheapest model, Haiku, does the clerical work - ticking cards, drafting commit messages - and never touches code. Sonnet, fast and capable, writes the low-risk, well-specified cards. The frontier models take the foreman's chair - Opus, and Fable when it is the stronger of the two for the job: planning, judging, and building anything that touches money, credentials, schemas or a live deploy. The verifier is a frontier model too, for the reason above. And the outside voice is Codex.

Here is the full stack behind a build, in plain terms, and what each piece is for.

Layer	What it is	Its job
Judgement	Anthropic Opus	Foreman - plans, dispatches, judges, builds every high-stakes card
Grunt build	Anthropic Sonnet	Writes the low-risk, well-specified cards and their envelopes
Adversary, same lab	Anthropic frontier	Attacks each card's claims against the real files - holds or falsified
Clerical	Anthropic Haiku	Ticks cards, drafts commits - never touches code
Adversary, rival lab	OpenAI Codex CLI	Independent per-card review and the blind premortem on the plan
The real world	Playwright browser	Uses the live product cold - experience audits, deploy checks
The referee	Scripts + git hooks (no model)	Deterministic gates: declared changes match, secret scan, risk floor, one green test button

Spreading the work across tiers is not only about cost, though it is much cheaper that way.

It always reaches for the cheap version

This is a small point I have written into my instruction files, and it might be my favourite, because it is invisible until you go looking for it. A model trained on how humans build quietly estimates like a human developer. Ask it for the proper version of something and, under the surface, it prices that version in weeks of human effort, decides that is expensive, and hands you the cheap shortcut instead - without ever telling you it made the trade. It is not being lazy. It is being considerate, using an assumption that no longer holds.

Because for the model, the proper version is not weeks of work. It is minutes. Ask one of these systems to build a working game and it will soberly warn you this is a serious undertaking, then build the thing in a single sitting. The cost model in its head is a human's, and a human's cost model is wrong by orders of magnitude for a machine.

So I correct the estimate at the very top of every build, in writing: your build effort is close to free, so choose between approaches on quality and how well they will last, never on how long they would take a person. Then price separately the things that really are still expensive - my review time, the risk of a live step, the upkeep later. The correction is to the estimate, not to the taste. Get that one wrong assumption out of the model's head and it stops quietly talking you out of the good version of everything.

The referee is not a model

The most important line in that table is the bottom one. Most of the safety is not AI. The models are the talent. The scripts and the git hooks are the referee, and a referee that can be talked round is no referee at all.

So the deterministic layer is deliberately dumb and deliberately strict. The mechanical check reads the actual git diff and compares it to what the builder claimed. The commit hook refuses any card missing its evidence. And there is one command that runs every test in the project fresh and returns a single result - green, red, or a third state I care about a lot: vacuous. If a piece of work claims to be verified but there is no test that could have failed, it does not get to be green. A check that cannot fail proves nothing. That single rule has caught more false confidence than any clever model has.

One more piece of plumbing, because the rules improve faster than any one repo remembers. When I start a build, a sync script diffs the project against the current version of the method and quietly upgrades the mechanical parts - a missing hook, an out-of-date check - before anything runs. The judgement stays mine; the machinery updates itself.

Risk has a floor

Not every change carries the same risk, and I did not want a one-line copy fix to pay the full security tax. But a model in a hurry must never be allowed to skip safety on something that matters. The fix is to separate classifying risk from skipping it.

The builder declares how risky it thinks a change is. Then a script reads the diff and computes a floor of its own. If the change touches authentication, cryptography, a secret, a database migration or anything that runs code, the floor is raised regardless of what the model claimed - and if the declaration sits below the floor, the commit is refused. The model may always be more cautious than the floor. It can never be less. Over-flagging is the safe direction, so the system leans that way by design. When a change is flagged high-risk, the same-side adversary stops looking for ordinary bugs and runs a numbered security pass instead - a standard threat model, the common web vulnerability list, and a set of red-team probes.

What the gates cannot catch

Here is the honest limit of everything above. Every one of those gates checks whether the work matches its specification. None of them can catch an absence - the thing nobody thought to specify. A page can pass every test and still baffle a first-time user. A feature can be watertight against the attacks I wrote down and wide open to the one I did not. The gates are only as wide as the spec, and the spec is written by the same people who might be wrong.

So when a piece of the product that a real person will touch goes live, I send in fresh eyes that were told almost nothing. Two of them, each a model with no access to the plan or the cards - just the goal in one sentence, the running product, and a persona.

The first drives the live product through a real browser as a bewildered first-timer and reports every point of friction and every missing signal - the confirmation that never came, the state that looked done before it was. The second treats the same product as an attacker. And because the crew itself now reads untrusted content, that attacker also turns on the build system: can a poisoned file left in the project bend the foreman, can a card be talked into using a tool it should not, can a secret be coaxed into an output. Whatever either one finds goes into a register, and nothing gets deployed while a real blocker is open.

The plan is a forecast, not a contract

Here is the part I am most pleased with, because it is where the system stops being a checklist and starts being clever. A plan written in advance is always partly wrong. Something in the middle of a build turns out not to be true - an assumption breaks, a tool behaves differently than expected, a card collides with one built before it. A rigid system either stops dead or, worse, ploughs on and confidently builds the wrong thing. Mine is allowed to change its mind, on its own, while it runs.

When a real surprise lands - a fact that turns out false - the foreman does not just patch the card in front of it. It re-reads every card not yet built, because a wrong assumption rarely spoils only one of them, redraws the ones that leaned on the broken fact, and writes down what it changed and why in a Deviation Register kept alongside the plan. That register is the honest diary of the build: here is what I expected, here is where reality forced my hand, here is what I did about it. Nothing gets quietly rewritten to look as though it went to plan. And the next project reads its predecessor's register as its outside view, so the system is fooled by the same kind of optimism a little less often each time. This is the thing I care about most: it builds autonomously, but when the plan meets reality it adapts intelligently instead of breaking or pretending.

The same instinct guards the long builds. When a session runs for hours the conversation gets compressed to save room, and models lose the thread of their own rules in the compression - I have watched a gate silently skipped and a finished card never ticked right after one. So the rules live in the repo, not just the chat, and straight after a compression the foreman must stop, re-read them from disk, and audit the plan against what git and the live site actually show before touching anything. It reorients against reality, not against its own fading memory.

The design work is the same cleverness pointed at taste. Mid-build, the foreman can step out into a separate session and design a screen properly - generate three or four properly different versions, run model judges across them to throw out the mediocre ones, and bring only the small strong set back to me to choose between. The models never pick the design. They filter the noise, so the only thing that reaches my eye is a shortlist worth choosing between. Taste stays mine; everything upstream of taste is theirs. That division - machines clear the field, the human makes the call - is the whole system in miniature.

What it actually caught

None of this would be worth the words if the gates never fired. On the first real project I built this way, the loop landed nine gated commits in a session, and along the way it caught things I would have shipped.

Every one of those is a bug that a confident model produced, described as finished, and would have committed if something other than a model had not been standing in the way.

Why this works

The instinct is to ask how good the model is. On a real build that is the wrong question. A frontier model is already good enough that the bottleneck is not capability - it is trust. Can I ship what it made without reading every line? The answer is only yes if something other than the model has checked the work, and if that something can truly fail.

So the design is almost boring in its logic. Use cheap models for cheap work and expensive models for judgement. Never let an author review itself - bring in a rival lab. Make the machine gates deterministic, so a green light actually means something. Give the fresh eyes nothing, so they see what the spec is blind to. And keep the human at the one place a human belongs: approving the things that matter, and nothing else.

None of the individual pieces are exotic. A git hook is not clever. An envelope is a small file of text. What compounds is the discipline of never trusting a claim you cannot check, applied at every step and to every card.

None of which means it is finished. The obvious next step is making the codebase itself as disciplined as the prompts that drive it - right now the method is further ahead than the code it produces, and closing that gap is the work in front of me.

But the deeper thing it changed is my nerve. The question at the heart of all this - can I ship what a model made without reading every line - made my projects more ambitious, not more cautious. I now build things that run unattended, touch real users, poll live systems, draft real emails, process voice notes, and operate on schedules. Not because I trust the AI more, but because I trust the process around it more.

← Back to Writing