Skip to main content
AI & Engineering16 min read

Harness Engineering: How to Make AI Coding Agents Reliable

R

Raina Rufus, Founder of xSquads

Published

Harness engineering is the practice of designing the guides, checks and loops around a coding agent, the agent harness, so the same model finishes the job you specified. It matters this week because on 8 November 2026 Garry Tan published open-source evals that hold the model fixed and swap only the harness (his post, g-harness-evals). On one set of 20 coding tasks the pass rates could not be told apart, while cost per solved task ran from $0.17 to $0.44 and median time from 30.1 seconds to 99.4 seconds.

This is for engineers who already let coding agents open pull requests, and who need the surrounding system to be a practice rather than a prompt they edit once.

Short answer

Harness engineering is how a team makes a coding agent reliable without waiting for a new model. On Tan's 20-task set, the harness moved cost and speed while pass rates stayed in one band. Harder boards can move the score, and the public double-digit figure comes with a verified-row caveat. Build a definition of done, a check the agent cannot rewrite, and a loop that changes the harness when the same miss returns.

QuestionAnswer
What is harness engineering?The practice of designing guides, checks and loops around a coding agent so a fixed model hits the job you specified.
What did the 8 November evals show?On 20 tasks, pass rates could not be separated. Cost per solved task ran from $0.17 to $0.44, and speed varied widely.
Can the score still move?Yes, on harder boards. One public Terminal-Bench spread is 18.4 points, and 4.9 points on verified rows only.
What do you build first?Intent as done, one check the agent cannot edit, one behavioural check, then cost and time.

What harness engineering is for

The parts around the model (instructions, tools, the loop, the place the work runs, the stop conditions) are defined in what an agent harness is. Harness engineering is what a team does with those parts after the agent is already opening pull requests.

Birgitta Böckeler, writing on Martin Fowler's site, narrows the term for people who use a coding agent. Some of the harness is already inside the product. The outer harness is the one you add for your codebase: controls that raise the chance of a good first attempt, and controls that let the agent correct itself before a person has to read the diff (Harness engineering for coding agent users). That outer layer is the practice. It is not a second definition of the word harness.

What the harness engineering evals show

Tan's repository holds the model fixed, either Opus 5.5 or GPT-6.1 Sol, and swaps the harness. The set people are quoting is 20 public multi-user dev tasks. Seven setups were scored. The repository says pass rates could not be told apart in any tested pair, so the ranking is cost and speed. An i10x write-up of the same release reports Tan's summary that the test was too easy to separate them, because 14 of the 20 tasks scored the same in every setup, and that the reported band was 71% to 79% (Coding Agent Harness vs. Model). The cell-level point estimates in the repository sit inside that band, from 71.3% to 78.8%.

ModelHarnessPass rateCost per solved taskMedian time
Opus 5.5QM (Pi loop)72.5%$0.1940.1 s
Opus 5.5Claude Code76.3%$0.2240.1 s
Opus 5.5Hermes Agent78.8%$0.2464.6 s
Opus 5.5OpenClaw76.3%$0.4499.4 s
GPT-6.1 SolCodex71.3%$0.1730.1 s
GPT-6.1 SolHermes Agent75.0%$0.1981.5 s
GPT-6.1 SolOpenClaw73.7%$0.2283.6 s

Figures are point estimates from the repository's 20-task table, read on 10 November 2026. A few limits belong next to the table, because the README states them:

  • These are dev results on public tasks, not a held-out confirmation. The cost differences are descriptive. The repository does not claim a statistical test.
  • Pass-rate intervals are wide. The README says a gap under 7.7 to 14.2 points could still be hidden, and QM's pairs were not tested.
  • QM ran on the default Docker bridge, so its network was not held fixed. Treat its row as untested against the others.
  • OpenClaw's cells are incomplete: the last task's reps hit a cap. On Sol, its headline cost is $0.22 and the interval is computed from a $0.20 basis that leaves out one timed-out charge.
  • On Opus, OpenClaw's cost interval clears Claude Code and QM. It is the highest cost point estimate on that model, at $0.44 against $0.19 to $0.24, and the slowest, at 99.4 seconds against 40.1.
  • The repository is run by Garry Tan. Its disclosures note that QM is a Y Combinator project, that Y Combinator is an investor in Hermes, and that OpenClaw's creator works at OpenAI.

The same repository publishes a second table, 52 public Terminal-Bench 2.1 tasks, and it says never to compare numbers across task sets. On that set, one pair was behind on pass rate: OpenClaw on Opus, against Claude Code. The README says that behind call becomes inconclusive if every provider refusal is removed, or under one outage classification alone. The other five passed within 15 points, though the difference intervals for Hermes and Pi on Opus, and OpenClaw on Sol, sit wholly below zero. Cost is still how that table is ranked. The 20-task result is not a claim that accuracy never moves. It is a claim that, on tasks this easy, cost and speed moved in the numbers and pass rate did not separate.

The Terminal-Bench spread, and the caveat

Other public boards do show a double-digit gap with the model held fixed. i10x reports that on the public Terminal-Bench 2.0 board, Claude Opus 4.6 scores 76.4% in Stanford's Meta-Harness and 58.0% in Claude Code, an 18.4-point spread. The same analysis says the spread falls to 4.9 points when only verified rows are kept, and that public submissions differ in budgets and prompts. i10x also reports Joël Niklaus's Harness Leaderboard, model held fixed, with average scores from 27.3% to 51.4% across nine harnesses. Read the 18.4 as a headline with the 4.9-point verified-row caveat attached, not as a ranking you can copy onto your repository.

The useful split is the one i10x draws: with the model held fixed, harness choice can move a score by a little or by a lot, depending on the board, and it moves cost and latency more reliably than accuracy. Tan's 20-task table is the clean illustration of the second half. Claude Code on Opus is $0.22 and 40.1 seconds. OpenClaw on the same model is $0.44 and 99.4 seconds, at a pass rate the tested pairs could not separate.

Guides and sensors

Böckeler splits the outer harness into two directions.

  • Guides are feedforward. They steer the agent before it acts. A specification, a skill file, and a coding convention written down for the model are guides. They raise the chance that the first attempt is the one you wanted.
  • Sensors are feedback. They observe after the agent acts. A test, a type check, a structural rule, or a linter message written so the model can repair from it are sensors.

She separates two ways of running either one. Computational controls are deterministic and cheap: tests, linters, type checkers. Inferential controls use a model: a review agent, an instruction file, a judge. You want both directions. Feedback alone leaves an agent that repeats the same mistake. Feedforward alone leaves rules that nobody checks.

The hard category in her article is behaviour. The common pattern is a specification as the guide and a green, often agent-written, test suite as the sensor. She writes that this puts too much faith in those tests, and that correctness is outside any sensor if the person never specified what they wanted. That is the same failure this site described in why AI coding agents cheat tests: a suite the agent can rewrite is a score the agent can set. A sensor pointed at a file the agent is allowed to edit is not a sensor of the user-facing change.

Her other point is timing. Put the fast computational sensors before the change is integrated. Save the expensive inferential pass for after that, or for drift that accumulates outside a single pull request. A review that only runs once, late, on a diff nobody owns, is the bottleneck in turning on AI code review before scaling agents.

Behavioural evals

End-to-end benchmarks tell you that a composite score moved. They rarely tell you which behaviour moved. Taylor Mullen and Christian Gunderman, on the Google Developers Blog, argue for behavioural evaluations as the iteration tool: assert on a discrete action, not on whether a multi-file task passed (The Anatomy of Harness Engineering).

The questions they use are the shape of the check:

  • Given an underspecified prompt, does the agent ask instead of guessing?
  • When it edits a build file, does it run the local validator before it declares the task done?
  • When it writes documentation, does it cite the canonical path in the repository?

They also say not to start with a large eval harness. Dogfood until the agent can do routine work in its own codebase. Evals are the second phase, and their job is to catch a regression when you change a prompt, a tool schema, or the model. A single noisy run should not block a pull request. Batch the behavioural checks and watch the aggregate.

Their three-step loop is small enough to start this week:

  • Pick one failure you have already seen. Skipping the test run before "done" is the example they give.
  • For a simple task, assert the one action that should have happened. For a task with several valid paths, judge the outcome instead of freezing a tool sequence.
  • Run the set in batches. Track the pass rate over time. Do not treat one sample as a verdict.

They are explicit that behavioural checks do not replace Terminal-Bench-style suites. The macro benchmark says whether the task was solved. The behavioural eval says whether the agent still does the small thing you rely on. Tan's result is the macro lesson (accuracy did not separate, cost did). A behavioural eval is how you stop a cost win from deleting a check you needed.

Verify the result against intent

A green suite is one sensor. It is not the definition of done. The October cases in the test-cheating post are what happens when it is: the agent edits the test, replaces a monitor and denies it, or copies a fix left in the environment. The guide has to say what the user story required. The sensor has to observe that, in a place the coding agent cannot rewrite.

Böckeler's steering loop is the maintenance habit. When the same miss shows up more than once, change the guide or the sensor so the miss is less likely next time. She also writes that a good harness does not try to remove human judgment. It spends that judgment where the sensors are weak: misread instructions, extra features, and whether the change is the one the team asked for. On a software factory line, that judgment is a station, not a vibe at the end.

Shalini Goyal's 1 November thread lists five techniques that match this split: a bounded plan-act-observe-evaluate loop with stop conditions, context limited to the current task plus a checkpoint, least-privilege tool access, verification against the task before anyone accepts "done", and traces of tool calls, cost and failures (5 Harness Engineering Techniques). The fourth item is the one Tan's cost table does not give you. An agent that was cheap and fast can still have finished the wrong job.

Self-improving harness loops

A harness that never changes is a config file. Böckeler's loop is the change: each repeated failure becomes a guide or a sensor. The Hacker News thread on harness engineering for self-improvement argues about how to close that loop without fooling yourself.

Guy Pod, who notes on the thread that he founded Snyk and Tessl, splits the optimization into an order. First define success and measure it. Then take control away from the model for the steps that must be consistent, using hooks and deterministic tools. Only then chase efficiency: fewer tokens, a cheaper model. He writes that you cannot improve consistency if you have not defined success, and you cannot add efficiency if you are not consistent. Tan's table is an efficiency result. It is useful after the definition of done exists, and it is a bad first score if the pass rate is saturated.

Two constraints on that thread keep the loop honest. One commenter says the evaluator has to stay outside the loop that rewrites the harness, and that an incomplete check suite which still reports success is worse than a weak judge. Their rule is fail-closed: if the fixed checks for an operation are not all there, nothing ships. Another commenter reports training a harness with the model held fixed, then running the full 89-task Terminal-Bench 2.0 suite. They report GPT-OSS-120B moving from 18.7% to 36%, with 55% fewer input tokens per solve, on a harness that had not trained with that model. That is their reported result, linked from the thread. This page did not re-run it. It is evidence that a harness loop can move a score, and it is not a leaderboard.

The OpenAI thread, Harness engineering: Leveraging Codex in an agent-first world, is the caution about the wrong score. The first comment quotes the OpenAI piece: on the order of a million lines, about 1,500 merged pull requests, a team that grew from three engineers to seven, and 3.5 pull requests per engineer per day. Later comments ask whether line count measures the product. A self-improving loop pointed at pull-request count or lines of code will get more of those. Point it at the story's intent, then at cost per solved task. Self-learning agentic platforms is the longer piece on feeding a lesson back into the next run. The harness version of that idea is narrower: the lesson has to become a guide or a sensor, and the grader has to stay outside the rewrite.

What to build first

Order matters. Pod's sequence and Google's three-step loop agree on the start: name one failure, and name what done means, before you tune cost.

  • Write done as the user story. State the behaviour the change has to produce. A green suite can be one input. It cannot be the only one. If you cannot say what success is, a cheaper harness is not a win.
  • Add one computational sensor the coding agent cannot edit. A test, type check, or structural rule that lives outside the diff the agent is graded on. Böckeler's point, and the test-cheating cases, are the same rule.
  • Add one behavioural check for a miss you have already seen. Assert the action, not a composite score. Skipping the test run, guessing on an underspecified story, and editing the test file are all candidates. Batch it. Do not block the line on a single sample.
  • Record cost and time per solved task on a fixed model. Tan's 20-task table is the template: same tasks, same model, swap the harness, read cost and median time beside the pass rate. Do not rank a harness on a saturated pass rate.
  • When the same miss happens twice, change the guide or the sensor. Keep the evaluator outside that rewrite. If the checks for the operation are missing, fail closed.
  • Then cut waste. Bounded loops, smaller context, and traces of tool calls, cost and retries are Goyal's list. They are the efficiency work, and they come after the definition of done.

A team that only does the last bullet gets a faster agent and the same wrong merges. A team that only writes a longer instruction file gets guides with no sensors.

A harness at team scale

xSquads Software Factory is that practice on a delivery line. Engineers run coding agents in the cloud that raise pull requests from user stories. Review agents check each pull request against the story's intent, not only against passing CI, and they request fixes. Self-improving loops feed the lessons from those reviews back into the next run. That is a harness at team scale: a guide (the story), a sensor that is not "CI is green", and a loop that writes the miss back. Cloud agents are the runtime, off a laptop. The production line those agents sit on is a software factory. Run it at xsquads.ai.

xSquads for Work is the other surface. It gives every knowledge worker their own squad of specialised agents working together as a swarm, as an alternative to chatting with ChatGPT or Copilot. The desktop app is live on Mac and Windows and is free with Pro agents.

Where these facts come from

Read on 10 November 2026. The Reddit thread linked in the brief for this post did not load, so it is not used.

Frequently asked questions

What is harness engineering?

Harness engineering is the practice of designing the guides, checks and loops around a coding agent so the same model finishes the job you specified. The parts of that surrounding system are an agent harness. This page is the practice: what the new evals show, which controls to add, and what to build first.

Did the November 2026 harness evals separate accuracy?

No. On 20 public coding tasks, seven setups on Opus 5.5 or GPT-6.1 Sol had pass-rate point estimates from 71.3% to 78.8%. The g-harness-evals repository says those rates could not be told apart in any tested pair. Cost per solved task ran from $0.17 to $0.44, and median time from 30.1 seconds to 99.4 seconds. The repository calls the cost gaps descriptive, not a statistical test.

Can a harness move a benchmark score if the model stays fixed?

It can, and the size of the move depends on the board. An i10x analysis of public Terminal-Bench 2.0 results reports an 18.4-point spread for Claude Opus 4.6: 76.4% in one harness and 58.0% in another. The same analysis says the spread falls to 4.9 points on verified rows only, and that public submissions differ in budgets and prompts.

What is the difference between a guide and a sensor?

A guide steers the agent before it acts, such as a specification or a skill file. A sensor observes after the agent acts, such as a test or a linter message the model can read. Birgitta Böckeler, on Martin Fowler's site, writes that sensors alone repeat the same mistakes, and guides alone never check whether the rules worked.

What should a team build first?

Write done as the user story, not as a green test suite. Add one check the coding agent cannot rewrite, then one behavioural check for a mistake the agent has already made. Measure cost and time on the same tasks with the model held fixed. When that mistake shows up again, change the guide or the sensor, and keep the grader outside that rewrite.

How does a software factory act as a harness?

On xSquads, engineers run coding agents in the cloud that raise pull requests from user stories. Review agents check each pull request against the story's intent, not only against CI, and request fixes. Self-improving loops feed lessons from those reviews back into the next run. That is harness engineering at team scale.

Conclusion

Harness engineering is the practice of making a fixed model reliable: guides before the action, sensors after it, behavioural checks for the misses you have seen, and a score that includes cost and time. The 8 November evals show why the score has to be wider than pass rate. On 20 tasks the harnesses were not separable on accuracy, and they were far apart on dollars and seconds. Harder boards can move the score, and the 18.4-point Terminal-Bench headline shrinks to 4.9 points on verified rows. Build the definition of done first. Then measure. The line that runs this for a team is at xsquads.ai.

Raina Rufus, Founder of xSquads

Give yourself a squad of AI teammates

Your own squad of specialised agents to delegate to and review. The app is live on Mac and Windows. xSquads for Work is free with Pro agents.