What is harness engineering?

Recently I read an OpenAI write-up from practice, “Harness engineering: leveraging Codex in an agent-first world.” It records how a small team used Codex to build a real product from an empty repo, and why they gradually shifted their focus from “getting AI to write more code” to designing project material, tools, rules and feedback mechanisms for the agent.

My first reaction was:

coding agents can already read projects, change code, run commands, even check problems on their own. Why invent yet another concept called harness engineering?

Reading on, I think what the term is meant to remind us of is actually something very plain:

whether an agent can get the work done doesn’t depend only on the model itself.

It also depends on:

what project information it knows; what tools it can use; what boundaries it mustn’t cross; and how it checks the result once it’s done.

Put together, these are the harness around the model.

The harness isn’t the model, and isn’t just the prompt

The prompt mainly tells the agent:

what to do this time.

The harness deals with a different set of questions:

under what conditions does it get this done?

Say I ask a coding agent to fix a sign-up problem.

With only a one-line prompt, it might:

search the relevant files; find a function that looks suspicious; change the code; run a few tests; tell me it’s fixed.

But it may not know:

the sign-up flow has different user types; some operations run as async jobs; the local environment doesn’t really send email; after the change you need to go through the full sign-up flow; this fix might break other features that were working.

This information doesn’t automatically appear in the project just because the model got stronger.

The agent still needs to know where to look, and needs a way to observe the real results.

What exactly does harness engineering design?

I now understand it through four questions.

1. What does the agent know?

What’s the product goal? How are the directories organized? Which decisions have been confirmed? How far along is the current task? Which document is the latest?

If key information lives only in chat history, outside documents or someone’s head, an agent working in the project may not be able to see it at all.

2. What can the agent use?

Can it only edit files, or can it also:

start the app; operate a browser; read logs; query test data; run the full flow; compare results before and after a change?

The more complete the tools, the less the agent has to guess from the surface of the code whether the product works.

3. What must the agent obey?

Which files can’t be changed? Which data can’t be accessed? Which operations need confirmation first? When should it stop and hand over to a person?

Some requirements can be written in the project instructions; the more important boundaries may need to be truly enforced by permissions, checks or product flows.

4. How does the agent prove it’s done?

Finishing the code change isn’t finishing the task.

It may also need to:

run a fixed set of tests; walk through a real user flow; confirm existing features weren’t broken; provide logs, screenshots or a change record; keep a version that can be rolled back.

Without these conditions, an agent easily mistakes “I’ve changed the code” for “the problem is solved.”

Why did OpenAI start talking about harness engineering?

OpenAI shared an internal experiment: the team had Codex generate the code, tests, docs, CI and internal tools for a software product.

They found that when early progress fell short of expectations, it wasn’t only a matter of model capability. The project lacked tools, structure and feedback suited to an agent. Later, the engineers’ work shifted more and more toward designing the environment, expressing the intent of tasks, and building feedback loops that let the agent check and correct its results.

That doesn’t mean every personal project should copy OpenAI’s complex setup.

OpenAI also states clearly that the autonomy their agents achieved depends heavily on structure and tools designed specifically for that codebase, and can’t simply be assumed to emerge naturally in other projects.

But the experiment still brings out a useful judgment:

model capability is one part; the working conditions around the model are another.

When does the harness matter more?

If you’re just having an agent change a line of copy or tweak a style, and the result is easy to check by eye, the harness can be very light.

But when a task starts to show these features:

spanning many files; continuing across several sessions; needing different tools; having broad operating permissions; errors being hard to spot; wanting the agent to work independently for longer;

project information, tools, boundaries and verification methods become more and more important.

Anthropic ran into similar problems studying long-running coding agents: each new session can be like a shift change with no memory. If the last round didn’t leave clear progress and project state behind, the new agent has to guess what happened.

My most useful takeaway from this

From now on, when a coding agent doesn’t do a task well, I won’t only check two things:

is the model not strong enough? Was the prompt not written well?

I’ll also check four more:

can it see the information it needs? Does it have the tools the task requires? Does it know which boundaries it mustn’t cross? Does it have a way to verify results and recover from errors?

Harness engineering isn’t saying every agent problem can be solved by building an environment.

It just fills in a perspective that’s easy to overlook:

AI being able to write code doesn’t mean it automatically has everything a real project requires.