Harness engineering and the AI PM

After learning about harness engineering, I had a very direct question:

this clearly sounds more like engineers’ work, so why connect it to AI PMs?

That feeling isn’t wrong.

Harness engineering involves code repos, tool integrations, tests, logs, permissions, version control and runtime environments. Actually building these things is usually led by the engineering team.

It doesn’t start naturally from a product question the way eval design does:

when does this AI count as getting it right? Which errors are acceptable? Which must be stopped?

So I wouldn’t call harness engineering a core skill every AI PM must master.

Its relationship to the PM depends on what AI is actually responsible for in the product.

If AI is only responsible for generating some content

For example:

rewriting some copy; summarizing an article; generating a few titles; providing a first draft.

The PM still needs to define quality standards, of course, but may not need to be deeply involved in a complex harness.

Because AI’s range of action is small.

If the result isn’t satisfying, you can edit or regenerate it; even when it’s wrong, the impact usually stays at the level of content.

If the agent starts calling tools and taking action

Things are different.

Say an AI customer service agent doesn’t just answer questions but can also:

look up accounts; read orders; change ticket status; request refunds; send replies to customers; hand problems to other teams.

Now the product can’t just define:

the user raises an issue, and AI handles it automatically.

It also has to answer:

Which account information can the agent see? Up to what amount can it issue refunds directly? When must it get the user’s confirmation first? Which issues must go to a human? How does it recover when execution fails? When handing off to a human, what context should be kept?

These questions will ultimately be implemented by engineering systems.

But they can’t be decided entirely by engineers on their own, because they involve business policy, user expectations, lines of responsibility and trade-offs of risk.

That’s where AI PMs and harness engineering meet.

What’s the difference between eval design and harness engineering?

Let’s keep using the AI customer service agent as the example.

Eval design might define:

replies must comply with the refund policy; it can’t make up order status; it can’t promise compensation that doesn’t exist; account security issues must go to a human; without evidence it can’t answer confidently.

That is:

which results count as right, and which count as wrong.

Harness engineering goes on to handle:

where the agent reads the latest refund policy; which account fields it can query; whether it has permission to issue refunds directly; under what conditions a refund automatically pauses; which checks a reply passes through before being sent; how existing information goes along when handing off to a human.

That is:

when the agent is actually working, what it can get, how far it can go, and what checks and limits are in between.

You can see how the two connect through one concrete change.

Suppose the eval uncovers a high-risk failure:

the agent occasionally promises users compensation that hasn’t been approved.

Just recording it as a failure type isn’t enough.

The team might then decide to:

• tighten the reply rules;
• check whether replies contain promises of compensation;
• prohibit automatic sending when money is involved;
• require a human agent to confirm;
• add such cases to the fixed tests.

Here, the eval helped the team discover and define the problem.

The harness turns part of that judgment into checks, permissions and stopping conditions the agent actually runs into while working.

So I’d now rather put the relationship like this:

eval design helps the team identify which errors matter; harness engineering decides how to reduce, catch or handle those errors through information, tools and system boundaries.

They overlap, but they aren’t the same concept.

Which part does the PM actually need to be involved in?

The PM doesn’t necessarily have to build a test framework, or configure the code repo.

But if the product’s agent will read data, call tools or act for users, the PM should at least help define five things.

1. Information

What does the agent need to see when making decisions?

Which sources are trustworthy? When materials conflict, which wins? Which data shouldn’t be opened up to it?

2. Permissions

Can it only make suggestions, or can it execute directly?

Can it draft replies, or send them automatically? Can it recommend refunds, or actually issue them?

3. Boundaries

Under what circumstances must it stop?

When information is insufficient, should it ask? Does a high-risk operation need confirmation first? After repeated failures, must it go to a human?

4. Completion

When the agent says “done,” what evidence does it need to provide?

An operation log, cited sources, test results, or the final state?

5. Failure handling

If the agent misjudges, a tool call fails, or it executes only halfway, what happens next?

Can it be undone? How are users told? How is the work so far handed to a person to continue?

Will harness engineering become a core skill for AI PMs?

I wouldn’t make that a blanket conclusion now.

For PMs mainly responsible for simple AI generation features, it may just be an engineering concept worth knowing about.

But for people responsible for products like these:

agents; internal enterprise assistants; automated workflows; AI that reads real business data; AI that can call tools and take action;

understanding harness engineering matters more.

Because at that point, what the PM designs isn’t only what AI outputs.

It also includes:

what it acts on; what permissions it has; where it must stop; how it proves it’s done; how, after failing, it hands control back to a person.

Implementing harness engineering is very much engineering.

But the decisions within it about information, permissions, boundaries, acceptance and human intervention were always part of product design.