Evals, Harness, Forward-Deployed: new kinds of work

Evals, Harness, Forward-Deployed: AI hiring is growing kinds of work that didn’t exist before

This is the third post in the series “Who is hiring AI PMs right now.”

The first two posts did two things:

the first broke AI PM down into several completely different jobs;

the second looked at where the so-called “technical requirements” of different roles actually differ.

Having organized things this far, I noticed another quite interesting change:

AI product teams themselves are redividing the work.

As of August 20, 2026, among the 43 public AI product roles I collected, besides the familiar AI Product Manager, some names rarely seen a few years ago kept appearing:

AI Evaluations, Agent & Harness, Model Behaviors, Human Data Platform, Safeguards, Forward-Deployed…

Seeing these words at first, it’s easy to think:

is the AI industry inventing new buzzwords again?

But after going through the JDs line by line, I actually think their appearance is quite natural.

As an industry starts to mature, one thing often happens:

a problem someone used to handle “on the side” slowly becomes complex enough to need a clear owner.

AI is going through this process now.

1. Evals: someone starts owning “what counts as good”

Evals are easily understood as:

the AI version of QA.

But there’s a very fundamental difference from traditional testing.

In ordinary software, we can often write down in advance:

what the input is, and what the correct output should be.

Not necessarily with AI.

The same question can have many reasonable answers; a version may improve overall while regressing in some key scenarios; a high average score can coexist with an occasional completely unacceptable mistake.

So the first thing an eval has to answer isn’t even:

did the test pass?

But:

what actually counts as good?

Only then come:

what to test; where the test set comes from; how the rubric is defined; what can be judged by rules; what can use LLM-as-a-judge; what must be reviewed by people; how to run regressions when the model version changes; how new failures seen in production get back into the test set.

Box now has a role literally called Product Manager, AI Evaluations.

The PM owns datasets, graders, offline / online evaluation, model selection and so on.

At this point, evals are no longer just “testing once the product is done.”

They start to affect:

how models are chosen, how the product iterates, and when it can go live.

So I now lean toward understanding evals as:

the quality infrastructure of AI products.

Even if not every company ends up with a dedicated eval PM, this capability will very likely make its way into more and more ordinary AI PMs’ work.

2. Harness: beyond the model, a whole layer decides whether an agent is any good to use

Harness is a concept I’ve only recently started understanding more seriously.

Different companies draw its boundaries a bit differently, so I don’t want to give it a very rigid definition.

If I had to put it plainly in one sentence:

the model “thinks”; the harness lets it “get things done” in a real system.

Once an agent is really running, it often also needs:

context; memory; tools; permissions; a sandbox; orchestration; retries; fallbacks; observability; evaluation.

These things used to be easily seen as engineering details.

But once an agent no longer just answers a sentence, and starts to:

call tools, query databases, change files, submit requests, take real actions,

they become product questions directly.

For example:

what exactly can this agent see?

Which tools can it call?

Which operations must require confirmation?

If it fails halfway through, does it retry, roll back, or hand over to a person?

Why did it make this decision, and is there a trace to look at?

Binance already has, outright:

Product Manager, AI Agent & Harness

The JD explicitly covers task flow, capability boundaries, failure fallback, harness evaluation, task sets and more.

I find the job title itself quite interesting.

It shows that some companies no longer treat the layer around the agent as:

“just hook the model up to a few APIs.”

But are starting to treat it as a product system that needs designing and running over the long term.

3. Model Behaviors: even “how the model should behave” is getting a product owner

Anthropic currently has another role I find very novel:

Research Product Manager, Model Behaviors

Model capability used to be easily thought of as:

how high the math benchmark is; how strong the coding is; how broad the knowledge is.

But in real use, there’s a set of questions much harder to express with a single benchmark:

when it doesn’t know, should it admit it?

Faced with a vague request, does it guess directly, or clarify first?

When should it refuse?

How should it express uncertainty?

Will it over-accommodate to make users happy?

What should it do with conflicting instructions?

All of these strongly affect the product experience.

In other words, the “product object” PMs are responsible for in the AI era is extending further forward.

Before, we mainly designed:

pages, flows, features.

Now someone is even specifically responsible for:

how the model should behave.

And this takes research, data, evals and product judgment working together.

4. Human Data Platform: human feedback is becoming a product system too

There have always been lots of “people” behind AI.

Someone has to judge:

which of two answers is better;

whether this result is risky;

whether the agent really completed the task;

which kind of error is more serious.

When the volume is small, it can be handled by hand.

Once it scales, a whole set of problems quickly appears:

how are tasks designed? Who’s suited to judge? Are different people’s standards consistent? How is quality checked? How are costs controlled? Which feedback goes into training? Which goes into evals?

Anthropic now has product management roles related to its Human Data Platform.

I think this change also says a lot:

human feedback is no longer just a back-office operations step.

It too needs:

a platform, processes, quality standards, tools and product design.

In other words:

even the “human” in “human in the loop” is being systematized.

5. Safeguards / safety measurement: safety turning from principle into metrics and systems

Talk of AI safety used to easily stop at:

AI should be safe.

True, of course.

But when actually building products, that sentence is far from enough.

You also need to keep asking:

what specific harms are there?

How are they detected?

Do the safeguards actually reduce risk?

How high is the false positive rate?

Which risks are acceptable?

Which must be blocked?

When an incident happens, how does it feed into the next round of product iteration?

OpenAI now has a Safety Measurement PM.

Anthropic’s hiring has gone further, splitting out:

rare harms, child safety, vertical safeguards and other directions.

By this point, safety is no longer just policy.

It’s:

metric + system + operations + product decision.

I think this is also an area where AI products differ very clearly from ordinary software.

Many “value judgments” ultimately have to be translated into:

how exactly the system works.

6. Forward-deployed PM: someone dedicated to getting the demo into the real world

Forward-deployed is a kind I paid less attention to before this research, but have come to think more and more important.

Scale AI is currently hiring a Forward Deployed Product Manager.

This kind of PM doesn’t just:

gather requirements → write the PRD → go back to headquarters to plan the roadmap.

They go deep into the customer’s site:

understanding the business problem; connecting data and APIs; building a prototype together; testing; deploying; then slowly abstracting one-off customer solutions into reusable standard products.

So it’s a bit like:

PM + solutions architect + consultant + builder.

Why do AI companies increasingly need people like this now?

Perhaps because building a demo today isn’t that hard anymore.

What’s really hard is:

the customer’s data is messy; the permission system is complex; the process has lots of exceptions; nobody in the organization wants to change how they work; model performance starts dropping once it hits the real scenario.

Where many AI projects end up stuck is no longer:

the model isn’t smart enough yet.

But:

it never really entered the workflow.

Forward-deployed is really dedicated to solving this “last mile.”

7. AI transformation: someone starts owning a company’s AI product portfolio

Another new division of labor happens more inside traditional enterprises.

Imagine a company suddenly having:

dozens of AI ideas; lots of departments each running their own pilots; a pile of vendors coming to demo; everyone saying their scenario is especially suited to AI.

Very soon someone needs to answer:

what should be done first?

What’s worth building?

What should simply be bought?

Which capabilities should be built once as a platform?

Which data can be used?

How should permissions be governed?

How is ROI calculated?

How do you get employees to actually adopt it, rather than everything ending after the demo?

So these are increasingly common now too:

AI Transformation PM, AI Product Owner, Internal AI Product.

They aren’t necessarily responsible for a single AI feature.

It’s more like running:

how a company should use AI.

What I think it has in common with forward-deployed is:

once AI reaches this stage, the model itself is only part of the problem.

Organization, process and implementation take up a larger and larger share.

Put these new roles together, and there’s a very clear logic

I later tried ignoring the job titles and looking only at why they exist:

models change → so you need evals

agents act → so you need harness / permissions

model behavior itself affects the experience → so Model Behaviors appears

AI depends on lots of human judgment → so a Human Data Platform appears

AI creates new risks → so you need safeguards / safety measurement

enterprise deployment is hard → so forward-deployed appears

a company sprouts lots of AI projects at once → so you need AI transformation

Seen this way, these names no longer look so much like “AI jargon.”

They’re all answering the same question:

once a new complex problem appears, who should own it for the long term?

This has a very practical effect on job hunting too

If you only search for:

AI Product Manager

it’s actually easy to miss opportunities that suit you well.

Now you can also look at:

AI Evaluations · Agent Platform · Agent Harness · Model Performance · Research Product Manager · Model Behaviors · Safeguards · Forward-Deployed Product Manager · AI Applications · AI Transformation

Of course, these titles haven’t fully settled yet.

Some names may still be around in a few years; some may disappear.

So rather than memorizing every word, I think what’s more useful is:

first look at what problem it’s solving.

Because titles will change.

Problems usually don’t disappear that fast.

With three posts written, this research’s map of the market is basically laid out:

the first is what jobs AI PM actually breaks into;

the second is how deep the technology needs to go for different roles;

and this one is which new specializations AI product organizations are growing.

In the last post, I want to bring the perspective back from “what companies are hiring for” to the individual:

if you weren’t an AI PM before, facing so many roles and skills, how should you actually find your place?

I think the answer may not be:

adding an even longer study list to yourself.

Part 3/4 of the series “Who is hiring AI PMs right now.” The research is based on 43 public AI product roles collected as of August 20, 2026; this post mainly draws on current postings from Anthropic, OpenAI, Box, Binance, Scale AI and others. The job titles mentioned represent emerging trends in the division of labor, and don’t mean these titles have become a unified industry standard.