Why eval design is becoming core to AI PM

In the previous three posts, I kept coming back to one question:

AI coding / vibe coding has made demos easy. But once a demo is built, how do you judge whether it really works?

The first post said: building it isn’t the same as it becoming a product. The second said: eval design isn’t grading AI; it’s defining what counts as right. The third used Asterline as a real worked example: how an AI feedback demo, through a Golden Set, Rubric, Review Log, Failure Types and Iteration History, turns output that “looks usable” into something checkable.

Having written that far, I’m more convinced: eval design isn’t a peripheral evaluation step in AI products; it’s part of how an AI PM manages quality.

Because what an AI PM has to deal with isn’t only:

has this feature been built?

But also:

When does this AI count as getting it right? What are the consequences when it’s wrong? Which errors are acceptable? Which must be stopped? When should it go to a person? How does the next version prove it really got better?

These are exactly the questions eval design answers.

1. The new problem AI PMs face: product output has become uncertain

Traditional software has complexity too, of course, with bugs and edge cases.

But the output of many traditional products is mostly produced by fixed rules: buttons, forms and filter results all execute according to preset logic.

AI products add a layer of uncertainty.

Because product output is no longer entirely produced by fixed rules, but generated by a model based on context.

The same feature can behave completely differently with a different batch of inputs. The same prompt can look fine for the first 20 samples and suddenly start talking nonsense on the 21st. The same agent workflow can get the first few steps right and then, at the last step, overstep its permissions, misjudge or over-promise.

So an AI PM can’t only ask:

Is the feature live? Does the flow run end to end? Is the page done well?

They also have to ask:

With which inputs is it reliable? With which inputs does it go wrong easily? When it’s wrong, will users notice? Can the system stop it? How does the team know the next version really got better?

The hard part of AI products isn’t only “bringing AI in.”

It’s managing its uncertainty once it’s in.

2. AI PMs don’t only define features; they also define “what counts as right”

Many AI features sound simple at first.

A customer service bot: answer user questions. Document Q&A: answer based on documents. An AI feedback tool: organize user feedback. An agent workflow: help users complete tasks.

But when you actually turn them into products, “getting it right” isn’t obvious.

A customer service bot isn’t right just because its tone is friendly.

It can’t make unauthorized refund promises. Can’t violate policy. Can’t just recite rules at an angry user. Can’t keep auto-handling a complaint that should go to a human.

Document Q&A isn’t right just because the answers are fluent.

It has to cite the right sources. Can’t mix information from several documents. Has to admit when it doesn’t know. Can’t cover thin evidence with a confident answer.

An AI feedback tool isn’t right just because its summaries are pretty.

It has to keep the original evidence. Separate the user’s words from product inferences. Know which feedback is just emotion, which may be a bug, which is a vague need. Can’t translate “this page is hard to find” straight into “users need global search.”

An agent workflow isn’t right just because it gets through the steps either.

It has to pick the right tools. Execute in the right order. Confirm before key operations. Stop when it lacks permission. Know how to recover after failing, instead of continuing to make things up.

None of these judgments is purely technical.

They involve user situations, business rules, risk boundaries, product responsibility and user expectations.

So “what counts as right” was always part of the product definition.

An AI PM who only defines features and not the boundaries of quality can easily end up with an AI feature that looks like it runs but that nobody dares to rely on.

3. Eval design is the AI PM’s language for quality

These days I’d rather understand eval design as a language for quality.

It lets the team stop saying only:

This AI works pretty well. This answer isn’t great. This agent is a bit unstable. This version feels better than the last.

All of that is too vague.

What really moves product iteration forward are more specific statements:

It got the classification wrong. It cited the wrong source. It missed a key business impact. It inferred a requirement from things the user never said. It didn’t trigger human review in a high-risk situation. It generated a promise that can’t be kept. This isn’t a prompt problem; the workflow can’t see the context it needs.

The value of eval design is breaking “is this AI reliable” into a set of questions that can be discussed:

What counts as right. What counts as wrong. Which errors are acceptable. Which must be stopped. Which outputs need evidence. Which situations need human review. How the next version proves it really got better.

Without this language, teams easily swing between two extremes.

One is optimism by feel:

“This demo already looks pretty good.”

The other is pessimism by feel:

“AI still isn’t very reliable.”

But the productized way of putting it should be:

In which situations is it good? In which is it unreliable? What kind of unreliable is it? What does this error cost? Should we change the prompt, add retrieval, add a checker, change the UX, or bring in human review?

An AI PM doesn’t necessarily have to train models personally.

But an AI PM must help define this language for quality.

Because a model can generate answers, but the product has to decide which answers can enter real use.

4. Eval design changes four parts of the AI PM’s work

Eval design isn’t a standalone step.

It affects product definition, launch acceptance, risk control and version iteration.

1. Product definition: not just writing requirements, but also success criteria

In a traditional PRD, we often write:

What task the user needs to complete. What flows this feature has. What states and exceptions there are. What metrics to watch after launch.

AI products need one more layer:

in this task, what counts as the AI getting it right?

For document Q&A, for example, it isn’t enough to write:

users can upload documents and ask questions.

You also need to define:

Answers must be based on the documents. Sources must be cited. When documents conflict, say so. When there’s no basis, don’t make things up. When users ask beyond the scope of the documents, admit not knowing.

These shouldn’t be added only at the evaluation stage.

Otherwise, once engineering is done, everyone starts arguing about “does this answer count as good,” and by then it’s too late.

2. Launch acceptance: not just whether the demo runs, but the edge cases too

AI demos easily perform well on ideal inputs.

Clear input. Complete context. Clear user intent. A question right in the model’s sweet spot.

But real users aren’t that cooperative.

They express themselves vaguely. Mix several questions together. Leave out key information. Ask things beyond the system’s abilities. Use the product while very emotional.

So when an AI PM does launch acceptance, they can’t only look at a few pretty demo cases.

They also need to look at:

Do the typical situations pass? How do the edge cases fail? Are high-risk situations stopped? When evidence is thin, does it admit not knowing? When user input is messy, does the system keep answering anyway, or stop and ask?

That’s the value of eval cases / a golden set.

It isn’t about making an exam, but about stopping the team from proving it’s got things right using only the smoothest inputs.

3. Risk control: not just chasing automation, but designing boundaries too

AI products easily make people want to keep automating.

Can it summarize automatically? Reply automatically? Create tasks automatically? Execute operations automatically? Complete the whole workflow automatically?

These directions have value, of course.

But an AI PM also has to ask another set of questions:

Where shouldn’t it automate? Where must there be evidence? Where must there be confirmation? Where is human review needed? Where should users know this is a draft, not a final decision?

Human review isn’t a patch.

It should be part of the workflow.

For example:

AI can organize user feedback, but customer replies involving money, policy or promised timelines must be reviewed by a person. AI can generate product suggestions, but they can’t automatically go onto the roadmap. AI can recommend next steps, but high-risk operations need confirmation first. AI can answer document questions, but without evidence it must say it doesn’t know.

A good AI workflow isn’t just able to do more.

It also has to know which things it can’t do on its own.

That boundary won’t grow out of the model by itself.

It needs product definition.

4. Version iteration: not just tuning prompts by feel, but looking at failure types and regressions

AI product iteration easily turns into:

Tweak the prompt again. Try another model. Add a few more examples. Make the output a bit more natural. Make the tone a bit more professional.

All of these may help.

But without evals, teams are easily fooled by “it feels better.”

A prompt change may improve some cases while making others worse. A more complex rule may sound more complete but actually introduce new misjudgments. A more natural reply may sound more human but start over-promising. A more proactive agent may be more efficient but more likely to overstep.

So an AI PM needs to look at failure types:

Is it misclassification? Wrong citations? Missing information? Over-inference? Failing to trigger human review? Generating processes that don’t exist? Or is the workflow itself missing context?

And look at regressions:

Is this version really better? Or do just a few samples look nicer? Has it got cases wrong that it used to get right? If it got worse, can it be rolled back?

That’s the iteration discipline eval design brings.

It means AI product iteration doesn’t rely only on feel, but has something to compare against, a classification of failures, and grounds for rolling back.

5. This isn’t something engineers can decide alone

Many people think evals are the technical team’s job.

The technical team matters a lot, of course.

Automated checks, dataset management, model evaluation, log monitoring, experiment frameworks — all of these need engineering ability.

But eval design can’t be left to engineers alone.

Because the most critical question isn’t:

what’s the model’s score?

It’s:

what counts as right for this task?

That question itself requires product judgment.

Whether a customer service bot can promise a refund isn’t only a model question; it’s a question of policy and responsibility. Whether document Q&A should answer questions beyond its material isn’t only a retrieval question; it’s a question of the boundaries of trust. Whether an AI feedback tool can summarize a vague complaint into a clear requirement isn’t only about summarizing ability; it’s about the PM’s workflow. Whether an agent can automatically perform an action isn’t only a tool-calling question; it’s about permissions, confirmation and risk.

Engineers can help you implement the checks.

But the PM must help define what to check.

Otherwise you easily end up with:

Technical metrics that look good. Model answers that are fluent. But product risks that were never defined, user situations not covered, and lines of responsibility never written down.

That’s exactly where an AI PM adds value.

Not doing the model evaluation in the engineers’ place, but putting users, the business, risk and system capability together to define quality standards that can actually be implemented.

6. So why is eval design becoming a core skill for AI PMs?

Because the core challenge for an AI PM isn’t only getting AI features built.

It’s putting AI’s uncertainty back into a manageable product system.

That system includes models, data, context, tools, permissions, checks, human intervention, user expectations, and business risk.

Eval design lets an AI PM say clearly:

what counts as AI getting it right in a specific situation. How errors are found. Where it went wrong. Which errors are acceptable. Which must be stopped. Which outputs need evidence. Which operations need confirmation. Which situations go to a person. How the next version proves it really got better.

That isn’t all of an AI PM’s job.

An AI PM still has to understand users, needs, the business, collaboration and trade-offs.

But as long as a product has model generation, agents calling tools, or AI taking part in judgments, it will run into uncertainty.

And eval design is one of the core ways an AI PM manages that uncertainty.

7. A practical landing point: beyond the demo, show your quality judgments

This isn’t the main thread of this post, just a practical landing point.

If eval design really is part of an AI PM’s skill set, then when showcasing an AI project, a demo alone may not be enough.

A demo proves:

this idea can run.

But it doesn’t necessarily prove:

you know where it’s reliable. You know where it isn’t. You know how to diagnose it when it’s wrong. You know which risks need human review. You know whether the next version is really better.

More convincing AI PM proof-of-work should show some of the quality judgment behind it:

What the task boundaries are. What the eval criteria are. What the failure types are. How human review is designed. Which versions got worse, and why they were rolled back. Which problems a prompt can’t solve.

This isn’t a packaging trick.

It’s part of the AI PM skill itself.

Because in real work, an AI PM has to keep answering these questions too.

Finally

After writing these posts, my understanding of AI PM has become more concrete too.

It isn’t knowing lots of AI terms. Isn’t writing a few prompts. And isn’t enough to build a demo.

What matters more is:

whether you can define what counts as AI getting it right in a specific situation. Whether you can find where it went wrong. Whether you can judge which errors are worth fixing and which must be stopped. Whether you can design human review and system boundaries. Whether you can prove the next version really got better.

The core skill of an AI PM isn’t plugging AI into a product.

It’s knowing where AI can be relied on and where it can’t, and how to make it more reliable, version by version.