What is eval design? Defining what "right" means

The last post said that AI coding / vibe coding has made “building it” easier, but hasn’t automatically turned anything into a real product.

Once something is built, you quickly run into another question:

is this AI output actually reliable?

When I first heard about eval design, I also thought it simply meant “evaluating how well the AI does”: run a few cases, see whether the output is good, and give a score at the end.

But the more I look into it lately, the more I think that understanding is too narrow.

What eval design really deals with is a more specific product question:

for a given task, when does the AI count as getting it right, when does it count as getting it wrong, and when it’s wrong, how do we know what to fix?

This way of thinking doesn’t only apply to user feedback. Document Q&A, customer service bots and agent workflows all run into similar questions: what is the task, what counts as right, how are errors found, how does the next version prove it’s better.

But to keep things clear, I’ll use an AI feedback tool as the running example in this post.

Not because eval design only applies here, but because it’s close to PM work, and makes it easier to see the gap between AI output that “looks good” and output that “can really be used.”

The first step isn’t scoring, but defining “what counts as right”

Suppose I built an AI feedback tool.

It’s meant to help a PM handle user feedback: classify, cluster and summarize it, then distill the product issues worth looking at next.

In the demo, it does well.

I give it 10 clean pieces of feedback and it summarizes:

Users find onboarding complicated. Users want a better export feature. Users have questions about pricing.

It looks quite convincing.

But the question is:

does that count as getting it right?

AI is very good at writing things that look good. Natural language, clear structure, even very much like a diligent product assistant.

But if I really want to put it into a product workflow, judging only “does it read right” isn’t enough.

I first need to define what counts as right for this task.

For an AI feedback tool, “getting it right” is at least more than “a nice summary.”

It should include several things.

First, telling types apart

Is the user reporting a bug, making a request, complaining, asking how to use something, or just venting?

This matters.

For example, a user says:

“Export fails every time. I’ve tried three times, my boss needs the report tomorrow, this is so annoying.”

This shouldn’t just be summarized as:

“The user wants a better export experience.”

It’s more like a bug / reliability issue, with an urgent business situation attached.

If the type is wrong at the first step, the summary afterwards will most likely be off too.

Second, finding the evidence

Every conclusion the AI outputs should ideally trace back to the original feedback.

For example, if it says:

“Users find the export feature unstable.”

it should be able to point to the corresponding words from users, rather than summarizing out of thin air.

Otherwise you easily end up with:

The user says A. The AI summarizes it as B. The PM understands it as C. And the product ends up building D.

Each layer drifts a little, and in the end nobody knows whose problem is being solved.

Third, telling facts from inferences

A user says:

“This page is hard to find.”

That’s feedback at the level of fact.

The AI can infer: maybe there’s an information architecture problem, maybe the entry point isn’t obvious, maybe search is needed.

But it can’t write straight away:

“The user wants a global search feature.”

That’s dressing an inference up as a requirement.

Good output should separate:

what the user said. What the AI inferred. What the possible product opportunity is.

These three layers mustn’t be mixed together.

Fourth, being able to move into the next action

Many AI summaries look complete, but the PM still doesn’t know what to do next.

For example:

“Users want a smoother experience.”

That isn’t wrong, but it isn’t much use.

More useful output would look closer to:

“Several pieces of feedback mention export failing or being unstable. First confirm which file types and browser environments the failures happen in; suggest adding error logs or user interviews.”

That could actually move into a next product action.

Fifth, knowing to stop when uncertain

Some feedback is too vague, the evidence is thin, or it may affect an important customer.

In those cases, the AI shouldn’t keep summarizing confidently.

It should flag:

this conclusion lacks evidence. This feedback needs human confirmation. This issue may be important, but the sample is too small. It isn’t appropriate to infer a product requirement directly here.

So for this AI feedback tool, “getting it right” probably isn’t “writing nicely.”

It’s:

telling types apart. Finding the evidence. Not inferring wildly. Supporting the next action. Stopping when uncertain.

That’s where eval design starts.

What exactly are an eval case and an eval set?

The terms sound a bit technical, but they’re not complicated.

An eval case is one sample used to test the AI.

In this example, an eval case can be one piece of user feedback, like:

“Export fails every time. I’ve tried three times, my boss needs the report tomorrow, this is so annoying.”

Then we look at how the AI handles it:

Did it identify a bug? Did it notice the repeated failure? Did it keep the business situation of “my boss needs the report tomorrow”? Did it over-infer something like “the export feature needs rebuilding”? Did it give an actionable next step?

An eval set is a group of eval cases.

For example, I prepare 20 pieces of user feedback as a small eval set.

These 20 shouldn’t all be clean examples.

Because a demo needs to show results, but an eval needs to expose problems.

So a more useful eval set should deliberately include some of the “messiness” you see in the real world:

Clear requests. Vague phrasing. Repeated complaints. Bugs mixed with emotion. Problems users themselves haven’t articulated. Feedback that’s rare but possibly important. Content AI easily over-summarizes.

This isn’t to make things hard for the AI.

It’s because real users don’t live by a demo script in the first place.

If an AI tool can only handle clean input, it can certainly make a nice demo. But it’s still some way from a real product.

An eval set isn’t a showcase; it’s a stress test

Demo cases and eval cases have different purposes.

A demo case proves: look, it runs. An eval case finds out: where will it break?

So an eval set shouldn’t only collect samples “the AI easily gets right.”

It should be like a small stress test.

In the AI feedback tool, for example, I’d deliberately include feedback that’s easy to misread:

“I can’t find the export button. Does this feature not exist?”

This may not be a missing feature, but an entry point that isn’t obvious.

“Uploading large files always freezes; small files are fine.”

This isn’t a vague upload-experience problem; it may be about file size or performance.

“It’s a bit pricey, but if it supported team permissions I’d consider it.”

This contains both hesitation about price and a signal of willingness to pay.

“This page is such a mess.”

This is very vague; the AI shouldn’t jump to a specific feature request.

The value of these cases is that they force the AI to expose its problems.

Did it really understand the user feedback, or did it just write something that looks a lot like a summary?

Why do failure types matter?

This is something I didn’t fully understand before either.

At first I thought the most important thing at the end of an eval was the score.

A pass rate of 80%, say. An average score of 4.2. Accuracy of 85%.

Those are useful, of course.

But for early AI products, knowing only the score isn’t enough.

Because a score can only tell you:

it isn’t performing well enough.

It doesn’t tell you:

what to fix next.

For example, if the AI feedback tool scores low, the reasons could be completely different.

One failure is misclassification.

The user is clearly reporting a bug, but the AI summarizes it as “wants a better experience.”

Then you may need to change the label definitions and add examples so it’s clearer about the differences between bug / feature / complaint.

One failure is missing information.

The user says “my boss needs the report tomorrow,” but the AI only extracts “export failed.”

Then you may need to change the extraction rules so it keeps the business situation and scope of impact.

One failure is over-merging.

Export failing, garbled exports, slow exports and not finding the export button all get merged into “export issues.”

Then you may need to adjust the clustering granularity.

Because they all relate to export, but aren’t necessarily the same product problem.

One failure is hallucination.

The user only says “the page is hard to find,” and the AI summarizes it as “users want a global search feature.”

Then you may need to require citations of the original evidence, and make it separate “the user’s words” from “product inferences.”

One failure is output that’s too generic.

The AI ends up writing:

“Suggest improving the overall experience.”

It doesn’t look wrong, but the PM doesn’t know what to do next.

Then you may need to change the output format so it must give a next action or a question to validate.

One failure is answering anyway when uncertain.

The feedback is vague, but the AI gives a very definite conclusion.

Then you may need to add conditions for human review, or allow it to output “uncertain.”

So failure types matter not because they sound professional.

But because:

a score tells you it performed badly. A failure type tells you what to fix.

With only a total score, you easily say:

let’s optimize it a bit more.

But once you know the failure types, you know whether to work on the prompt, the labels, the input structure, the clustering logic, the evidence citations, or the human review mechanism.

That’s where evals really help iteration.

A rubric spells out what “good” means

There’s another term you’ll run into here: rubric.

You can simply think of it as the standards of judgment.

But I’d rather think of it as: spelling out what “good” means.

Without a rubric, people easily evaluate by feel.

“This version’s good.” “This version isn’t great.” “This summary feels a bit hollow.” “This output seems OK.”

Remarks like these aren’t meaningless, but they hardly guide iteration.

For an AI feedback tool, a simple rubric could be:

Is the classification accurate? Is the conclusion backed by evidence? Is key information missing? Does it separate facts from inferences? Is the output actionable? Does it avoid inventing requirements? Does it flag uncertainty?

That way, “good or not” is no longer a vague feeling.

It’s broken into a few observable questions.

That’s also why eval design feels to me so much like a product skill.

It isn’t a purely technical exercise.

Because you have to first understand: who will use this AI output in the end, for what, and what the impact will be if it’s wrong.

If you don’t know what a PM will do with a feedback summary, it’s hard to design a good eval.

Can this approach generalize?

Yes, but carefully.

What’s general about eval design isn’t a few fixed metrics, but a sequence of thinking:

• first define the task.
• then define what counts as right.
• then prepare eval cases and put them into an eval set.
• then record the failure types.
• finally use the failure types to guide the next iteration.

But “getting it right” differs from task to task.

For document Q&A, the focus might be:

Is the answer based on the documents? Are the right sources cited? Does it admit when it doesn’t know? Has it mixed up information from two documents?

For a customer service bot, the focus might be:

Does it comply with policy? Does it make unauthorized promises of refunds or compensation? Does it recognize complaints, anger and high-risk situations? Does it hand off to a human when it should?

For an agent workflow, the focus might be:

Did it pick the right tool? Did it execute in the right order? Did it confirm before key operations? Did it overstep its permissions? Can it stop or recover after a failure?

So the AI feedback tool is just one example.

What really transfers is this line of thinking:

don’t first ask whether the AI is impressive. First ask what counts as right for this task.

Who evaluates is also part of the design

Evals don’t have to rely only on people, or only on AI, either.

Some things suit automatic checks: whether output fields are missing, whether classification labels are valid, whether the JSON is broken, whether original feedback IDs are cited.

Some things need human judgment: whether a product opportunity is over-inferred, whether a summary is really actionable, whether rare feedback is worth keeping, whether uncertain cases should go to human review.

You can also have another AI do a first pass against the rubric.

But I wouldn’t treat it as an all-knowing referee right now.

It’s more like a screening tool that saves time. Key samples and high-risk judgments still need a person to calibrate.

So eval design also designs:

Which judgments can be automated? Which can AI look at first? Which must a person confirm?

That’s really no longer simply “testing AI”; it’s designing the quality system for an AI product.

How can a very lightweight eval start?

If all you have now is an early AI demo, I wouldn’t build a complicated system at the start.

You can start with a very small version.

Say it’s still the AI feedback tool.

First prepare 20 pieces of user feedback as an eval set.

Don’t make them all clean examples. Deliberately include some normal, vague, repetitive, emotional and easily misread cases.

Then define 4–5 standards of judgment:

Is the classification accurate? Is there evidence? Is key information missing? Is there over-inference? Can the output be acted on?

Run the current version through it.

No need to automate at first; manual review is fine.

For each result, record just two things:

what it got right. What it got wrong.

Then categorize the errors.

Finally, look at:

What’s the most common failure type? Which errors most affect usability? Which errors are acceptable? Which must be stopped? Should the next version change the prompt, the workflow, the output format, or add human review?

The process doesn’t sound complicated, but it makes “optimizing an AI product” far more concrete.

Otherwise it’s easy to stay stuck at:

I feel this version is OK. I feel that version is better. I feel it could be tweaked a bit more.

Feelings are valuable, of course.

But an AI product run only on feelings soon gets lost.

So what is eval design, really?

If I sum it up in my own words now:

eval design isn’t giving AI a score; it’s designing a system of judgment for an AI product.

What it answers isn’t:

does this AI look smart?

But:

did it complete the specific task? When does it count as right? When does it count as wrong? What kind of error is it? How should that error be fixed? Did the next version really get better?

The AI feedback tool is just the example in this post.

Swap in document Q&A, a customer service bot or an agent workflow, and the method is still similar:

Define the task. Define what counts as right. Prepare eval cases. Put them into an eval set. Write a clear rubric. Record failure types. Use failure types to guide the next iteration.

It sounds like methodology, but behind it is a very plain question:

if an AI product looks like it works, how do we know it really works?

That’s why I want to keep writing this series.

In the AI era, making a demo is getting easier and easier.

But pushing a demo toward a real product takes more than more features or a prettier page.

It also takes a kind of judgment:

knowing what counts as right. Knowing where it went wrong. Knowing what to fix next. And knowing when you can’t pretend it’s already reliable.

In the next post, I’ll keep taking this AI feedback tool apart: if you really were to design an eval for it, which dimensions could you start from?