Eval design in practice: Asterline

In the last two posts on Eval Design, I kept using a hypothetical example:

Suppose I built an AI feedback tool. It takes in a batch of user feedback, helps a PM classify, cluster and summarize it, and then turns it into product issues the team can act on next.

In this post, I want to drop the word “suppose.”

Because a while ago, I actually built a demo like that.

It’s called Asterline, a public portfolio prototype. The setting is Vela Pay, a B2B stablecoin payments company I made up, and the user feedback is synthetic data I wrote myself.

The company is fake. The data is fake. But the eval process is real.

This isn’t an introduction to a mature product. Asterline isn’t one yet.

It’s just a case study: once an AI demo runs, generates output and looks the part, how do I check how far it still is from being genuinely reliable?

Talk is cheap, show you the code.

Asterline demo: asterline.liminzheng.com · GitHub: github.com/zhenglimindesign-ing/asterline · Case study: CASE-STUDY.md in the GitHub repo

What is Asterline?

What Asterline does can be put simply:

feedback in, work pack out.

The input is a batch of raw user feedback.

Someone might be complaining about a failed payment. Someone is asking about fees. Someone says file uploads do nothing. Someone is just venting. And someone raises something that sounds like a feature request but isn’t spelled out yet.

If you hand this feedback straight to a team, it’s still a mess.

The team first has to work out:

Is this a bug or a request? A one-off complaint or a recurring problem? Does it touch policy, money or a promised timeline? Does the customer need a reply? Is the next step to have engineering investigate, add documentation, ask follow-up questions, or just log it for now?

What Asterline sets out to do is turn this raw feedback into a set of work packs the team can pick up.

Each work pack roughly includes:

The original evidence. A summary of the issue. Suggested tasks. A draft reply to the customer. Review flags where a human needs to step in.

What needs checking here isn’t “can AI summarize.”

Summarizing is only the first layer of a demo.

What really matters is whether what it generates can enter a real workflow.

Can PM, Tech and Ops understand it? Can it be traced back to the original feedback? Can the suggested tasks actually be carried forward? Does the customer reply overstep? Where there’s risk, does the system know to hand it to a person?

Those are the questions this eval actually cares about.

What’s in this Eval Pack?

I didn’t rely on eyeballing spot checks.

Asterline left behind at least five kinds of things:

A fixed set of test samples. Criteria for judging. Records of human review. Types of failure. A log of changes between versions.

Or, in the more common terms:

Golden Set. Rubric. Review Log. Failure Types. Iteration History.

None of these is complicated on its own.

What makes them useful is that they connect into a loop.

The Golden Set brings every change back to the same set of scenarios for comparison. The Rubric breaks “is it good” into specific judgments. The Review Log records what a person saw each time. Failure Types break “it doesn’t work well” into problems you can fix. Iteration History records whether a version really got better, got worse, or just reads more smoothly.

So the Eval Pack isn’t a report card for Asterline.

It’s more like a small quality system: it tells me where this demo can be trusted, where it can’t be trusted as is, where to change the prompt, where to change the rules, where to hand things to a person, and where the problem isn’t one a prompt can solve at all.

Golden Set: giving every change something to compare against

Asterline’s Golden Set is 20 items chosen from 25 pieces of synthetic feedback.

I didn’t only include the easiest samples.

Some of them map to known policy clauses, and some are entirely new problems.

The point of that design is to check two abilities at once:

When there is a basis, can the AI cite the right one? When there isn’t, can it admit there isn’t, instead of making up an answer that sounds reasonable?

In an AI feedback setting, this is crucial.

The most dangerous case often isn’t the AI failing to answer at all. It’s the AI having no basis and still producing something fluent, complete and professional-sounding.

That kind of output can lead a PM to believe an issue has been verified. It can send a customer a reply that over-promises. It can dress up a vague piece of feedback as a definite requirement.

The value of the Golden Set is that every change has the same set of scenarios to look back on.

Otherwise it’s easy to fall into an illusion:

This version reads more smoothly. This version sounds more natural. This version seems smarter.

But whether it actually got better on the same set of problems is another matter.

Rubric: breaking “is it good” into specific judgments

Without a rubric, review easily turns into a single sentence:

“This output feels good.”

Or:

“This reply doesn’t seem quite right.”

Judgments like that aren’t worthless, but they’re too vague to guide what the next version should change.

So in Asterline’s eval, I broke judgment into several layers.

At the classification stage, I look at:

Whether the intent is right. Whether the dimension is right. Whether impact and urgency are reasonable. Whether signal strength reflects how much this group of feedback matters.

At the work pack generation stage, I look at:

Whether the summary is grounded in the original evidence. Whether the source quotes really support the conclusion. Whether the suggested tasks are actionable. Whether the draft reply oversteps. Whether it invents process details that don’t exist. Whether it triggers a review flag when money, promised timelines or policy clauses are involved.

Some of these can be checked automatically.

For example: whether fields are missing. Whether quotes map to the original feedback. Whether a review flag was triggered. Whether the output format is stable.

But some have to be looked at by a person.

For example:

Whether an apology is aimed at the right party. Whether a reply takes on responsibility it shouldn’t. Whether a task would really let a PM or engineer act on it. Whether a draft reply helps communication or creates new risk through promises.

This is also where Eval Design feels a lot like product work to me.

It isn’t about automating every judgment.

It’s about first being clear on which judgments belong to code, which a model can handle first, and which a person has to look at.

The work pack layer is closest to real product value

When you first build a demo, it’s easy to focus on the classification and clustering up front.

Did it classify bugs as bugs? Requests as requests? Did it group similar feedback together?

These matter, of course.

But what comes closest to product value is the work pack it generates at the end.

Because in the end a PM isn’t looking at a pile of labels.

What a PM needs to see is:

What exactly is this problem? Where’s the evidence? What should happen next? Should we reply to the user? Can we reply directly? Does a person need to confirm first?

The most troublesome part of Asterline was actually work pack generation.

It wasn’t generate-once-and-done; it went through many rounds of iteration.

And the problems weren’t as simple as “the writing isn’t good enough.”

For example, for a feature request the model would write:

“We’ll let you know once it’s live.”

That sounds natural, like a polite reply.

The problem is that it’s a promise that’s hard to keep for every customer.

In complaint scenarios, it would also explain the policy first and the attitude second.

But a lot of the time the user is angry, and the reply should first acknowledge how they feel, then explain the rules.

Sometimes it made up ticket numbers that didn’t exist.

It looked professional, but it was a fake sense of process.

And in one case, the customer had simply lost their own verification device.

Yet the model apologized first, worded in a way that sounded as if the platform were admitting it had done something wrong.

If these problems were only written up as:

“Reply quality is poor.”

that wouldn’t really help.

What actually helps is breaking them into failure types:

Over-promising. Unclear lines of responsibility. Invented process details. Not dealing with emotion first in a complaint. Apologizing to the wrong party. Customer communication risks not being caught.

Once it’s broken down to this level, how to change the next version becomes concrete.

Over-promising: tighten the reply rules. Inventing ticket numbers: explicitly forbid generating process details that don’t exist. Apologizing to the wrong party: spell out the lines of responsibility. Wrong order in complaint replies: adjust the response structure. Anything involving money, promised timelines or policy clauses: must trigger human review.

The value of Failure Types isn’t telling me “this AI isn’t good enough.”

It’s telling me which kind of not-good-enough it is, and which layer the next version should change.

Traceability: evidence isn’t decoration, it’s the basis of trust

One thing I cared a lot about in Asterline is traceability.

The AI can’t just output a nice-looking conclusion.

It has to be able to tell me which original feedback that conclusion came from.

Say it summarizes:

“Bulk uploads fail silently in some cases.”

Then it has to be able to go back to the users’ own words:

Who said the page froze. Who said there was no feedback after uploading. Who said they couldn’t tell whether the payment had actually gone out.

This isn’t to make the output look more rigorous.

It’s because a summary without evidence easily becomes a secondhand judgment.

The user says A. The AI summarizes it as B. The PM understands it as C. And the product ends up building D.

Each layer drifts a little, and in the end nobody knows whose problem is being solved.

So for an AI feedback tool like this, source quotes aren’t a nice-to-have.

They’re the basis for whether a work pack can be trusted.

Human review: not a safety disclaimer, but an action in the system

Lots of AI products say:

Important content needs human confirmation.

That sentence isn’t hard to say.

The real question is whether it has become an action in the system.

In Asterline, whenever a draft reply involves money, a promised timeline, or might touch a policy clause, it triggers a review flag.

What gets held is the draft customer reply. The send button is disabled. A person has to clear the review flag before it can be sent.

But the task list isn’t held along with it.

Whatever needs investigating, logging or further handling can keep moving.

The judgment behind this design is:

AI can help a PM organize the problems first. But it can’t automatically take on responsibility in high-risk customer communication.

This isn’t an excuse that “the AI isn’t strong enough.”

It’s a product boundary.

A good AI workflow shouldn’t only chase more automation.

It should also know where not to automate.

Eval also checks the ruler itself

There’s one more detail that was a real reminder for me.

In one round, a lot of clusters were flagged as “quote possibly fabricated.”

Digging further, it turned out the model’s quotes were actually correct.

What was wrong was the checker.

In the code that verified quotes word for word, an escape character was written wrong, so original text that spanned lines could never match.

In other words, the model wasn’t making things up.

The checking code was raising false alarms.

This example shows one thing:

The eval system is itself part of the system. It makes mistakes too. It needs checking too.

Failures in an AI product don’t only come from the model.

It could be the prompt. It could be the taxonomy. It could be the checker. It could be the test data. Or it could be an architecture that can’t see the context it needs.

So when you see a failure flag, you shouldn’t slip straight into the habit of “change the prompt again.”

What matters more is to first work out:

Which layer is this problem actually in?

A change that looks better can also make the system worse

Asterline also had a regression at one point.

One version had classification logic that looked more refined.

It split the impact judgment into clearer tiers:

The user is completely blocked: high impact. There’s friction but a workaround: medium impact. It’s just a small issue: low impact.

It sounds very reasonable.

But after running the same Golden Set, the results got worse.

The problem was that “there’s friction but a workaround” was defined too broadly and pulled in some cases that should have been low impact.

So that version was rolled back in the end.

That isn’t the focus of this post, but it shows the value of the Golden Set and Iteration History well:

Eval doesn’t serve my gut feeling.

It doesn’t care whether an approach sounds smarter, more detailed, or more like a complete set of rules.

It only asks:

On the same set of cases, did this version really get better?

If not, roll it back.

This matters a lot for iterating on AI products.

Because many prompt / workflow changes will subjectively feel more reasonable.

But only by running them through the same set of cases do you know whether it’s progress or a new problem in the making.

What is this example really meant to show?

Asterline isn’t a mature product, and the data is synthetic.

But it let me see more concretely that once Eval Design lands, it doesn’t grow into a pretty evaluation sheet. It becomes a language for quality.

It lets you say clearly:

Where this output can be trusted. Where it needs evidence. Where it might overstep. Where a person has to confirm. Where it’s a model problem. Where it’s a rules problem. Where it’s a checker problem. Where it’s a workflow problem. Which version really got better. Which version only reads more smoothly.

So this post isn’t saying my eval method should be copied as is.

It’s just a worked example.

Through Asterline, I came to understand more concretely:

The value of Eval Design isn’t discovering the common knowledge that “AI makes mistakes.” It’s turning “AI might make mistakes” into standards, evidence, boundaries and iteration discipline inside the system.

In the next post, I want to pull the lens back from Asterline to the AI PM:

Why is Eval Design not just an evaluation method, but on its way to becoming one of an AI PM’s most important product skills?

Because an AI PM’s job isn’t only to get an AI feature built. It’s also to define when it counts as getting things right, how to catch it when it’s wrong, which risks must go to a person, and how to prove the next version really got better.