Eval design

Vibe coding has made “building it” cheap, but the difficulty of a product hasn’t gone away; it has moved later. A demo proves something is possible; a product has to prove it keeps creating value when inputs get messy and users don’t behave as expected.

I used to think eval design meant running a few cases and giving AI a score. Now I think what it really has to answer is: for a specific task, what counts as right, what counts as wrong, and what to fix when it’s wrong.

I took Asterline, a demo I built, through the whole Eval Design process. The company and the data are fake, but the iteration loop — from Golden Set and Rubric to Failure Types — is real.

The hard part of AI products isn’t only bringing AI in; it’s managing its uncertainty. Eval design answers exactly “what counts as right, which errors must be stopped, and how the next version proves it’s better,” so it’s a core skill for how an AI PM manages quality.