After vibe coding: a product, or the shape of one?

This is the first post in the Eval Design series.

But before getting to eval design, I want to record a question that comes before it.

Lately I’ve been building some AI demos and small products myself. Along the way, one feeling has grown stronger and stronger: AI coding / vibe coding really has made “building it” easier, but it has also made me ask myself more often:

is what I’ve built actually a product yet, or only the shape of one?

This isn’t to dismiss demos.

On the contrary, I increasingly think demos matter a lot. They let an idea stop living only in your head and become something that can actually be opened, operated and tested.

But precisely because “building it” is getting faster and faster, I’ve started caring more about another question:

what actually counts as starting to get close to a product?

Demos are valuable, but what they prove is “possibility”

It used to be that turning an idea into something interactive took design, front end, back end, database, deployment and debugging, pushed forward layer by layer. Now one person with AI coding tools, ready-made APIs, templates and hosted services can build something that looks quite complete in a few days, even a few hours.

It has pages, buttons, login, a database, and can even take payments.

From the outside, it already looks a lot like a product.

That’s a big step forward, of course.

But these days I look at demos and products a little separately.

A demo is more like proving:

this workflow is roughly feasible. This model can handle this kind of input. This interaction has a chance of working. This automated flow can at least run end to end once.

That’s already valuable.

But what a demo usually proves is: under a fairly ideal set of conditions, the thing can happen.

Once you really enter product territory, the question is no longer “can it run once end to end,” but:

when the inputs get messy, the situations get complex, and users don’t use it the way I expected, can it still reliably create value?

The two sound close, but they aren’t the same question.

Something being live doesn’t make it a product either. Being live only means other people can access it.

A product goes a step further: when users keep running into the same problem in real situations, do they naturally think of it, are they willing to come back to it, can they comfortably hand part of their judgment over to it?

For me, that’s the harder part.

AI products are especially easy to misjudge

There’s something very seductive about AI products: they’re very good at “looking useful.”

Give it a few clean inputs and it can summarize. Ask it a few questions and it can answer. Ask for structured output and it writes something quite convincing.

At first I was easily encouraged by that smoothness too.

But gradually you find that AI isn’t bad at producing. AI is too good at producing.

The real question is:

whether those outputs can enter a real workflow.

Take an AI feedback tool. In the demo you give it 10 clean pieces of user feedback, and it summarizes three problems:

Users find onboarding complicated. Users want a better export feature. Users have questions about pricing.

Looks good.

But real feedback is usually messier: bugs, complaints, duplicates, signals of willingness to pay and needs that aren’t spelled out all get mixed together.

If AI ends up summarizing it all as:

“Users want a smoother experience.”

that certainly looks like a conclusion.

But if I’m the PM, I still can’t use it directly.

I still don’t know:

where exactly is it not smooth? Is this a bug, an information architecture problem, or a request for a new feature? How many people mentioned it? Were they high-value users? Which original feedback is the evidence? Should the next step be to fix, ask, observe, or leave it for now?

That’s the difference I only slowly came to see:

“can generate” isn’t “can be used.”

So what’s really missing here may not be one more summary template, but a set of standards for judging:

What counts as classifying correctly? What counts as keeping the evidence? What counts as output that can move into the PM’s next action? When must something be marked as uncertain, instead of letting AI keep summarizing confidently?

That’s where I started paying attention to eval design.

It isn’t about making AI’s output look nicer, but about judging whether the output can actually be used.

The difficulty of a product hasn’t gone away; it has moved later

Building products used to involve a lot of difficulty in “getting it built.”

Now AI has driven that cost down. That’s great.

But I increasingly feel that the difficulty of a product hasn’t gone away; it has moved later.

It has moved from “can it be built” to:

once it’s built, does anyone really need it? Does it still work once real inputs get messy? Will users come back a second time? Can the output support action, not just a screenshot? Will errors be noticed? Is there a way to recover after a failure? How will the next version know it really got better?

These questions aren’t as exciting as a demo, but they’re very real.

An interface that runs only shows it can be opened. A product still has to slowly prove that people will keep coming back to it in real situations.

If something only performs well in the situations I prepared, it may be a decent demo.

If it can repeatedly help people get a specific thing done in real situations, it starts getting close to a product.

AI’s mistakes are more troublesome, because they often don’t look like mistakes

When traditional software breaks, a lot of the time it’s obvious.

The button doesn’t respond. The page shows an error. Data fails to load. The flow gets stuck.

When AI breaks, it often isn’t like that.

It gives you a wrong answer that’s fluent, complete, natural in tone, even well organized.

It may miss key facts. May summarize things users never said into needs. May merge two different problems into one. May write “uncertain” as if it were “certain.” May give a conclusion that looks very professional but can’t actually be acted on.

That’s also why I no longer judge an AI demo only by how it looks in a demonstration.

Because the most dangerous case isn’t that it doesn’t work at all. The most dangerous case is that it already looks like it works.

These days I remind myself: don’t rush to add features

After building many small AI products, my most natural urge is also to keep adding features.

Add login. Add history. Add export. Add more models. Add a prettier page. Add more automation.

All of these may be useful, but they aren’t always the most critical question.

Sometimes what you should ask first is:

when does it count as getting it right? Which errors must be stopped? After changing a version, how do you prove it really got better?

That’s what draws me to eval design.

It isn’t giving AI an exam. It’s helping us answer a very plain product question:

is this AI output actually reliable?

Or, more specifically:

In what situations does it count as getting it right? In what situations does it count as getting it wrong? When it’s wrong, how is that discovered? How does the next version prove it’s better?

If vibe coding solves “can it be built,” eval design is about “once it’s built, how do you judge whether it can be used for real.”

Three questions I use to check my AI demos

If you’re also building AI products or AI demos, or seriously pushing a vibe-coded project forward, maybe hold off on asking “what else can I add.”

These days I first ask myself three questions.

First, does it solve a problem that keeps recurring?

A one-time wow isn’t enough. A product needs repeated situations.

Users don’t open it once out of curiosity; they come back the next time they hit the same kind of problem.

Second, once real inputs get messy, can it still produce usable results?

Don’t only test clean examples.

Real users don’t live by your demo script. They express themselves messily, leave information out, contradict themselves, bring emotions, and raise edge cases I never thought of.

Third, how do I know the next version is really better than the last?

This may be the weakest point of many small AI products, and it’s something I’m still learning myself.

If you’re only tuning prompts by feel, it’s easy to fall into an illusion:

this version sounds more natural, so it’s better.

But maybe it’s just better at talking. Accuracy didn’t improve. Hallucinations didn’t decrease. Errors got harder to spot. Even cost and latency went up.

Without standards of judgment, iteration easily turns into guesswork.

What this post really wants to record

AI has made “building it” cheap.

That’s a big deal. It lets more people turn ideas into reality, and it amplifies what an individual builder can do.

But it hasn’t removed the difficulty of products.

It has just brought us to the next question sooner:

once it’s built, how does it get used for real? Once it’s used, how do we know it’s reliable? When it isn’t, how do we discover, correct and iterate?

That’s why I want to keep writing about eval design.

It isn’t a new buzzword to sound professional; it answers a very plain question in AI products:

did this thing actually get it right, and is the next version really better?

As more and more people can build the shape of a product, what becomes truly scarce may be the ability to judge.

In the next post, I’ll keep going: what eval design actually is, and why it isn’t simply “grading AI.”