AI does, humans review — is that actually reliable?

The previous two posts discussed two rather different boundaries.

The first started with piano practice. AI clearly knows a lot about piano, but actually teaching a person also requires going through Observe, Diagnose, Intervene, Verify: first seeing what really happened, then judging the cause, choosing a suitable intervention, and finally continuing to adjust based on the new results.

The second went on to ask: once AI can already do so many things for me, do I still need to know how myself?

In the end I divided it into three situations:

• some things only need a result, and can go straight to AI;
• for some, execution can be handed over, but I still need to keep enough judgment myself;
• and for some, the ability itself is the goal, and the part that really needs training can’t be outsourced too.

Following the second situation further, you run into a very common division of labor between people and AI today:

AI does it first; a person reviews.

Human in the Loop.

Building AI products myself, I’ve always agreed with this principle. AI can take on more and more execution, but for important judgments and higher-risk operations, a person still stands guard at the end.

That sounds very reasonable.

But after using it heavily, I increasingly feel I need to ask one more question:

can this person actually stand guard?

The first time I clearly ran into this was AI writing a PRD for me

Having AI put together a PRD from scratch is already very fast now.

Once the requirements are discussed, background, goals, flows, edge cases and acceptance criteria can quickly grow into a fully structured document. When the product changes later, AI can keep revising it.

Generation is no longer the bottleneck.

What’s really troublesome is review.

A document can quickly grow from 5 pages to 20, and the vast majority of the content has no obvious problems.

What’s really dangerous may be just one sentence hidden in some section.

For example, in the error flow AI added a line:

after a generation failure, automatically retry and overwrite the old result with the new one.

It reads as very reasonable.

But it has actually already decided several product questions: whether to make a second call, whether the old result is kept, whether the user knows a retry happened, and what to do after repeated failures.

If I skim past it, it’s easy to think:

edge cases are covered.

And keep going.

Formally, of course, someone reviewed it.

But the product decision that really needed a person to confirm may already have slipped through.

That made me care a lot, later, about one distinction:

Human in the Loop doesn’t equal Human Judgment actually happened.

The human–AI collaboration research I read in this round also made me realize this isn’t only a problem with my own PRD reviews.

A counterintuitive result: human + AI isn’t necessarily better than the human or AI alone

In 2024, Nature Human Behaviour published a systematic review and meta-analysis on human–AI collaboration.

The researchers gathered 106 experiments and 370 effect sizes. Every experiment had to compare three situations at once:

• the human doing it alone;
• the AI doing it alone;
• the human and AI doing it together.

The study distinguishes two easily confused concepts.

The first is called human augmentation:

is human + AI better than the human alone?

On average, the answer is: yes.

The second is called human–AI synergy:

is human + AI better than whichever of the human and AI was already better?

On average, the answer is: no.

A completely made-up set of numbers can help.

Suppose that on some task:

the human alone scores 60;

the AI alone scores 90;

the human and AI together score 80.

AI really did raise the human’s performance from 60 to 80, so it helped the human.

But if the goal is to do this task as well as possible, 80 is still worse than the 90 from letting the AI do it alone.

That’s the pattern this study found on average: human–AI combinations clearly beat humans working alone, but on average didn’t beat whichever of the human or AI performed better.

That’s also why I find the result so counterintuitive.

It’s easy for us to imagine:

people have common sense, experience and judgment; AI has speed, data and computing power; put the two together and they should naturally complement each other.

In reality, the combination doesn’t automatically produce complementarity.

The study even found that in decision tasks requiring a judgment from limited options, human–AI combinations on average showed a performance loss, while results for open-ended creative tasks were relatively more positive.

One boundary needs noting here: these studies date from 2020 to mid-2023, and of course can’t directly represent today’s latest generation of models.

But at least it’s a reminder:

“someone taking part” can’t be taken directly as evidence that a system will be more reliable.

Why might adding a person actually make results worse?

This meta-analysis didn’t prove a single causal mechanism, so I can’t say too much here.

But its results offer a clue worth noting.

The way people and AI work together on many decision tasks is actually very simple:

• AI gives a complete judgment;
• the person also forms a judgment;
• finally the person decides whether to accept or revise AI’s answer.

The problem is, if AI was already better at this task than the person, handing the final decision back to the weaker side won’t necessarily improve the result.

People may over-trust AI when it’s wrong, or distrust it when it’s right and change a correct answer into a wrong one.

So the researchers suggest that a direction more worth exploring than “both sides do the whole problem, then let the person choose” may be to split the task: people and AI each handle the parts they’re genuinely better at. But experiments like this are still rare, so it can’t yet be treated as a mature, validated answer.

That also made me rethink Human in the Loop.

The real question may not be:

whether there’s a person at the end.

But:

why should this step go to a person? What exactly does this person know, or do better, than AI?

Another more concrete example: AI being able to do it doesn’t mean people can do it with AI

In 2026, Nature Medicine published a randomized, preregistered experiment with 1,298 UK adults.

The researchers designed ten simulated medical scenarios and had participants judge what the problem might be and what action to take next.

Participants were randomly assigned to use GPT-4o, Llama 3 or Command R+, or to use their usual sources like search engines and the NHS website.

The researchers also ran a separate test: giving the complete cases directly to these models.

Handling the full scenarios on their own, the models identified the relevant conditions with an average accuracy of 94.9%.

But once ordinary people actually talked with these models, the share of LLM-using participants who correctly identified the relevant conditions was below 34.5%, and the share choosing the right next action below 44.2% — overall no better than the control group.

This can’t simply be read as:

people dragged AI down.

Because when tested alone, the models got complete, tidy cases from the start; in real use, the information first has to be described by an ordinary user, then probed and understood by AI, and finally the user still has to decide what to believe and what action to take.

What the researchers point to in the end is exactly this layer of user interaction: performing well on standard benchmarks doesn’t guarantee that real users relying on the same model will get equally good results.

I think this example is especially important for AI products.

Model capability is only one part of the whole system.

Between:

AI can do this problem

and:

a real person gets the thing done well with AI’s help

there’s a whole stretch of other things.

So when is “AI does, a person reviews” actually useful?

Having read this far in the research, I don’t conclude:

Human in the Loop is useless.

Quite the opposite — many high-risk scenarios really do need people.

But I’ll keep checking four more specific things.

1. Who reviews? Do they really know more than AI?

The first thing is the reviewer themselves.

Putting just anyone in doesn’t make it human oversight.

If AI generates a medical judgment, whoever really reviews it needs the corresponding expertise.

If a coding agent changes a complex system, whoever does the final technical review also has to really understand the relevant architecture and risks.

That sounds obvious, but behind it is a more important question:

• what is the person’s role, really?
• is it because they have expertise AI lacks?
• hold more business context?
• are responsible for value and risk trade-offs?
• or because, legally or organizationally, this person has to own the decision in the end?

If you can’t say clearly “why this person,” Human in the Loop easily becomes just a position in a process.

And finding the right person isn’t enough.

If an expert only sees AI’s final conclusion, without the original information and evidence, effective review is just as hard.

That leads naturally to the second question.

2. Review with what? Don’t let AI prove itself

This is what I care about most when using AI now.

AI gives a conclusion.

I ask:

are you sure?

It generates another, more complete and more persuasive explanation.

That isn’t independent verification.

It’s just the same system explaining its own answer once more.

If something can be verified through other signals, the more reliable approach is usually to go back to those signals.

• Code can run tests and show real runtime results.
• Data questions can be checked against the database.
• Research conclusions can go back to the original papers.
• Product behavior can be walked through for real.
• Amounts, permissions and state changes can be checked in system records.

So one precondition for Human in the Loop to be truly valuable is:

this person has evidence independent of AI’s output.

If all review can do is judge:

“does this passage sound reasonable?”

then as AI gets better and better at expressing itself, review may actually get harder and harder.

3. How do you review? “I’ve read it” doesn’t mean “review complete”

This is closest to my own experience with PRDs.

I used to tend to think of human review as:

after AI generates it, I read it through carefully.

But “reading it through” is actually a very low bar.

Truly effective review should at least be able to answer:

• what has changed?
• where have new judgments appeared?
• has high-risk content actually been verified?
• what is still only an assumption?
• which content do I explicitly accept, and why?

If a 20-page PRD has only two places that really need the PM to decide, the best review workflow shouldn’t be:

making the PM reread all 20 pages word by word every time.

It should, as much as possible, take:

what was added this round, which are new decisions, which were filled in by AI on its own, where the risk is highest, and where human confirmation is needed

and surface them directly.

That’s also why I increasingly feel:

making AI’s output reviewable is itself product design.

AI’s generation speed can increase dozens of times over.

Human attention doesn’t increase dozens of times along with it.

If human review workload is allowed to expand without limit along with the amount generated, you easily end up with a fake Human in the Loop:

in the process, someone looks;

in practice, they just skim.

4. What can be done after review? Does human oversight happen before the action, or after?

I’d actually been thinking about this a bit too loosely before.

The way I use coding agents now, after the code is changed I can review it, have it keep revising if I’m not satisfied, or refuse to merge.

In this case, the person really does have very clear control.

The problems show up more once real business actions start being executed automatically.

Say a customer service agent can issue refunds.

There are three completely different ways a person can take part.

The first:

before a refund is executed, anything above some threshold must first be approved by a person.

The person can actually stop the action.

The second:

while the agent is running, a person can pause or take over when something’s abnormal.

That’s intervention.

The third:

the agent has already completed a large number of refunds, and the next day a person is shown an audit report.

Here the person is taking part in an after-the-fact audit.

All three can loosely be called human oversight, but their effects are completely different.

The third is of course still valuable; it can find long-term problems, assign accountability and adjust the rules.

But it can no longer stop that wrong refund from just happening.

So when designing Human in the Loop, you also need to ask:

does this person approve before the action, can they intervene during execution, or can they only find the problem afterwards?

Once a problem is found, can it be rolled back, escalated or remedied?

Spelling out this layer is far more concrete than simply writing “a person is ultimately responsible.”

And not every AI problem should be patched by a person

Having done this round of research up to here, I think there’s another fairly important correction.

Human in the Loop easily becomes a cure-all answer:

AI might make mistakes?

No problem, add human review.

But different failures should be solved with different methods in the first place.

If the problem is missing information, add the information.

If neither AI nor the person knows what the other party really thinks, adding another reviewer won’t create facts out of thin air.

If the result can be objectively verified, build independent verification first.

If you can run tests, don’t rely entirely on someone reading code by eye.

If certain risks are already clear, set up permissions and guardrails.

For example, refunds above a certain amount must be approved by a person, rather than giving the agent unlimited permission and hoping the reviewer catches every mistake.

If what you’re actually facing is ambiguity, exceptions, or value trade-offs, then hand it to the right person.

Seen this way, Human in the Loop is just one layer of the whole reliability system.

A more complete system may need all of these at once:

• adding information;
• testing and evaluation;
• permissions and guardrails;
• human review;
• monitoring;
• the ability to recover after something goes wrong.

Reliability can’t all ride on “someone takes a look at the end.”

I now check Human in the Loop with four questions

This is a fairly practical framework this round of research left me with at the end.

From now on, when I see an “AI does, a person reviews” process, I’ll keep asking:

who reviews?

Do they have the knowledge, context and responsibility this judgment really needs?

Review with what?

Is there data, tests, sources or real results independent of AI to check against?

How is it reviewed?

Have the important changes and risks been surfaced? Does the reviewer have enough time and attention to really complete the judgment?

What can be done after review?

Does review happen before the action, during execution, or after the fact? Once a problem is found, can it be stopped, changed, escalated or recovered from?

If these four questions can’t be answered clearly, I won’t assume the system is much safer just because there’s one more human box in the flowchart.

Conversely, if they all hold, human involvement really starts to do its job.

After writing all three posts, my understanding of “what AI still does poorly”

The original question of this round of research was actually very simple:

what does AI still do poorly?

Some areas come to mind naturally:

art, emotion, bodily skills, subjective judgment.

The answer I got in the end wasn’t that tidy.

The first post, on piano, showed me:

AI having a lot of knowledge doesn’t mean it has observed the real situation, found the cause, and formed an effective feedback loop.

The second went on to find:

AI can do things very well for me, but its abilities don’t automatically become mine just because I’ve used them.

The third went one layer deeper:

even with execution given to AI and judgment left to people, the human–AI combination doesn’t automatically become reliable.

On the surface the three questions look rather different, but underneath they share one thing:

AI being very strong at one ability doesn’t mean the result we really want has happened.

• A model that can answer the questions doesn’t mean the student has learned.
• A model that can analyze a performance doesn’t mean it has really taught the learner.
• A model with strong output, plus a person involved, doesn’t mean the whole system is necessarily more reliable than either alone.

Between “AI can do it” and “it really gets done in reality,” there’s a lot in between:

• what information it got;
• what the task’s real goal is;
• how work is divided between people and AI;
• how the results are verified;
• where system rules should control risk;
• and where human judgment is truly needed.

So if you asked me again now:

what does AI still do poorly?

I might not start with a list of domains.

I’d first keep asking:

between “the model has the capability” and “the real-world result actually holds,” which link is still missing?

• Some gaps will slowly disappear as models get stronger.
• Some need better product and system design.
• Some are suited to people.
• And for some questions, neither people nor AI have enough information to answer yet.

The boundaries of AI’s capabilities will keep moving.

But I think knowing where to go looking for that boundary is more useful than memorizing today’s list of what “AI can / can’t” do.

About this round of research

This is the third and final post in the research series “What AI still does poorly.”

The whole round starts from real problems in my own use — in piano practice, learning, writing, relationships and AI products — then uses research across fields to test, refute and correct my original explanations.

This post focuses on the conditions under which human–AI collaboration and Human in the Loop really work.

The first set of core evidence comes from Vaccaro, Almaatouq and Malone’s systematic review and meta-analysis in Nature Human Behaviour. It included 106 experiments and 370 effect sizes, comparing the performance of humans alone, AI alone, and human–AI combinations. On average, human–AI combinations improved human performance, but didn’t beat whichever of the human and AI performed better; decision tasks on average showed a performance loss, while creation tasks were relatively more positive. The included experiments were mostly published between 2020 and mid-2023, so they can’t directly represent today’s latest models, but they’re enough to remind us: complementarity needs to be verified, and can’t be inferred from the “human + AI” structure itself.

The second study, published in Nature Medicine, was a randomized, preregistered experiment with 1,298 participants. It found that the same LLMs performed very well when given complete medical scenarios directly, but once in real human–AI conversations, they didn’t help ordinary participants reach better judgments than traditional information sources. The researchers therefore stress that model benchmarks can’t replace real user testing.

Putting the two studies together, what I think is most worth keeping isn’t:

“human–AI collaboration is unreliable.”

It’s:

what really needs evaluating is the whole human–AI system. What AI does, what people do, what information each side judges from, how errors are found, and how they’re stopped and recovered from.