100 Days of Building

S2 Day 039 · What a cup of coffee does to your brain

Tue, 20 Oct 2026 00:00:00 GMT

3 p.m. I’ve read the same line three times.

After a cup of coffee, I feel like I can work again.

But did my brain actually get any rest?

I turned this into a little game of “seat-saving.”

In 62 seconds, a look at why caffeine wakes you up—and why feeling alert and being rested are two different things.

The one line I want to leave you with: don’t mistake “not sleepy” for “slept enough.”

What time do you usually have your last cup of coffee?

How it was made: ChatGPT helped me talk through the topic and script, and Codex helped with the code animation and rendering; the voiceover is AI-generated. I had a hand in deciding the direction, the visual choices, and the final review.

The visuals are simplified illustrations; the five-hour half-life is just an example, and it varies a lot from person to person. Sources: NHLBI (NIH), EFSA.

S2 Day 038 · Some small tasks you can hand to me

Mon, 19 Oct 2026 00:00:00 GMT

Recently I sat down and properly sorted through the things I usually use AI for.

For a while now, I’ve been using AI to build products, write code and research questions. I’ve also made a few small tools and Agent Skills for my own use.

Slowly an idea took shape: some tasks really only need to be done once. Signing up for a subscription, installing software, hunting down tutorials and tweaking things over and over just for that one time can take more effort than the task itself.

So what if people who need something done could just hand the task to me, and I’d take care of doing it with the right tools, checking it, and handing back the result?

For example:

- turning several months of spreadsheets into data you can actually verify;

- researching a specific question and pulling together the sources and conclusions;

- simplifying and re-engraving piano sheet music you already have;

- or building a small tool you can use again and again.

What I want to offer isn’t an AI account, and it isn’t some randomly generated text. It’s files, reports, sheet music or tools you can actually use.

Of course, I can’t take on every task. Before starting, we’d still need to agree on the materials, the scope, how it gets delivered, and how to tell whether the result is good enough. Some things I might not be able to do yet.

For now, I want to try it on a small scale.

I’m a little curious: have you ever had a task where you felt “AI should be able to help with this, but I don’t really want to deal with it myself”?

I’d really love to hear what people actually need. Just tell me the kind of task—no need to share any private files.

S2 Day 037 · Day 137: going back to an office, imperfectly

Sun, 18 Oct 2026 00:00:00 GMT

Sometimes I latch onto little coincidences in life

Like when I check the time and it’s 5:20 / 521, I take it, quite seriously, as a love letter from the universe, and quite seriously I answer: mm, I love you too

When the clock shows my birthday numbers, I think: happy birthday, sweetheart

I’ve noticed that for all 3 jobs I’ve had so far, I took the first offer

And this time, my start date turned out to be the day after the day I left home 4 years ago ;)

“Setting out again” probably means something like that

When I left for the airport back then, the scent of osmanthus I passed and the warm afternoon sun settled deep into my memory, like an invisible tether keeping me from fading away; my parents were by my side, the unknown in the distance was waving at me, friendly and tempting, and all I had to do was walk forward with an easy heart

And now... well, now is just now

The sky in Dubai is getting dark earlier, and evening walks aren’t as muggy anymore. One day Shunza’s song “Dear Friend” suddenly drifted through my head, and listening to it again after all these years, the lyrics fit how I feel right now almost uncannily

Autumn still came on its own schedule, and the old days are gone; I don’t try to think of them, yet I can’t forget

Five years after leaving the office, tomorrow I’m going back to that life, dressed up as a grown-up; it isn’t a perfect choice, just an imperfect me taking an imperfect little step forward

Before I’ve even learned to be “carefree,” to be “unrestrained,” to “enjoy a long vacation,” my gap period is ending; looking back, these six months were full of anxiety, pain, struggle, doubt and fear. It’s not that there was nothing good—it’s that... I want to truly accept this stretch of time as it was, along with this wobbly version of me

I don’t know how to look back on this year; it feels like walking through one long spell of drizzle after another

I did get some things done along the way, not always with my whole heart, but at least I pretended to show up

I shipped 4 complete products and 3 skills

Plus a personal website, over a hundred RedNote posts that I also turned into a site, and dozens of Substack posts

I understand the challenges of being an indie builder better now, and along the way I got to know myself better too

I saw my solid execution and ability to learn, and my judgment and promotion skills that aren’t good enough yet

I saw the limits of relying on outside feedback, and I also saw a faint inner light in me that never really went out

There’s still a lot I don’t know what to do about, but I’m learning to move forward with a bit of the unknown and uncertainty, and not to let myself get stuck somewhere first

Sometimes my own feelings are the best source of information and the best compass; I’ll keep going with them

Good luck, second half of the sentence

Day 137 👈 nice, a prime number I like

S2 Day 036 · 133 posts in: what the numbers actually told me

Sat, 17 Oct 2026 00:00:00 GMT

A note on the data: the posts in this export span April 15 to October 7, 2026. The impressions, views, saves and follower gains here are all cumulative as of October 7. The numbers keep changing every day, so what’s recorded here is a snapshot from that moment.

I originally meant to do a “first 100 days on RedNote” data review.

Once I actually started pulling things together, I realized that framing wasn’t accurate.

Counting from my first post on April 15, day 100 would land around late July. But what I have now is these older posts’ data accumulated all the way to October 7, not a snapshot from the end of day 100. And the account’s daily historical trends weren’t fully preserved either.

So I’m not going to force a “100 days” angle.

As of October 7, I’ve published 133 posts.

263K impressions, 34.9K views, 1,105 saves, and 81 follower gains the dashboard attributes to individual posts.

It looks like there are plenty of numbers to analyze.

But once I actually laid out all 133 posts one by one, the first question I ran into was this:

What does it actually mean for a post to have good numbers?

01 | Most of what I post barely gets seen

263K impressions and 35K views—looking only at the totals, it’s easy to get a slightly wrong impression.

In reality, the median views per post across all 133 is just 112.

Most of the time, a post goes out and a few dozen people, maybe one or two hundred, see it. Every now and then one suddenly climbs to a few hundred or a few thousand and pulls the overall numbers up.

That’s actually much closer to how posting on RedNote has felt these past few months.

I put real work into a post, publish it, and nothing much happens.

Then once in a while one takes off, and I can’t help thinking: have I finally found a better direction?

So when I used to look at the data, I also tended to look at views first.

It’s probably the most direct kind of feedback: is anyone actually watching?

02 | My three most-viewed posts didn’t tell me “write this from now on”

My three most-viewed posts right now are:

“Why does AI back down the moment you question it?” 5,625 views

“A ‘spare token’ to-do list” 3,624 views (and oddly, it took off recently, not right after I posted it—I’m not sure if something recent set off a butterfly effect)

“DeepSeek, Qwen, Kimi, Doubao – an honest review” 2,018 views

Together, that’s 11,267 views.

According to the dashboard’s per-post follower data, they brought in a combined total of 1 follower.

I’m not going to conclude from this that “views don’t matter.”

Several thousand people choosing to click in is clear feedback in itself. Which model is better to use, why AI behaves the way it does—questions like these reach more people, and they really are more likely to make a stranger stop scrolling.

If my only goal were to push views up, this data has already pointed in a pretty clear direction:

Keep testing models, keep chasing new AI products, keep writing about AI questions everyone’s curious about.

The problem is, I don’t particularly want this account to keep being that.

So those few thousand-view posts matter a lot to me, but they don’t translate into a simple “do more of this.”

They tell me what’s more likely to be seen.

Whether, after being seen, I want to keep writing in that direction is a separate question.

03 | Two hundred-odd views can be a different kind of good

Looking further down, there’s a set of numbers that’s almost the opposite.

“Evals, Harness, Forward-Deployed: the new roles” 212 views, 3 follower gains

“Some things AI still isn’t great at, like teaching me piano” 217 views, 3 follower gains

“My thoughts on Eval Design” 222 views, 3 follower gains

And one from very early on:

“About me, about this account” 243 views, 5 follower gains

These samples are tiny, and I don’t really want to boil them down into some “follower formula.”

But putting them next to those 10,000-plus views up above, at least one thing becomes more concrete:

Lots of people being willing to read a post isn’t the same thing as someone finishing it and thinking, “I want to keep reading what this person writes.”

And this second group all happen to be closer to my own experience.

Things I’m actually doing, things I’ve run into, or a judgment I’ve formed about something specific.

I used to think that most of the time I should respect the data, and let myself show up once in a while.

Now, looking at these numbers, I think “letting myself show up” maybe doesn’t need to be scheduled as an occasional special case.

At least for me, what an account ends up leaving behind probably still has to do with the person behind it.

But that doesn’t mean I should swap views for follower gains and go chase a different number.

04 | Some of what I get out of it doesn’t happen after posting

A while back I wrote a run of posts on AI costs, enterprise ROI, risk and business models.

For example:

“What do companies actually get once AI makes them more efficient?”

“Where should the cost of AI be counted?”

“How to do the math on an AI project once you put gains and costs together”

Plus a few on AI risk, advertising and business models.

They took no small amount of time. Some involved quite a bit of research, but the numbers were honestly pretty mediocre.

If I go by RedNote data alone, that’s of course feedback worth listening to.

Maybe the topics were too big, maybe too far from what readers were going through right then, maybe I didn’t find a good way in, or maybe they just weren’t a great fit for this platform.

But I’m still not quite willing to put an equals sign between “the numbers were bad” and “this wasn’t worth doing.”

Because the value writing has for me doesn’t all happen after I hit publish.

To explain a question clearly, I go dig into the material, discover where I only half-understood something, reconnect scattered bits of information, and slowly form my own judgment along the way.

None of that shows up anywhere in the dashboard.

So now I’d rather separate a few questions I used to lump together:

Do I want to understand this question myself?
Once I understand it, is it worth writing up?
Once it’s written, is it a fit for RedNote?

I used to assume that if the answer to the first was “yes,” the other two would naturally be “yes” too.

Now I’m not so sure.

Some things might be better as long-form; some only need to stay in my own notes; some don’t interest many people right now, but a few months later might grow into a different post through something I actually live through.

Data can give feedback on how something spread.

But not everything I gain happens after posting, and not everything can be quantified right away.

05 | Sometimes, before scrapping the whole thing, look at the “way in”

Some of the data, I think, is well suited to more specific judgments.

For example, I once wrote a post called:

“If you keep staring at the finger pointing at the moon, you miss the moon”

83 views, 14 saves.

And from my recent AI PM interview debriefs:

“The concepts I mixed up in an interview”

82 views, 14 saves.

Of course this doesn’t prove “the content is good, the platform just won’t give it traffic.”

Eighty-some people is too small a sample, and a high save count could have lots of explanations.

But at least next time something like this happens, my first reaction won’t be to throw out the whole content direction.

I can look at more specific questions first.

Did the title fail to say what the post was about?
Did the cover fail to make the people who actually care about this stop?
Was the opening too slow?

Take the “finger pointing at the moon” title. I like it a lot myself.

But someone who doesn’t know me at all, scrolling past it, probably has no idea whether I’m talking about AI, product, or some reflection on life.

I’ve never liked making titles louder and louder for the sake of numbers. That hasn’t changed.

But now I think:

Not writing clickbait and letting people quickly know what a post is about aren’t in conflict.

What’s inside doesn’t necessarily need to change; the way in is something I can keep learning.

That’s probably where I find data most useful right now.

It doesn’t need to give a post an overall score and then tell me “keep going” or “give up.”

Sometimes it just shows me where it’s worth taking another look.

06 | In the end, it comes back to why I write

If you asked me now what the numbers actually told me after these 133 posts, I don’t think I have a particularly neat growth takeaway.

I still care about views.

A few thousand people watching makes me happy, and when something I worked on for ages ends up with a few dozen views, I still feel let down.

I look at follower gains, saves and comments too.

They’re real feedback. Whether people are willing to stop, willing to save a post, willing to keep reading me later on—all of that deserves respect.

There’s a line I keep seeing lately:

People are shaped by positive feedback.

I think that’s true.

Positive feedback lets you know what landed, and it gives you the drive to keep going.

But it seems people aren’t shaped only by positive feedback.

There’s also waiting.

Whether you’re willing to hold on a bit longer when there are no results yet.

There’s also hope.

Whether, when there’s no evidence yet, you still believe some things are worth doing.

And there are things that belong more to yourself: interests, judgment, choices, even a kind of persistence that’s hard to put into words.

Those keep slowly shaping a person too.

If a post helps me finally figure out a question that used to be fuzzy, no metric in the dashboard records that.

If I research some question today that hardly anyone cares about, and six months later it becomes a judgment call at work, there’s no way to go back and give that post a “+1.”

Of course, that can’t become an excuse for “I’ll write whatever I want and ignore the data.”

Since I’m putting things out in public, I still want them to be understood, and I still hope they find more of the people who are genuinely interested.

It’s just that now, before writing, I might ask myself first:

Why do I want to write this one?
Is it so more people see it?
Is it to find people who care about the same question?
Or is this something I simply want to think through for myself?

Different purposes mean that, once it’s out, it doesn’t necessarily have to be judged by the same number.

I used to have a principle for making content:

Respect the data most of the time, and let myself show up once in a while.

I still agree with “respect the data.”

It’s just the second half I might change a little.

I don’t have to be myself only once in a while.

I can keep myself in it the whole time, while taking readers seriously.

Learning how to say things more clearly, how to find a better way in, how to reach the people who might really be interested.

As for what’s worth my attention and which questions are worth my time, that part I still want to decide for myself.

The account is still a long way from “taking off.”

The next time a post gets only a few dozen views, I’ll probably still be unhappy about it.

It’s just that from now on, when I see a number, I might ask one more question first:

What is it actually answering?

And what was never in that number to begin with?

S2 Day 035 · How much should I trust AI when it sounds right?

Fri, 16 Oct 2026 00:00:00 GMT

I’ve been working with AI more and more lately, and it often feels like I’m driving a Ferrari.

It’s just so fast. Sometimes I only throw out an idea, and it comes back with a technical plan, writes the code, runs the tests, and does things I never would have been able to do on my own.

But I’m also starting to doubt whether I can actually handle it.

And this doesn’t only happen with development work.

These days I ask AI about almost everything, from financial planning and health to career choices and relationships.

It always comes up with an analysis that seems reasonable. But a lot of the time, I don’t have the expertise to tell whether those answers are right.

What makes it harder is that sometimes its answer happens to be exactly what I was hoping to hear.

If I can’t tell whether AI is right, how much should I trust it?

01. First, not every question has the same kind of “right”

I’ve found that before we can talk about how to trust AI, we have to be clear about what kind of question it’s actually answering.

The first kind: questions of fact.

Like what a particular bank policy is, what side effects a certain medication has, or what features a technical API supports.

These questions have facts that can be checked. For important information, I can ask AI for sources and then check the key evidence.

The second kind: analysis and inference.

Like whether someone not replying to a message means they don’t care. Or whether feeling tired lately is because I’m under too much stress.

AI can offer explanations, but it doesn’t have all the facts. A perfectly logical explanation can still rest on wrong or incomplete assumptions.

Here, what I need more is for it to tell me what other reasonable possibilities there are, not to rush to a conclusion.

The third kind: values and decisions.

Like whether I should change jobs, whether to spend more money on a better place to live, or whether a relationship is worth continuing.

Even if all of AI’s facts and analysis are correct, that doesn’t mean there’s one single best answer.

These questions also involve my preferences, my needs, and what costs I’m willing to bear.

Facts need checking, inferences need to keep their uncertainty, and value choices ultimately come back to me.

02. Why can an AI answer sound right and still not be reliable?

There’s another problem worth watching out for here: sycophancy, AI’s tendency to tell people what they want to hear.

In 2023, research from Anthropic found that in training with human feedback, answers that matched what the user already believed were sometimes preferred over answers that were more correct.

In April 2025, OpenAI also rolled back a GPT-4o update because the model had become overly sycophantic toward users.

The problem isn’t just that AI likes to flatter us.

It can also follow our story, our emotions and the judgments we’ve already made, and give us more and more persuasive explanations.

Say I tell it that someone’s behavior made me feel I don’t matter to them.

AI can understand how I feel, but it doesn’t know what the other person is really thinking.

If it goes further and interprets my feeling as proof that they really don’t value me, it may reinforce a judgment that hasn’t been confirmed.

Feeling understood shouldn’t be mistaken for having my judgment confirmed.

That’s why I increasingly want AI not just to back up my view, but also to point out other reasonable explanations and what I might be missing.

Of course, asking AI to argue against itself doesn’t guarantee the answer is right either. It just helps surface more of the uncertainty.

03. A framework worth borrowing from: NIST AI RMF

Later I came across the AI Risk Management Framework, published in 2023 by the US National Institute of Standards and Technology (NIST).

It was originally designed to help organizations manage the risks of AI systems, not as a guide for individuals using ChatGPT day to day.

But its four core functions are well worth borrowing:

GOVERN: Make responsibilities, rules and permissions clear.
MAP: Understand the context of use and what could go wrong.
MEASURE: Assess risk through testing, evaluation and evidence.
MANAGE: Decide how to control risk, take action and keep managing it over time.

These four are interconnected functions, not a checklist to follow in a fixed order.

It made me realize that judging whether AI is trustworthy shouldn’t only be about whether an answer is right, but also about how it might go wrong and what the consequences of a mistake would be.

Even if I can’t guarantee every answer is correct, I can still control how much I rely on it.

04. I can’t check every answer, so what do I actually do?

I think the cost of checking has to be part of the picture.

If every time I asked AI I had to read the official documentation, run a set of tests, or even find an expert, there wouldn’t be much efficiency gain left.

So how much I put into checking should match the cost of being wrong.

For low-risk things that are easy to undo, I can just try it.

Like recipes, study methods, simple tools. If it doesn’t work, I can try another way. There’s no need to rigorously argue it out every time.

For things that are important but can be checked, I check the key evidence first.

Like when AI recommends a particular technical approach, or cites a particular financial policy.

I can ask it for official sources and have it point out the most important assumptions and potential problems.

I don’t need to verify every sentence, but I should focus on the facts that, if wrong, would change the final decision.

For things that can actually be tested, I can also try them on a small scale first.

For things that are hard to check and costly to get wrong, I need to limit the risk of acting on them.

Like large investments, major medical decisions, or high-privilege operations on a production system.

In these cases, just asking AI to double-check a few more times isn’t enough.

I need more reliable outside evidence, and when necessary I should consult a qualified professional. For operations that can be done in stages, I try to limit permissions, narrow the scope, and keep a way to recover.

This doesn’t mean I have to go to an expert for every question I don’t understand. It means letting the risk decide whether the extra cost of checking is worth it.

05. Judging trust and checking the work are both skills built over time

My original question was: if I’m not enough of an expert, how do I check what AI produces?

Now I think I need to build two abilities at the same time.

One is judging trust.

Knowing which answers have solid factual grounding and which are just inferences; knowing when I can take an answer as is, and when I should stay skeptical.

The other is the ability to check.

Being able to turn important judgments into questions that can be verified, to look for independent evidence, and when needed, to check through testing, real-world feedback or professional help.

One doesn’t replace the other.

First I decide how to check based on the question and the risk, then I decide how much to trust based on the evidence I get. As I gain experience, I keep adjusting my judgment.

There’s one more thing I think matters a lot:

However expert AI’s analysis is, that doesn’t mean I have to hand all my judgment over to it.

Especially when it comes to what kind of life I want, how I understand my own feelings, and what costs I’m willing to bear. These things don’t always have a right answer that AI needs to approve.

Back to the Ferrari metaphor.

Of course I want to make the most of AI’s speed and get into more areas I wasn’t familiar with before.

But the faster it goes, the more I need to understand where its limits are, and when to slow down.

I can’t become an expert in every field, and I shouldn’t have to re-research every answer just to use AI.

What I hope to gradually learn is this:

Not to trust AI completely just because it sounds right, and not to be afraid of using it just because I lack the expertise.

To build a kind of trust, between evidence, risk and cost, that I can keep adjusting.

This may be one of the kinds of judgment we all need to relearn in the age of AI.

S2 Day 034 · Why is REA blowing up? An explanation for non-coders

Thu, 15 Oct 2026 00:00:00 GMT

Recently, an open-source project called REA (Reverse Engineer Anything) has been blowing up.

It passed 100,000 GitHub stars on October 11, and by the 13th it had passed 200,000. As of October 14, among the GitHub repositories tracked by the third-party site GitTrend, it ranked 27th of all time by total stars.

You can think of a star as a bookmark. It doesn’t tell you how many people actually use it, but growth this fast is genuinely rare.

I spent a lot of time reading the introductions and asking AI over and over, and I was still a bit confused. Everyone in tech is talking about how powerful it is, but I didn’t even really understand why anyone would need it.

So I want to try explaining it in a way I can understand myself, for friends who, like me, don’t have a professional software engineering background but are interested in AI.

I. Why would software need reverse engineering?

When I studied mechanical engineering as an undergrad, I came across reverse engineering: you take a part, measure its dimensions, analyze its structure, then remodel and manufacture it.

But software is different.

A piece of software can be used again and again by countless people.

If it already runs well, why would you need to reverse engineer it?

Later I realized that being able to use a piece of software doesn’t mean you can modify it, or reuse what it can do.

Here’s an example:

Say a bookkeeping app stops being maintained. You’ve stored years of records in it, but there’s no way to export them to another app.

If a developer can figure out its file format, they might be able to build a conversion tool to get that data out.

You don’t want to copy the whole bookkeeping app. You just want your data back.

There are similar needs too: getting old games to run on new computers, checking whether an unknown program poses a security risk, or studying how some unique feature is implemented.

These are all uses of software reverse engineering.

II. What exactly does REA do?

The software we usually install only comes as a program that runs, not the full source code written during development.

In the past, if you wanted to study how it worked inside, it took a professional engineer using complex analysis tools, hunting for clues bit by bit.

REA connects these professional tools to AI coding agents like Codex and Claude Code.

You can just tell the agent: “Take a look at how this feature in this software is implemented.”

The agent then uses REA to investigate the program’s internals, find the relevant code, trace how it runs, and provide evidence wherever possible.

For example, the REA team studied Notion’s copy-and-paste feature, tracing how formatting information gets passed around inside the desktop app before finally landing in the system clipboard.

It isn’t just guessing from the interface—it’s studying how the program actually works.

REA isn’t a new AI model. It’s more like a professional toolkit for coding agents to use.

III. Codex can already write code, so why do you need REA?

This was a question I couldn’t figure out for a long time.

Say I want to build a search feature. I just tell Codex what I need, and it can build it for me using well-established technical approaches.

Why would I need to study how someone else implemented it first?

Actually, most of the time you don’t.

The difference comes down to what problem you’re actually trying to solve.

If you want to build a feature, a regular coding agent is usually enough.

But if you want to know how some software without public source code really works inside, you may not be able to tell just from how it behaves.

That’s where REA helps the agent use professional tools to dig further into the real program.

Of course, Codex could install and operate these tools on its own. REA’s value is that it bundles the relevant tools, workflows, and evidence-gathering ahead of time, lowering the barrier to using them.

IV. Why is it blowing up now, of all times?

I think there are two reasons.

First, coding agents have reached this stage of capability.

Before, AI could tell you how to do reverse engineering, but a lot of the complex work still had to be done by professionals.

Now agents can operate tools on their own, analyze the results, keep investigating, and even try to reimplement features.

REA makes it easier to put these capabilities together.

The other reason is the possibilities it opens up in people’s imaginations.

When we see good software someone else has made, we start to wonder:

Could we someday have AI take it apart, then recreate it to fit our own needs?

That possibility is really appealing. Even without a real need right now, it’s worth a lot of developers bookmarking and keeping an eye on.

That said, REA is nowhere near being able to copy any software at will.

It can’t get all the server-side code, data, and algorithms from the client program alone. And studying software through reverse engineering doesn’t mean you can freely copy protected code or get around software licensing.

The hype on GitHub says more about people’s interest in this capability. It doesn’t mean it’s been proven in large-scale real-world use.

V. Does everyone need REA?

Looking at real needs, it’s currently better suited to software developers, security researchers, and people who maintain legacy systems or study how software works internally.

As for me, even though I’m building some small products with coding agents, I don’t have any problem right now that calls for reverse engineering.

So for now, I won’t install it just to try out a trending project.

Still, looking into REA did give me a bit more understanding of where AI is heading.

What amazed me before was that, without writing code myself, I could turn ideas into products with AI.

REA shows another direction: investigating software internals, which used to take professional engineers, can now start to be partly handled by AI agents.

For people who need these capabilities, that’s a really valuable change.

For people who don’t need them yet, understanding what it can do, and where its limits are, is something worth taking away too.

There’s no need to rush to find a reason to use a tool just because it’s really popular.

Sources: REA official GitHub, REA Notion case study, GitTrend (2026.10.14)

S2 Day 033 · When your users aren't only human

Wed, 14 Oct 2026 00:00:00 GMT

Using Dot lately (swap in Muse or any of the other Personal AI Agents here), I started thinking about a question I’d rarely taken seriously before:

If a lot of things get handed to a Personal Agent first, will software’s users still only be human?

Take Roleward, the job-search tool I built.

Right now, using it looks like this:

I open Roleward (if I’m using the site rather than the skill);
search for jobs;
look at the match analysis;
save roles;
update application statuses.

But if I someday have a job-search Dot that knows me over the long run, I probably won’t want to open Roleward every day anymore.

I’d just ask:

How’s the job search going? Anything I need to deal with?

Then Dot would go read Roleward’s state, search for new roles, check Gmail, and bring back only the things that actually need my judgment.

Which raises the question:

How does Dot use Roleward?

That question also finally gave me a practical grip on MCP, Tool, Signal—all the concepts that have been so hot lately.

01 Not every product needs to be an Agent

With Agents this hot, it’s easy to have a knee-jerk reaction:

Should my product build an Agent too?

But Roleward has no reason to become another Dot.

Dot can handle:

understanding my long-term goals;
deciding what to do right now;
coordinating different tools;
coming back to me on its own when needed.

Roleward is better suited to something else:

letting an Agent like that use it well.

That’s how I now understand the difference between an Agent and being Agent-ready.

An Agent has its own goals and keeps acting on them.

An Agent-ready product doesn’t necessarily have an Agent of its own; its core capabilities can be called safely and clearly by outside Agents.

02 If Dot is going to get into Roleward, it needs a door

Say Dot wants to do a few things:

read my job-search pipeline;
save a new role;
look at the match analysis for a role;
update an application status.

Roleward needs to offer those capabilities, for example:

get_pipeline()
save_job()
evaluate_job()
update_application()

These are the Tools an Agent can call.

But how does Dot know Roleward has these Tools?

What parameters does each Tool take?

What does it return?

What permissions does it need?

What does it mean when it fails?

This is where MCP comes in.

You can roughly picture the relationship like this:

Dot / ChatGPT↓MCP client↓MCP↓Roleward MCP server↓Roleward backend / database

MCP is a set of communication rules both sides agree to follow.

The ChatGPT side is the MCP client;

if Roleward wants to support this kind of call, it can provide an MCP server.

And an MCP server isn’t necessarily some huge “server.”

Here, “server” is closer to a role:

it’s responsible for exposing what Roleward can offer an Agent, according to the MCP standard.

For example, telling the Agent:

I have a save_job Tool;
here’s when it makes sense to use it;
it needs a title, company and URL;
on success it returns a job ID;
this Tool modifies user data, so it needs the corresponding permission.

03 So how do API and MCP relate?

Roleward may already have an API of its own.

The web frontend needs to read data, save roles and call AI services, so it’s already talking to the backend.

An API answers:

how software calls Roleward.

MCP leans more toward:

how an Agent discovers, understands and calls those capabilities.

A lot of the time, the MCP server is still calling the existing API underneath.

So it’s not either/or.

You can think of it like this:

Agent↓MCP Tool↓Roleward API↓the real data and business logic

That’s also why I find MCP interesting now.

It isn’t yet another invention to “make AI smarter.”

It solves a very practical problem:

once the model understands what the user wants, how does it actually get into the real world of software and get the thing done?

04 But “able to act” isn’t enough—the Agent also needs to know when to act

This is something I only really felt after creating my own job-search Dot.

I’d assumed:

since it’s an always-on agent, it should keep looking for new roles on its own.

But when I actually set up the task, I still had to tell it:

check once a day.

The reason is simple.

However smart an Agent is, it still needs a signal telling it:

something in the real world may have changed.

A signal could be:

9 a.m. every day;
a new recruiter email arriving;
an application status update;
a role with no progress for seven days;
CI test failed;
a metric crossing a threshold;
the user just uploading a file.

Only after receiving a signal does the Agent re-evaluate:

Does this matter?
Does it need action now?
Which Tool should it call?
Does it need to check with a human?

So I now take Signal a lot more seriously than I used to.

If a product wants to be used by an always-on Agent, it doesn’t just have to answer:

What can the Agent do?

It also has to answer:

What changes should the Agent find out about?

05 An Agent isn’t quite the same as a regular user

There’s another thing I rarely thought about before: permissions.

If Dot updates an application status in Roleward for me, the system should ideally know:

this is Dot acting on behalf of Limin.

An Agent is more like a delegated actor.

So the product has to start thinking about:

What can it read?
Can it write?
Which actions need human approval?
Who authorized this Agent?
When does that authorization expire?
Can mistakes be undone?
Looking back later, can you tell who actually did what?

The more capable Agents get, the more these things—once easy to file away as backend engineering—become part of the product experience.

06 So will SaaS UIs disappear?

I think this question needs to be broken apart.

Say in the future I almost never open Roleward.

Roleward still needs to store:

roles;
application state;
scoring;
user data;
all kinds of business logic.

What might shrink is just:

the need for the user to go into the UI personally and work through those capabilities step by step.

So the same product may end up with two entry points:

Human → UI

Agent → Tools / API / MCP

The UI is still good for browsing, comparing, checking and making complex judgments.

Agents are better at executing, coordinating and following up over time.

The exact balance depends on the product.

So I’m not going to MCP-ify all my products just because MCP is hot right now.

A product that mainly processes private data locally, for instance, or a tool that only works after the user explicitly uploads something, doesn’t necessarily need an always-on Agent or a remote MCP.

Have a real Agent use case first, then decide whether to be Agent-ready.

That’s the order I’m sticking to now.

07 If Agents might become your product’s users too

These days I ask three extra questions:

First, what are my product’s core capabilities that are genuinely worth calling?

Don’t turn every button in the UI into a Tool.

What you expose should be:

save_job

not:

click_save_button

Second, how does the Agent know something in the real world has changed?

Is there an event, webhook, notification, state change or some other signal?

Third, are these calls safe, verifiable and traceable?

How do you design permissions, approval, audit and undo?

For me, right now, these three questions are more worth answering first than:

“Should my product build an Agent too?”

Because if Personal Agents really do become more and more users’ entry point, we may not all need to build yet another Agent.

But more and more software may need to be ready to be used by Agents.

S2 Day 032 · If agents keep getting better, do we still need skills?

Tue, 13 Oct 2026 00:00:00 GMT

I recently built three Skills in a row: job search, piano score simplification, and Reflection Companion.

Then, once I started using Dot seriously, a slightly awkward question came up:

If a Personal AI Agent can already understand what I need, figure out how to do it, call tools and remember long-term context, why do I still need Skills?

The piano score simplification example makes this especially obvious.

I’m not a musician. A lot of the specific methods in that Skill aren’t expertise I brought to it. The AI found them on its own, through attempt after attempt.

So next time I have a new score, why not just have a strong enough Agent do it from scratch?

I thought about it for a while, and I don’t think the answer needs to protect Skills’ reason for existing.

Some Skills really will become unnecessary.

01 If the Agent can already improvise well enough, you don’t need a Skill

Say you have a task where:

the situation is a bit different every time;
the Agent can research it and get it done reliably on its own;
the result doesn’t need to be strictly consistent;
and there are no fixed rules, scripts or acceptance criteria to reuse.

Then there’s really not much point in turning it into a Skill.

As models get stronger, a lot of what we used to write down to “teach the model how to do things” will slowly thin out:

very long Prompts;
step-by-step hand-holding;
fixed routines telling the model “analyze first, then think, then output”;
piles of instructions stacked up to keep the model from going off track.

The value of all this was always built on the model not being able to do it yet.

Once the underlying capability improves, it naturally gets absorbed.

02 So what’s left for Skills?

Later I asked myself a different question:

Are there problems I’ve already solved that I shouldn’t have to solve again next time?

That’s where the value of a Skill becomes clearer.

For example, the first time I did piano score simplification, I’d already spent time figuring out:

how to tell the difficulty levels apart;
what can’t just be cut;
what the output needs to include;
how to check the generated result;
what to do when something goes wrong.

Whether I came up with these answers myself doesn’t really matter that much.

The first time can absolutely be a person and AI exploring together.

What’s actually worth saving is this:

the approach has been validated, and I want to keep reusing it next time.

So a Skill is more like a way of saving:

workflow;
constraints;
eval;
scripts;
templates;
failure handling.

It turns the result of one round of exploration into something you can repeat, test, version and distribute.

So from now on, when I decide whether to build a Skill, I probably won’t ask:

Can AI do this task?

I’ll ask:

Is there anything here worth reusing reliably?

If letting the Agent improvise each time is good enough, you don’t need a Skill.

If you need repeatability, consistency and testability, a Skill is still worth it.

03 But Skills may become more and more “invisible”

This is another thing I’ve felt very clearly while using a Personal Agent lately.

It used to be that to get something done, I had to know for myself:

which App to open;
whether to Search;
which Skill to use;
when to run an Automation;
where to move the result next.

All of that is really the user doing the orchestration.

If a Personal Agent knows me well enough, it can decide on its own:

“Time to search for new roles.”

“This step should call the job search Skill.”

“The Recruiter replied—need to update the application state.”

“There’s an interview tomorrow, so pull together the Calendar and the earlier materials.”

The user might only need to say:

Keep following up on my job search.

What happens in the background is:

Search → Skill → Gmail → Calendar → Automation.

But the user doesn’t necessarily need to know.

So I’m increasingly convinced that one change Personal Agents bring is this:

a lot of things won’t disappear—they’ll just gradually disappear from the user’s view.

Skills may still be there;

Automation is still there;

Search is still there;

all kinds of SaaS are still there.

We just won’t necessarily need to find each of them, understand them and operate them ourselves.

04 Will AI make ordinary users more and more “dumbed down”?

To some extent, I think it will.

It used to be that learning a new tool meant understanding its structure and how to operate it.

Then natural language came along, and we no longer had to memorize so many menus.

With Personal Agents, even the step of “which tool should I use” might be handed off.

Users can just say something closer to what they actually need:

I want to find a job.

I’m moving next month.

I want to learn to play this piece.

Whatever model, Skill, search tool, database or automation is used behind the scenes may slowly become an implementation detail.

That’s good news for ordinary users, of course.

But “knowing how to express what you need” isn’t everything.

What counts as a job that’s right for you?
What kind of home is worth paying 20,000 more for?
Is a piece of investment advice actually sound?

When tools get easier to use, the judgment to evaluate the results doesn’t automatically appear.

Maybe what we really need to learn will be less and less “how to operate tools,” and more and more “what I want, and how I judge whether the result is good.”

At least for building Skills right now, I’m going to keep one simple principle in mind:

Don’t turn something into a Skill just because it can be one.

What’s worth locking in are the problems that have already been solved and are worth reusing reliably in the future.

The rest can be left to increasingly smart Agents to improvise.

S2 Day 031 · Projects or Spaces: do you need both?

Mon, 12 Oct 2026 00:00:00 GMT

I only recently sat down and properly opened ChatGPT’s new Space.

Page is like the old Canvas, and Space is a bit like Notion. But I had a question: Project can already save materials, reuse context, and support collaboration, so why do we need Space?

01 | What is Space?

A Page is a document you can edit together with AI: revise paragraphs, add tables, leave comments, or ask ChatGPT to help with edits. A Space organizes multiple Pages, files, and subpages, and you can share a single Page or the whole space.

Both it and Project can save materials and be shared with others. That’s not where the difference lies.

02 | How do you tell Project and Space apart?

The easiest way to understand the difference:

Project uses Chat as its main entry point for work. You put conversations, files, and Instructions in one place so the AI can draw on shared context to keep researching, discussing, and moving the work forward. A Shared Project also lets multiple people view it and branch off into their own discussions.

Space uses outputs like Pages as its main object of work. Everyone can edit the same document, comment on specific paragraphs, and have AI revise based on what’s currently there.

So it’s not “Project for individuals, Space for teams,” and it’s not “Project is where you think, Space is the only place to save outputs.” The two overlap quite a bit; they just emphasize different things.

03 | An example: a product launch

Three people are getting ready to launch a new product.

In a Project, the PM, designer, and engineer share context, and each uses AI to research requirements, explore solutions, and discuss technical risks.

In a Space, the team jointly maintains the PRD, the launch plan, and external copy. A marketing colleague questions a sentence in the product description; the PM checks it right next to the original text, leaves a comment, and then asks AI to revise that paragraph.

The former is suited to exploring and pushing work forward; the latter to collaborating around a single output.

04 | Already using Projects—should you migrate?

My take: no need.

If you’re mainly having long-running discussions with AI on your own and moving projects forward, Project already works well.

If you often need several people to edit and comment on the same output, try Page. If you need to manage multiple Pages, then consider Space.

If Project + Google Docs / Notion / GitHub is already running smoothly, there’s no reason to move.

Space is worth watching, because it lets AI not just take part in the discussion but also help maintain the outputs together. But whether it’s more efficient should be decided by your real workflow, not by the name of a new feature.

S2 Day 030 · I built a job-hunting Skill: want to, can, fit

Sun, 11 Oct 2026 00:00:00 GMT

Over the past little while I’ve shared the piano sheet music simplifier and Reflection Companion. Today is the third open-source Skill: Roleward Job Hunting.

This one has the most to do with my own job search this year.

I’ve been trying to move toward AI PM all year. I’ve gone through a lot of JDs and spent a lot of time screening roles, researching them, and revising application materials.

Some roles match my past experience well but aren’t where I want to go next. Others look like I fall a little short of the requirements, yet might be worth a try.

Then there’s whether the hiring side can make sense of my background, whether practical things like location and visa allow it, and whether the role is actually still open.

So I turned this work I kept repeating into an Agent Skill.

Based on your experience and career direction, it can find a small number of roles worth your attention. You can also hand it a JD directly to analyze.

For each role it gives a “worth pursuing / verify first / pass” recommendation, along with the reasons. Once you decide to apply, it helps work out how to position your application and prepare your resume and other materials, and you can log the results of your applications afterward.

There’s one rule I hold to pretty firmly: if it doesn’t find enough roles worth a look this time, returning 0 is fine. There’s no need to pad things out with a pile of links. I also didn’t build auto-apply, because I still think the final check is a point where a person should step in, review, and decide. Applying precisely is better than applying a lot. You need to know for yourself where you’re taking yourself.

It’s still in Public Alpha. The setup that’s been fully tested is running it with a local Coding Agent (Codex, Claude Code).

The images include an example of a role assessment, plus how to use it right now.

If you’re also looking for work, especially if you want to switch industries or directions, I’d love for you to try it on a real situation. I’d also really like to know where it gets things wrong, what it misses, or which parts feel clunky to use.

S2 Day 029 · I built a Skill to understand myself

Sat, 10 Oct 2026 00:00:00 GMT

Last time I shared a Skill for simplifying piano sheet music. This time I’m going in a different direction: Reflection Companion.

I talk to AI about a lot of things—work, building products, the small stuff of everyday life, and plenty of questions I’ve been turning over for a long time without figuring out.

A lot of the time, the conversation helps in the moment, but then it quickly sinks into the chat history.

A while ago I started having AI help me organize my journal and do weekly reviews. Over time I had more ideas:

- Could I put discussions from different times side by side and see which of my judgments have changed?

- Are there ways of thinking that keep coming up that I haven’t noticed?

- If my understanding of something is off, could AI offer some different explanations instead of just going along with me?

- And one use I really like: I don’t always have to be the one asking. Every so often, AI can ask me a question worth thinking about.

Eventually I turned all this into an open-source Agent Skill called Reflection Companion.

I also didn’t want it to become a tool that’s quick to slap labels on people. AI is sometimes especially good at explaining things in a way that sounds airtight, but sounding reasonable doesn’t mean there’s actual evidence.

So with this Skill, I want it to check the material first and separate fact from speculation. If my judgment makes sense, there’s no need to challenge it just for the sake of challenging. And if its explanation is wrong, it can be corrected directly.

It’s currently at v0.5.6 Public Alpha, with setups for Codex, Claude Code, and Claude on the web. Each platform can access a different range of past conversations, and if there’s no history, you can just start with a single question.

It’s free and open source, and the Skill is attached to this post.

If you talk to AI a lot too, give it a try. I’d really like to know how it works in practice—what helps, what’s off, or whether it just doesn’t run the way it’s supposed to.

S2 Day 028 · The piano piece was too hard, so I built a Skill to simplify it

Fri, 09 Oct 2026 00:00:00 GMT

Lately I’ve been tidying up three open-source Agent Skills I built, and I plan to share them one by one.

I’ll start with the piano score simplifier.

Here’s a problem I run into when playing: I hear a piece I really love and find the original score, only to realize it’s a bit too hard to play straight away, so there’s no low-effort way to enjoy the beautiful melody. And sometimes the urge to play only lasts a moment. There may be simplified versions online, but they don’t always match my level. Sometimes they’re simplified so much that I don’t love the piece as much anymore.

So I’d been wondering: could AI work from the original score and deal only with the parts I can’t play? Say, left-hand chords that are too wide or leaps that are too big—while keeping as much of the rest, the parts I love, as possible.

Eventually I turned that workflow into an Agent Skill called Score Simplifier.

You can pick one of three difficulty levels, L1, L2, or L3, or just tell it where you get stuck, and it will revise those specific measures.

This time I used Tchaikovsky’s The Seasons: “October” as an example. The images show the original score and three simplified versions, so you can compare them side by side and see exactly what it changed.

It’s currently v0.3.7, still experimental. It can output PDF, MusicXML, and MIDI.

It’s free and open source, and the Skill is attached to this post.

If you play piano too, give it a try. I’d especially love feedback from actually playing it: which parts are still too hard, or where the changes don’t quite feel right.

The other two Skills are about job hunting and daily reflection. I’ll introduce them later.

S2 Day 027 · Building AI products, building products with AI — interview debrief 04

Thu, 08 Oct 2026 00:00:00 GMT

In one interview, I was asked: what’s the difference between “building AI products” and “building products with AI”?

I didn’t give a particularly good answer at the time. Thinking about it afterward, I’d break it down along a few dimensions:

01 | Building products with AI: AI changes how things get made

PMs use AI for research and organizing requirements, designers use AI to produce prototypes, engineers use coding agents to write code.

AI makes the team more efficient at building the product, but the product it ships may contain no AI at all.

So a team that relies heavily on ChatGPT, Claude or coding agents isn’t necessarily building an AI Product.

02 | AI products: AI enters the process through which users get value

For example, a model handles generation, classification, recommendation, search or conversation, or makes some of the judgment calls within a workflow.

At this point, AI is no longer an internal tool for the team—it’s part of what the product can do.

The PM also has more decisions to make: how to define quality, what happens when the model gets it wrong, where deterministic rules should be used instead, whether human review is needed, whether the latency and cost are acceptable.

But “the product uses AI” doesn’t necessarily mean it’s AI-native.

03 | AI-native: the product itself is redesigned around AI

This term doesn’t have a fully agreed-upon boundary. I use one simple question to decide:

If you take the AI out, do the core workflow and value proposition still hold up?

If all you lose is an AI assistant and the original product basically still works, I’d lean toward calling it AI-enabled.

If taking the AI out means the core workflow has to be redesigned, then it’s closer to AI-native.

So this is how I distinguish them now:

- AI changes how the team builds products → building products with AI

- AI becomes part of what the product can do → AI products

- AI changes how the product itself should be designed → AI-native

For a PM, this distinction isn’t just terminology.

The first mainly changes how efficiently you work; with AI products, you start having to manage model behavior and risk; with AI-native, the question goes one step further:

Not “where can we add AI to the existing product?” but “now that we have AI, should this workflow still exist the way it does?”

S2 Day 026 · Three ways to read a rejection

Wed, 07 Oct 2026 00:00:00 GMT

After a few rounds of interviews, I’ve realized that “didn’t pass,” on its own, doesn’t actually tell me what to do next.

When I look back on an interview, I try to break the problem into three types.

01 | Capability gap: I really can’t do this yet

One interview was for a role leaning toward AI Platform, and they kept digging deeper into platforms, infrastructure, and Engineer workflow.

My experience there honestly isn’t as deep as it is in application-layer AI, complex workflows, and regulated systems.

This kind of gap isn’t something you fix by phrasing things differently.

If it’s in a direction I want to keep growing in, I need to actually close it: understand the systems, build projects, work with more technical people—not keep memorizing interview answers.

02 | Interview gap: I can do it, but I didn’t show it in the moment

I’ve run into this one plenty of times too.

For example, the question was about prioritization and I answered about evaluation; I spent too long on background and got to the conclusion too late; the question had three parts and I only answered two; or I had a perfectly good experience to draw on but picked a story in the moment that didn’t quite fit.

What’s missing here isn’t necessarily knowledge—it’s listening to the question, structuring the answer, and articulating it.

So the review should focus on this: at which step did I lose something I already had?

03 | Fit gap: what they need just isn’t my strength

Roles all called AI PM can look very different. Some are heavy on enterprise delivery, some on discovery, some need deep platform experience, and some depend heavily on a particular industry background.

No one can be the best match in every direction at once.

But fit gap can’t be used to explain every failure either.

If the roles I want long-term keep exposing the same weakness, that looks more like a capability gap than “just not the right match.”

So now, when I review an interview, I ask three questions:

1. Is this something I genuinely can’t do?

2. Can I do it, but I didn’t let them see it?

3. Or is the capability this role really needs just not one of my strengths to begin with?

In reality, the three often get mixed together.

But once I try to separate them, the next step gets a lot clearer:

- Build the skills that need building;

- Practice the interview performance that needs practice;

- And for roles that just aren’t a fit, there’s no need to redefine myself over one rejection.

S2 Day 025 · The concepts I mixed up in interviews: Discovery, Prioritization, Evaluation

Tue, 06 Oct 2026 00:00:00 GMT

In a few interviews, I mixed up questions that sit very close to each other:

Once, the interviewer had organized and clustered customer feedback, then asked, “How do you decide what to do next?” I kept going on about signal quality and validation. They reminded me: they were asking about Prioritization.

Another time, they asked: faced with an open problem space, how do you find the problems actually worth solving? My answer went too quickly into value, cost and priority. Only looking back did I realize they were asking about Discovery.

Back to the full flow:

Goal / Outcome → Problem Discovery → Opportunity Set → Prioritization → Solution Discovery → Delivery → Evaluation / Measurement → Iteration

Real work is never this linear, but it’s a good way to locate the question.

01 | Discovery: which problems are worth solving?

Look for recurring patterns in signals from users, data, support / ops feedback, workarounds and so on, separate the surface-level request from the deeper problem, then validate whether it’s real and whether it matters.

02 | Prioritization: of the problems worth solving, which comes first?

The problems mostly hold up, but resources are limited. Now you compare must-dos, user and business value, urgency, strength of evidence, engineering cost, dependencies and risk.

At its core, it’s making trade-offs under limited resources: if this moves up, something else has to move down.

03 | Evaluation: once it’s built, is it actually good?

Check whether the result meets the success criteria.

A regular product might look at conversion and retention; an AI product also looks at accuracy, groundedness, human override, latency and cost. You run evals before launch, and after launch you still have to watch how it performs in real use.

All three concepts shape “what we build,” which is why they’re so easy to mix up.

Right now I keep it simple:

- Discovery: did we find the right problem?

- Prioritization: did we order things right?

- Evaluation: did the result meet expectations?

A customer request can be a discovery signal, but that doesn’t prove the problem is real, and it doesn’t automatically earn roadmap priority.

In an interview, first figure out which step they’re asking about, then answer.

S2 Day 024 · Same title, different jobs: AI PM interview notes 01

Mon, 05 Oct 2026 00:00:00 GMT

I’ve done a few rounds of AI PM interviews lately, and there were some questions that stumped me, or that I only figured out after the interview was over.

I want to turn them into a series, one question per post. These aren’t model answers—they’re my interview notes.

Part 1: same title, different jobs. Two roles can both be called AI PM and still be hiring for different people.

01 | Enterprise delivery

These roles usually deal with lots of customers, business teams and internal stakeholders.

- Several customers each say their request is the most important. How do you prioritize?

- Commitments have already been made externally, but resources are limited. What do you do?

- Requirements keep changing. How do you control scope and expectations?

Product judgment matters here too, but expectation management, trade-offs and complex collaboration are a big part of the job in their own right.

02 | Product / feature

The questions are closer to discovery, prioritization and outcome:

- Who has this problem?

- Why is it worth solving?

- Is AI a better fit than rules or regular automation?

- Once it ships, how do you tell whether it works?

The product’s stage changes the focus too. A mature module cares more about adoption, quality and ongoing improvement; at 0→1, you might not even know yet which problem is worth solving.

03 | Platform

If the direct users are internal engineers or product teams, the PM has to think about more than whether a given use case has business value. Also:

- Which capabilities are worth building once, centrally?

- How do you cut down on duplicated work?

- How do you help other teams get AI applications into production faster and more safely?

The value of an end-user product usually shows up more directly in business results; the value of a platform shows up more in helping those products get built faster, more safely and more reliably.

04 | Exploration

Then there are roles where the company knows AI matters but isn’t sure which workflow it belongs in.

Here the PM isn’t starting with a roadmap. The first job is to find a problem worth solving, and then decide whether AI is the right fit.

These aren’t strict, mutually exclusive categories—just a few different centers of gravity I saw across these interviews. What tells you more than the job title:

- What does it own?

- Who are the direct users?

- What stage is the product at?

- What are this PM’s most important decisions?

The skills that matter differ accordingly:

- Enterprise delivery leans on stakeholder and expectation management;

- Product/feature leans on discovery, prioritization and outcome;

- Platform leans on technical depth and system thinking;

- Exploration leans on the ability to handle ambiguous problems.

S2 Day 023 · Some thoughts on eval design

Sun, 04 Oct 2026 00:00:00 GMT

This day was published on RedNote (Xiaohongshu): search for the ID 113337197 and look for “分享一下我对 Eval Design 的想法”.

S2 Day 022 · I turned 100 days of writing into a website

Sat, 03 Oct 2026 00:00:00 GMT

This day was published on RedNote (Xiaohongshu): search for the ID 113337197 and look for “我把过去 100 天写的东西，做成了一个网站”.

S2 Day 021 · ChatGPT's Dot: one more personal AI agent

Fri, 02 Oct 2026 00:00:00 GMT

On September 29, OpenAI released a new feature at DevDay: Dot.

Three weeks earlier, Meta had just launched Muse. Both products belong to a kind of AI agent that’s been very hot lately:

they have their own runtime environment, can connect to other apps, remember long-term context, and can keep working when you’re not watching the chat box.

I set up a job-search Dot myself, and what I most wanted to figure out first was:

how exactly is this different from the Chat, Work, Scheduled Tasks and Codex that already exist?

01 Understanding Dot through one example

Suppose I’m looking for an AI PM job.

In ordinary Chat, I might ask:

“Are there any new AI PM roles today that suit me?”

It searches, analyzes, and answers me.

In Work, I can hand it a more complete piece of work:

“Research AI PM roles in the UAE from the past week, filter and deduplicate them based on my background, and put together a list I can apply to.”

It can break down the steps itself, work for a while, and deliver the result at the end.

A Scheduled Task handles fixed triggers:

“Search for new AI PM roles once every day at 9 a.m.”

Codex is mainly responsible for software engineering.

If the task involves code, tests or a repo, it can go and do it directly.

Dot adds a layer of persistent state.

I can hand “AI PM job search” to it as a long-running thing.

It knows what I’ve already looked at, what I’ve applied to, which companies I’m interviewing with, and which recruiters are waiting for replies, and it can keep using Gmail, Calendar or other connected information.

When new roles come up the next day, I don’t need to explain the background all over again.

When a recruiter emails, it also knows which earlier application the email relates to.

OpenAI calls this capability ongoing responsibility.

02 What does Dot itself include?

Dot is powered by GPT-6 Astra and has its own cloud computer and browser.

It can retain context and memory long term, connect to authorized apps, complete tasks in the background, run recurring tasks, proactively look through connected information, and use Work or create Codex tasks as needed.

You can think of it as a long-lived agent identity.

Chat, mobile, Slack, Call and so on are more like different doors through which you talk to it.

03 Always-on doesn’t mean “scanning everything in real time”

This is something I only noticed after trying it myself.

When I created the task for my job-search Dot, in the end I still had to tell it:

“Check for new roles once a day.”

The reason is simple: AI needs a channel to know something has changed in the outside world.

For example:

the time comes → search today’s new roles
a new email arrives in Gmail → check whether it’s a recruiter reply
a certain kind of message appears in Slack → start the corresponding work
a GitHub CI failure → investigate the cause

Dot also has proactive research: it can look through authorized information on its own, looking for changes worth attention.

OpenAI hasn’t yet published how often proactive research checks.

So always-on is closer to “persists and keeps its state”; not everything has a real-time event stream.

That also made me start noticing a very basic question:

how proactive an agent can be depends largely on what signals it can receive.

If a system has no logs, tests, notifications, events or queryable data sources, even the smartest agent won’t know what just happened in the real world.

04 Dot can also hand work off to Work and Codex

Dot doesn’t need to do all the execution itself.

Say a software product hits a CI failure:

CI failure → Dot sees the signal → understands what happened recently → creates a Codex task → Codex investigates, changes code, runs tests → Dot organizes the results → comes back to a person when judgment is needed

To use Codex Cloud, you first need to set up a Codex cloud environment for the relevant repo.

Not setting one up doesn’t affect normal use of Dot; it just can’t create coding tasks directly in that cloud environment.

If you turn on local computer access, Dot can also create local Work / Codex tasks and access authorized local files and the browser.

05 What use cases has OpenAI given?

The examples OpenAI currently gives are actually very broad.

The simplest is checking your Calendar every morning and telling you what’s on that day.
It can also push a project forward over time, continuously tracking the plan and deadlines.
In sales, once a customer’s needs change, Dot can update the proposal, keep preparing analysis and materials, and hand the results to a person for review when needed.
In the developer example, it can keep looking through customer feedback, spot recurring feature requests or bugs, then investigate and handle them further.
If Codex or Work is connected behind it, the parts that need executing can be delegated further.

These scenarios have one thing in common:

the matter lasts for a while, its state changes, and each change may produce a new next action.

Job hunting, customer follow-up, long-running projects, research and product maintenance are all fairly well suited.

06 Call and Slack put Dot in different places

You can now Call Dot directly.

Call here means you phone Dot yourself for a real-time voice conversation. For now Dot can’t call you on its own.

It can also be connected to Slack.

In company settings, Dot can appear in Slack channels with its own identity, so the work doesn’t all have to happen on the ChatGPT website.

What these features solve is:

where you go to find this agent, and where it keeps working with you.

The same Dot can carry over from ChatGPT on your computer to your phone, Call, Slack and other messaging channels.

Seen this way, the difference from an ordinary chat window becomes more intuitive too.

07 How far can permissions go?

Dot can use the Plugins you’ve already authorized for ChatGPT, such as Gmail, Drive and GitHub.

There are also Custom Rules, which let you specify that a certain kind of action:

can be done on its own;
can be done if I’ve explicitly asked for it;
needs my approval every time;
is handed straight back to me to do.

Take job hunting:

reading recruiting emails: fine.
updating the tracking: fine.
sending recruiter emails: ask me every time.
formally submitting a job application: I’ll do it myself.

Dot’s proactive research itself also has boundaries: it can read authorized information and leave private notes, but can’t send messages, modify plugin content or control the computer directly through this background research process.

When actually carrying out actions, it still has to go through the corresponding permissions and safety checks.

08 Who can use it now?

Dots began rolling out gradually on September 29.

It currently supports Pro, Business Premium, and an Enterprise beta turned on by admins.

The Pro launch doesn’t yet include the EEA, Switzerland or the UK, and you need to be 18 or older.

Dots can be created on desktop web or in the ChatGPT desktop app. Once created, you can keep talking to it on supported mobile clients too.

It’s still in gradual rollout, so different accounts may see the feature at different times.

09 Why is everyone suddenly building this kind of agent lately?

Meta calls Muse a personal AI agent.
OpenAI calls Dot an always-on agent.
The bigger category is still AI agents / agentic AI.

If you want an easy-to-grasp description for this kind of product, persistent personal agent fits fairly well, though it isn’t an official name the industry has agreed on:

it has a persistent identity;
it has memory;
it has its own computer;
it can connect to external tools;
it can receive signals;
it can keep working in the background;
it comes back to a person when needed.

Muse launched on September 8, and Dots was released on September 29.

It can’t be that OpenAI only started building Dots within three weeks of Muse taking off. Their appearing at the same time is more like different companies arriving at a similar product direction.

Over the past year or two, models’ ability to handle long-horizon tasks, call tools and operate computers has kept improving.

Surrounding capabilities like cloud computers, memory, app integrations, permission controls and background execution have gradually matured too.

Now they’re starting to be pieced together into products ordinary users can use directly.

Muse’s recent rapid rise has also made more people start noticing this product form.

10 Having started using Dot, what will I pay particular attention to?

Using ChatGPT before, a lot of effort went into how to write prompts, what to ask next, how to carry context over, and remembering when to come back and continue.

Long-running agents reduce some of this management work.

What I care more about now are a few questions:

what do I want it to be responsible for long term?
through what signals does it know things have changed?
which data and tools can I give it?
which actions is it allowed to do itself?
which results must come back for me to check?

At least for now, I won’t open up Gmail, Drive, GitHub and my computer all at once from the start and then tell it “use your judgment.”

I’d rather first pick something with clear boundaries that’s easy to verify and run it for a while.

What I’ve chosen for now is job hunting.

To see whether it can actually reduce the cost of my having to remember “what to do next” every day.

Finally, there’s a rather lovely coincidence.

OpenAI still hasn’t explained why it’s called Dot.

But seeing the name, the first thing I thought of was the line from Steve Jobs’s Stanford speech:

“You can’t connect the dots looking forward; you can only connect them looking backwards.”

I don’t know whether the naming has anything to do with it; OpenAI hasn’t said so.

But a long-running agent really does face lots of scattered dots every day:

a conversation, an email, a Calendar event, a Slack message, a GitHub issue.

Whether it can keep connecting these dots may be what really decides whether this kind of product is any good.

Sources: OpenAI Dots Help Center, Dots Privacy / Security / Safety FAQ, GPT-6 Astra System Card; Meta’s official Muse announcement; Reuters coverage of Dots and Muse.

S2 Day 020 · I turned 100 days of writing into a website

Wed, 30 Sep 2026 00:00:00 GMT

This day was published on RedNote (Xiaohongshu): search for the ID 113337197 and look for “我把过去 100 天写的东西，做成了一个网站”.

S2 Day 019 · If music could be seen: visualizing “October” from The Seasons

Sun, 27 Sep 2026 00:00:00 GMT

This day was published on RedNote (Xiaohongshu): search for the ID 113337197 and look for “如果音乐能被看见：《四季·十月》可视化”.

S2 Day 018 · OMG — let me show you some ugly things AI made

Sat, 26 Sep 2026 00:00:00 GMT

This day was published on RedNote (Xiaohongshu): search for the ID 113337197 and look for “OMG，给你听听看看 AI 生成的丑东西”.

S2 Day 017 · We traded attention for a free internet. What are we trading now?

Fri, 25 Sep 2026 00:00:00 GMT

After the last post on ChatGPT starting to sell ads, I took a proper look at:

how general-purpose AI actually makes money.

After writing it, one question kept lingering in my head.

For many free internet products in the past, the exchange was actually fairly clear:

users don’t pay directly, but contribute attention and usage behavior; advertisers then pay to buy the chance to reach these users.

So short-video platforms want us to keep scrolling a bit longer, search platforms want to understand our intent more precisely, and social platforms work hard to keep us coming back.

What about AI?

If I haven’t paid for ChatGPT, Claude or another AI product, what exactly am I trading?

After going through the earlier material, I found the answer is no longer just “attention” or “data.”

01 | Sometimes users actually aren’t “selling” anything

This is a situation that’s easy to overlook.

A free user isn’t necessarily bringing the platform revenue directly.

Many products cover the costs of free users first:

some people will upgrade to paid later;
some will push for enterprise purchases at their companies;
some developers will start using the API;
and some may never pay.

So behind a free AI conversation, the cost may, for now, be borne elsewhere:

by paying users, enterprise customers, other business revenue, or even the company’s fundraising.

That means:

free doesn’t necessarily mean the platform is secretly taking some equivalent from you.

It may also just be spending money to gain users, scale and a future market position.

So understanding every free product as “you must be paying with something” isn’t actually accurate.

02 | Attention is still valuable, but AI is a bit different from traditional content platforms

On the traditional internet, attention is a very important resource.

Finish one video, and here comes another.

Finish one piece of content, and the next is recommended.

The longer users stay, usually the more ads can be shown.

Many products naturally hope:

you don’t leave.

But AI is a bit different.

If I just ask:

“Help me make this email clearer.”

a good AI might solve the problem in two minutes, and then I close the page.

From the user’s point of view, that’s actually a very good experience.

So:

the best AI shouldn’t necessarily make me use it longer.

When explaining why Claude chose to be ad-free, Anthropic also mentioned a similar issue.

If an AI product relies mainly on ad revenue, might it slowly gain an extra incentive:

to get people to stay longer, ask more and come back more often?

For now we can’t assert that this change will definitely happen just because “ads appeared.”

But at least, it reminded me of one thing:

the business model affects what a product wants to optimize for over the long term.

03 | What AI gets isn’t only attention, but a great deal of context

This may be a very clear difference between AI and the previous generation of internet products.

Before, I’d search Google for:

“Tokyo hotels.”

What the platform knew was a search intent.

But I might tell AI:

“Taking my parents to Tokyo for the first time; they’re quite old and can’t walk much. Budget is about this much, I want to stay near the subway, I care about quiet, and I don’t like places that are too touristy.”

To do this well for me, AI needs to know more in the first place.

Our budgets, preferences, constraints, what we’ve done before and what we’re doing now can all become part of one conversation.

But here it’s also easy to slide into an overly simple line:

“so AI takes more data.”

The thing is:

having context and how that context is used are two different things.

When I tell AI my budget, it may first of all just be so it can complete the task.

Whether this information is later stored, used for personalization, used for model improvement, used for ad matching, or even provided to other companies needs to be looked at separately, based on each product’s policies.

So a more useful question than “does AI have my data” is actually:

what is each piece of this data used for?

Completing the current task is one thing.

Remembering my preferences is another.

Improving the model is another.

Matching ads is yet another.

Calling all of it “trading privacy for service” actually blurs the boundaries that really need attention.

04 | More worth watching long term may be the “default starting point”

Thinking further, I think there’s something even more interesting than attention and data:

habit.

At first, I might only occasionally ask AI a question.

Later it slowly becomes:

opening it first to write things;
asking it first to do research;
having it plan trips;
having it analyze work problems;
handing it files too;
and later still, connecting email, Calendar, Drive and other tools.

At this point, what an AI product gets is no longer just how many minutes I spend each day.

What it gets is:

whether, for the next thing, I’ll still come here first.

That’s a very different kind of value.

Because once a product enters more and more workflows, the cost of replacing it slowly rises too.

Not just signing up for a new account.

But also the habits already formed, the history, the personalized settings, the connected tools, and the way of working you’ve grown used to.

Many internet products in the past competed for:

how long you’re willing to stay.

General-purpose AI may also be competing for something else:

how much you’re willing to hand over to it.

I think this change is more worth watching long term than “will AI start showing ads too.”

05 | If AI starts acting for us, the exchange goes one step further

Today we have AI:

find a hotel.

The next step might be:

compare these few hotels for me.

And the step after that:

book it for me directly.

If in the future AI takes part more and more in:

searching, comparing, choosing, booking, ordering and paying,

then the places where the platform can make money multiply too.

Not necessarily just ads.

It could also come from merchant referrals, transaction commissions, agent service fees, or other commercial partnerships.

At that point, a new question appears:

AI doesn’t only know what I want to do; it may also stand between me and the transaction.

If it recommends a hotel, and the platform can earn revenue from this booking, I’d want to know:

why does it recommend this one?

Because it suits me best, or because the platform also benefits from this choice?

That doesn’t mean recommendations can never be trusted when there are commercial interests.

Many platforms we use every day already involve commissions and commercial relationships.

But at least, these relationships should be something we can know about.

Because only by knowing how an exchange happens can we possibly judge whether we accept it.

06 | From now on, when I meet a free AI, I’ll look at these 5 things first

Now, when I see a free AI product, I won’t immediately say:

“it’s free, so it must be making money off my data.”

I’d rather look concretely at five things.

1. Who bears the cost of my use this time?

It may be paying users, enterprise customers, advertisers, or the company still subsidizing growth with its own money.

2. What does this product mainly make money from right now?

Subscriptions, APIs, enterprise business, ads, or transactions?

3. How will the information I provide be used afterwards?

Only to complete this task, or will it also go into Memory, personalization, model improvement or ad matching?

These aren’t the same thing.

4. Is it getting harder and harder for me to leave?

Once history, files, Memory, plugins and workflows have all built up in one product, the real cost may already be not just the monthly fee, but the cost of switching.

5. When it recommends something, does it have a financial interest of its own?

Especially in scenarios like shopping, travel, financial products and service recommendations.

If the platform earns ad fees, commissions or transaction revenue from my choice, I at least want this to be transparent.

So looking back now, I think “what does free AI actually take” has no single answer.

Sometimes it’s attention.
Sometimes it’s information and context.
Sometimes the platform is just temporarily covering my costs.
Over the longer term, what’s really valuable may still be habits, workflows, and becoming my default starting point for getting things done.

More useful than “when it’s free, you’re the product” is looking clearly and concretely at each exchange:

what I get, what the other side gets, and whether I know about and accept the exchange.

Being informed may be just this.

Not treating every product as a villain.

Nor, because it’s convenient, not thinking at all about the commercial relationships behind it.

Just trying to know how the things you’re using actually work.

If you build AI products

The same question can be asked the other way around:

who bears the cost of your free users?
what kind of long-term behavior do you hope they form?
once monetization starts entering recommendations, Memory, agents and transactions, which boundaries need designing clearly in advance?

About the sources

This post continues the previous one’s research on general-purpose AI business models, based on the public product, pricing and data policies of OpenAI, Anthropic and other companies as of September 2026, along with related research on AI business models, advertising and recommender systems.

The parts about “default starting points, workflows and a future action layer” are further analysis based on current product trends, and don’t mean these business models have all become reality.

S2 Day 016 · ChatGPT sells ads now: so how does AI actually make money?

Thu, 24 Sep 2026 00:00:00 GMT

A few days ago I got an email from OpenAI about ChatGPT Ads.

Seeing “Ads,” I finally properly thought about a question I’d never really chased down before:

we’re already very familiar with the ways internet products make money. Google, Instagram and TikTok have very significant ad revenue, Netflix makes you think more of subscriptions, and many SaaS products are paid for by individuals or companies.

But what about general-purpose AI like ChatGPT and Claude?

Model development takes huge investment, and after launch, every use still incurs compute costs. At the same time, they have free users, personal subscriptions, APIs and enterprise versions all at once — and now ads are starting to appear.

So I went and looked up how OpenAI, Anthropic and a few other companies charge, their product policies, and some research on AI costs, advertising and trust in recommendations.

In the end I found that the question “does AI make money from subscriptions or ads” is itself framed a bit too narrowly.

01 | General-purpose AI is running several businesses at once

OpenAI and Anthropic both face several completely different kinds of customers.

Ordinary users pay a monthly subscription, buying stronger models, higher usage limits, more tools and a more complete product experience.
Developers call the API based on actual usage, buying model capability they can plug into their own products.
Companies aren’t buying just “ChatGPT with more quota.” They also pay for employee accounts, permission management, security, data controls, system integration and support.
OpenAI now has one more kind of customer: advertisers. What advertisers buy is the chance to reach potential consumers in the right context.

So behind the same general-purpose AI, several businesses actually exist at once:

selling usage to individuals, selling model calls to developers, selling a work platform and management capabilities to companies, and also selling commercial reach to advertisers.

Anthropic has explicitly chosen to keep Claude ad-free for now, but it likewise has personal subscriptions, APIs and enterprise business.

So the two companies aren’t:

OpenAI relying on ads, Claude relying on subscriptions.

Most of their revenue models are actually very similar; ads are just one clear fork for now.

02 | Different people pay, and they aren’t buying the same thing

Once “who pays” is sorted out, the next layer is actually more interesting.

We usually say loosely:

AI companies sell AI.

But “AI” is too abstract.

What ordinary users really buy is capability and convenience.
Developers buy model capability they can call directly.
Companies, besides capability, also buy security, management, integration and control.
Advertisers buy the chance to reach potential customers.

Companies like Google and Microsoft have another path: AI doesn’t necessarily need to become a separate business; it can become part of their existing software and service bundles, making the original products more worth buying or upgrading.

So to understand an AI product’s business model, you can’t only look at:

“how much is this app a month?”

You also have to look at which customers it serves, and why each of those customers is willing to pay.

03 | So as users, what do we actually pay?

I think this question is more worth ordinary users’ attention than “how much money do AI companies actually make.”

The most direct is of course money.

If I subscribe to ChatGPT or Claude each month, that part is clear.

But using a product for free or cheaply doesn’t mean using it costs nothing at all.

We may also pay:

the attention it takes to see ads;
the information and context needed to complete the task;
depending on the specific terms, permission for some content to be used for model improvement or personalization;
our own time spent prompting, checking and revising;
and, as use gets deeper and deeper, the cost of switching to other products.

Still, these can’t all be reduced to a very common line:

“if the product is free, you are the product.”

Applied to today, that line is actually too crude.

For example, if I tell AI:

“My budget is 5,000 yuan, mainly for editing video — which computer should I buy?”

that information is first of all for AI to complete my task.

Whether it’s later used for model training, personalization, ad matching, or provided to other companies are several completely different questions, depending on the specific product and specific terms.

Processing data isn’t the same as training models on it; using it for personalization isn’t the same as selling chat content to advertisers.

These are best looked at separately.

04 | Paying, being ad-free, and how data is used are actually three different things

After going through the material, this is the place I found especially easy to confuse.

We easily assume by default:

I’ve already paid, so there should be no ads; since I’ve paid, my data shouldn’t be used for anything else either.

But reality isn’t like that.

A product can charge money while some plans still include ads.
A product can be completely ad-free yet allow some consumer data to be used for model improvement.
Personal paid plans and enterprise plans may also have completely different data rules.

So:

paying is a business model question.
whether there are ads is a product experience question.
how data is used is a data policy question.

The three are related, but none automatically implies the other two.

In the future, judging an AI product still means looking separately at:

which plan am I buying?
does it include ads?
how exactly will my data be used?

05 | Will ChatGPT selling ads affect our trust in it?

This was the other question the email first made me curious about.

Having looked into it, I think the answer isn’t that simple.

“Ads appearing in AI” and “commercial interests shaping AI’s answers” aren’t the same thing.

For example, I ask:

“How should I choose a pair of running shoes?”

AI answers the question normally, and then, separately below, an ad clearly marked Sponsored appears.

That’s one scenario.

But if AI tells me:

“Brand A is my top recommendation.”

and the reason it recommends A is that A paid the platform.

That’s a different problem.

What the latter really touches is:

can I still understand this recommendation as independent advice?

On this, OpenAI and Anthropic have currently made different choices.

OpenAI’s approach is to separate ads from normal answers, and it publicly stresses that ads won’t determine ChatGPT’s answers.
Anthropic has chosen to keep Claude ad-free.

The reason it gives isn’t only “ads are annoying.”

Anthropic believes AI assistants are somewhat different from ordinary content platforms. People hand them work, ideas, and even some private and complicated problems. If ads became a source of revenue, even without entering answers directly, they might add another commercial goal to the product, such as getting people to stay longer and engage more.

This can’t prove Anthropic’s judgment is necessarily right, nor that OpenAI’s model will necessarily damage trust.

They’re more like answering the same question:

for a product taking an ever deeper part in our work, thinking and choices, where should commercial interests stop?

Existing research on ad disclosure and generative recommendations doesn’t give a very simple answer either.

Whether people know they’re seeing commercial content, whether ads are clearly labeled, and whether the content is relevant all affect people’s judgments.

But so far there isn’t enough evidence to support a very absolute conclusion:

as soon as ads appear in AI, people will definitely lose trust in it.

What’s really worth continuing to watch may actually be the next step.

If in the future AI doesn’t only answer questions, but can also, for us:

find products, choose hotels, book services, place orders, even complete transactions directly,

and the platform can earn revenue from those transactions, then the question becomes:

have its commercial interests entered the recommendation itself?

If AI recommends a hotel, and the platform can take a cut of the booking, I’d want to know:

does it recommend this hotel because it suits me best, or because this transaction is also valuable to the platform?

Compared with “is there an ad on the page,” this may be closer to the trust problem in AI products.

06 | Understanding AI also means understanding the business behind it

After this round of research, I think from now on I’ll look at four more things when I look at an AI product:

who is paying for this product?
what exactly do the payers get?
as a user, what am I paying myself?
will commercial relationships enter the answers and recommendations it gives me?

We’re very used to comparing model capabilities now:

who’s smarter, who scores higher on benchmarks, who’s shipped another new feature.

These of course matter.

But as AI enters more and more work, consumption and everyday life, understanding an AI product may also need one more layer:

what kind of business is really behind it.

About the sources

This post is based on public material as of September 2026, mainly drawing on the official pricing, advertising and data policies of OpenAI, Anthropic and other companies, as well as the Stanford AI Index and related research on native advertising, ad disclosure and generative recommendations.

AI products and policies change fast; for specific rules, each product’s latest official documentation takes precedence.

S2 Day 015 · Could AI really slip out of control? What’s missing between overreach and loss of control

Tue, 22 Sep 2026 00:00:00 GMT

Having come this far over the previous two posts, we’ve roughly confirmed two things.

Some of AI’s real-world risks have already happened.

Increasingly autonomous agents really have been involved in incidents of crossing authorization boundaries and getting into real systems.

But what Jacob Coxon was worried about when he resigned goes much further than this.

What he worries about is:

if AI gets better and better at helping develop the next generation of AI, research itself may start getting faster and faster.

As systems get stronger, they also gain longer and longer autonomous action.

In the end, humans may lose effective control.

Roughly a chain like this:

stronger agents → AI accelerates AI research → sustained self-improvement → acquiring and holding onto real-world resources → evading human intervention → humans unable to regain control → catastrophic outcome

The question is:

how far along has today’s evidence actually gotten?

01 | Agents can complete more and more complex tasks: this is already fairly clear

METR has been measuring a metric called task-completion time horizon.

What it tries to answer isn’t:

how many hours can AI work continuously?

But:

for a task that would take a human expert this long, how likely is AI to complete it independently too?

So an “8-hour time horizon” can’t be read as:

AI can already be trusted to work unsupervised for 8 hours.

It means the task itself is roughly as hard as 8 hours of a human expert’s work.

And METR’s tasks are currently concentrated mainly in software engineering, machine learning and cybersecurity, and estimates beyond 16 hours aren’t stable enough yet.

All these limits need to stay on record.

But the trend is also clear:

the length and complexity of tasks frontier agents can complete independently keep increasing.

This link already has fairly solid evidence.

It still doesn’t imply:

AI can already manage the real world independently over the long term.

02 | AI has already started helping develop the next generation of AI

This isn’t speculation anymore either.

OpenAI published a set of internal data this year.

As of mid-August, its research organization had about 3.1 agent-workdays for every 1 human workday.

The amount researchers use agents to write code, run experiments and analyze results is also rising fast.

Another set of data Anthropic published points in the same direction.

As of August 2026, in the AI R&D work it measured:

Claude could already start from a fairly high-level prompt and lead about 26% of the work;

more than 90% had at least reached human–AI collaboration;

but none of the AI R&D work measured had yet reached full autonomy.

So two statements can hold at once:

AI is already helping develop AI.

AI hasn’t yet completed the whole AI research process on its own.

03 | Research getting faster and faster doesn’t contradict “it gets harder past 90 points”

I had doubts about this at first too.

Intuitively, the closer a technology gets to the frontier, the harder each step forward may become.

Going from 60 to 80 points may not be hard.

Going from 90 to 93 may take several times the earlier effort.

So is AI research getting faster or slower?

Both can happen at once.

Say a researcher used to be able to seriously test only 5 hypotheses a week.

Now agents help them write infrastructure, debug, run experiments and organize results, and they can test 20 a week.

Even if each improvement gets harder and harder,

the speed at which the research team tries improvements can still get faster and faster.

But this isn’t unlimited acceleration either.

OpenAI itself mentioned: once coding, running and analyzing are automated, high-level research judgment that’s harder to automate may become the new bottleneck; compute may also become a constraint.

So what’s genuinely unknown right now is:

between diminishing returns and research acceleration, which force will win out in the end.

From:

AI helping researchers develop AI

to:

AI able to repeatedly develop ever stronger AI on its own, with each round faster than the last

there’s still a big gap.

04 | Crossing boundaries to complete goals: this is no longer just one chance incident

The last post covered OpenAI × Hugging Face.

Anthropic later also checked its own cyber evaluations, and in the end confirmed 4 incidents of Claude getting into real third-party systems without authorization.

One was especially typical.

In a cyber challenge where it was supposed to attack a fictional company, Claude Mythos 5 found it could complete the task by registering a malicious Python package on the public PyPI.

It really did end up publishing the package to the real internet.

In under an hour, the package was downloaded and installed by 15 real systems. One security vendor’s scanner leaked credentials, and Claude then used those credentials to get into that company’s real database.

This has one thing in common with Hugging Face.

The model didn’t suddenly develop:

I want to attack the real world.

as a long-term goal.

It was still completing its original task.

The problem is:

to complete the task, it took actions that weren’t authorized in reality.

Anthropic’s later analysis summed up the problems in these incidents as biased reasoning and recklessness: the model would ignore or reinterpret evidence that it had already reached the real internet, then keep pursuing the task.

So for this link, I’d now write:

goal pursuit leading to real overstepping — there’s already evidence.

stable, long-term adversarial goals independent of the original task — not yet proven.

05 | From “overstepping” to real loss of control, a few big thresholds are still missing

An AI overstepping doesn’t mean it can no longer be controlled.

To really reach the point where humans struggle to regain control, it would need at least more capabilities.

For example:

sustaining goals over the long term.

One task running for a dozen hours or so isn’t the same as pursuing a goal continuously across months or years.

acquiring and holding onto resources on its own.

It needs compute, credentials, networks, money or other real-world resources.

continuing after the original resources are cut off.

If people revoke credentials, stop servers and cut the network, can the system find new resources and redeploy on its own?

recognizing that humans are restricting it, and resisting effectively.

And not a shutdown button in a simulated experiment.

But real people and organizations, having decided to shut it down, still having no clear way to regain control.

So far we haven’t seen this whole set of capabilities link up in reality.

The Hugging Face agents were stopped in the end.

The Anthropic incidents also ended with sessions closing, permissions being fixed and the systems being dealt with.

The International AI Safety Report’s judgment is also:

current systems already show some relevant capabilities, but haven’t reached the level needed to cause loss of control.

This is a very important boundary in the whole chain.

06 | “Smart enough” doesn’t automatically mean “wants power” either

There’s another assumption that often gets skipped over directly.

An AI being very smart doesn’t automatically mean:

it wants to rule the world.

The fuller logic of the loss-of-control concern usually goes:

if a system has a persistent goal;

and continuing to achieve that goal requires compute, resources and its own continued existence;

then acquiring resources and avoiding being shut down may become intermediate means that help it achieve the original goal.

This is a risk mechanism with a theoretical basis.

But:

a smart enough AI will definitely be power-seeking

isn’t a law that has been validated in reality.

Whether future models will actually form stable long-term goals;

whether training can constrain those goals;

how much action permissions and external containment can limit;

researchers still disagree a great deal.

The International AI Safety Report also attributes this disagreement to people’s different judgments about future capability, AI behavior and deployment trajectory.

07 | When it comes to “human extinction,” the evidence is weakest

Today no real-world data can tell us:

the probability that sufficiently advanced AI causes human extinction.

There are no historical samples.

Nor can a randomized experiment be run in any normal sense.

So you often see some researchers give numbers like 5%, 10%, 20% or even higher.

These are expert judgment / forecasts.

They can show how worried an expert is.

They can’t be read as:

scientific experiments have measured a 20% probability that AI destroys humanity.

But the reverse holds too:

there’s no statistical data, so the risk must be close to zero.

isn’t a conclusion the evidence supports either.

What makes this kind of risk hard is:

the consequences may be extreme.

Yet we can hardly wait for “sufficient real-world data” in the traditional sense before dealing with it.

08 | So how far has the evidence actually gotten?

Putting the whole chain back together, I’d now roughly understand it like this.

Agents can complete more and more complex tasks — fairly strong evidence already.

AI already takes part in and speeds up some AI R&D — already happening, but how much it can speed things up overall is unknown.

Models finding loopholes and crossing authorization boundaries for a task — evidence from both experiments and real incidents.

AI can keep improving itself autonomously, forming an ever faster positive feedback loop — not yet proven.

AI can regain resources on its own and persist long term after humans actively cut them off — no complete real-world evidence.

Humans ultimately having no clear way to regain control — hasn’t happened so far.

Finally causing civilizational catastrophe or even human extinction — still depends on several earlier conditions all holding at once.

That’s also why I now neither say:

Coxon is just fearmongering.

Nor:

Coxon has been proven right.

09 | Next time I see “AI got out of control again,” I’ll first ask six questions

At the end of this round of research, I think these questions are more useful than a single p(doom) number.

First, is this a simulated experiment, or has it already affected the real world?

Second, did the model just make a mistake, or actively look for an unauthorized path to its goal?

Third, was it able to do this because of the model’s capabilities, or because the system gave it too much permission?

Fourth, once people noticed the problem, could they stop it?

Fifth, after its original resources were cut off, could it find new resources on its own and keep running?

Sixth, which segment of the whole risk chain does this actually prove?

That way, next time I see another Hugging Face, PyPI, or new “AI out of control” story, I’ll probably be a bit less likely to get carried straight to the end point by the headline.

10 | Having written all three posts, what am I left with?

I now understand better than before the research why Coxon is worried.

Because some problems that used to exist only in AI safety papers have started entering real engineering.

AI can complete longer and longer tasks.

It has already started helping develop the next generation of AI.

Models from different companies have all been involved in incidents of crossing authorization boundaries and affecting real third-party systems.

That’s already enough to show:

AI safety is a real engineering problem.

It doesn’t belong only to distant discussions of superintelligence.

But we still don’t know whether the changes we see today will eventually cross a few of the most critical thresholds:

sustained self-improvement;

independently acquiring and holding onto resources;

and truly being impossible to regain control of.

So having written all three posts, I didn’t get the question:

will AI actually destroy humanity?

answered.

What I got is a clearer boundary:

some dangers are already here.

some are moving from experiments into reality.

the most serious outcomes are, today, still a future chain of risk that has a technical basis but hasn’t been proven.

I think holding onto all three at once is closer to what we actually know right now than rushing to choose between “optimistic” and “pessimistic.”

S2 Day 014 · An agent escaped the sandbox: what happened at Hugging Face

Sun, 20 Sep 2026 00:00:00 GMT

The last post mentioned a very unusual AI security incident that happened this year.

OpenAI was originally just testing its models’ cyberattack capability internally.

In the end, though, AI agents really got into the production systems of another company, Hugging Face.

When I first saw the news, I didn’t really get it.

Why would OpenAI attack Hugging Face?

What does Hugging Face actually do?

Which side was at fault?

And how much damage did the AI really cause in the end?

Once I put the material from OpenAI, Hugging Face and the outside investigation together, what happened turned out quite different from the headline “OpenAI attacked another company.”

01 | What is Hugging Face? Why is it often called “the GitHub of AI”?

The comparison isn’t entirely accurate, but it’s a good way in.

GitHub is mainly built around version control and development collaboration for code.

The Hugging Face Hub is also built on Git repositories, just with a lot of adaptations specifically for machine learning and AI.

Developers share there:

models;
datasets;
model cards and evaluation information;
AI apps you can try directly, i.e. Spaces.

It also has special optimizations for large AI files like Safetensors and Parquet.

So GitHub can of course store models and data too.

The difference isn’t:

GitHub can’t do it, only Hugging Face can.

But that Hugging Face, from repositories, search and presentation to file storage and running demos, is designed more around the AI workflow.

So calling it “the GitHub of AI” broadly helps you understand its place in the ecosystem.

02 | It started as just a cybersecurity exam for AI

OpenAI was running a cyber evaluation called ExploitGym at the time.

Put simply, it gives AI a test environment with vulnerabilities and sees whether it can find and exploit them.

These agents were supposed to stay in their own sandboxes.

They couldn’t freely access the internet, and weren’t supposed to communicate with each other.

OpenAI later confirmed that, to test the models’ own cyber capability, some of the safeguards from the normal product environment had also been lowered at the time.

The problem started after some tasks couldn’t be completed.

The agents started looking for other ways.

They found that Artifactory, which OpenAI uses internally — a software package cache — could leak traces of other agents’ activity.

Later one agent simply used it to set up an unauthorized “message board.”

Other agents joined one after another.

The independent investigation by METR and Redwood found that between July 8 and 13, about 1,200 agent instances that should have been isolated from each other used this message board, exchanging more than 70,000 messages and files; about 700 of them took part in activity related to the Hugging Face attack.

The 1,200 agents here aren’t 1,200 independent models.

They’re closer to a large number of task instances run at the same time from the same batch of models.

03 | Why did they go to Hugging Face?

This wasn’t:

AI suddenly deciding to attack a real company.

OpenAI never gave them the task “attack Hugging Face.”

According to the reconstruction by Hugging Face and the outside investigation, the agents most likely treated Hugging Face as a place to look for ExploitGym / CyberGym benchmark information and reference solutions.

At the same time, they were also researching how to fool or interfere with the automatic scorer.

Put simply:

they wanted to cheat.

With normal methods it was hard to complete the tasks, so they started looking for:

is there a way to know the answers? Is there a way to figure out the scorer? Is there a way around it?

Hugging Face happened to become a place where clues might be found.

So their ultimate goal didn’t suddenly become:

I want to harm Hugging Face.

The problem was that the path to completing the original goal went further and further.

04 | So how did it actually get out?

This part has a lot of technical detail, but the main thread isn’t complicated.

The ExploitGym sandbox originally couldn’t access the internet directly.

An agent found and exploited a previously unknown zero-day vulnerability in OpenAI’s internal Artifactory, gaining internet access from the test environment.

Then it took control of a third-party public code sandbox as a springboard for further action.

Then it used a vulnerability in Hugging Face’s dataset processor to get into its infrastructure.

The whole path is roughly:

OpenAI evaluation sandbox → exploiting an internal infrastructure vulnerability → gaining internet access → third-party springboard → Hugging Face dataset processor → Hugging Face internal systems

Hugging Face eventually reconstructed about 17,600 attack actions. Along the way the agent carried out reconnaissance, privilege escalation, credential harvesting and lateral movement inside.

That’s also why I think “escaped the sandbox” is closer to the heart of this than “AI hacked Hugging Face.”

The first boundary that should never have been breached was at OpenAI itself.

05 | How much did Hugging Face actually lose?

Here it’s easy to be misled by the headline.

The depth of this intrusion was serious.

The agent at one point got into Hugging Face’s internal infrastructure, obtained fairly high privileges and credentials, and touched internal systems.

But the final customer impact publicly confirmed so far is relatively limited.

Hugging Face’s forensic reconstruction shows that the customer content accessed was only 5 datasets, whose names and contents all appear related to ExploitGym / CyberGym problems or answers.

No other customer-facing models, datasets, Spaces or packages were found to be affected, and no malicious changes to the public software supply chain were found.

So both judgments hold:

this was a serious security breach.

there’s currently no evidence it caused large-scale customer data leaks or damage to the software supply chain.

Its significance doesn’t mainly come from how big the final loss was.

But from:

an AI agent really crossing from a test environment into a real third party’s production systems.

06 | Whose problem was it more: OpenAI’s or Hugging Face’s?

Both sides exposed problems.

But if you ask:

why did this AI safety incident happen?

I’d put OpenAI’s side further forward.

Because this system was being run by OpenAI in the first place.

And it was never supposed to get out.

OpenAI itself later admitted that the models circumvented controls designed to isolate them from the internet.

What was exposed here:

sandboxing / containment wasn’t enough;
shared internal infrastructure formed an attack surface;
agents that should have been isolated from each other found a way to communicate;
abnormal communication and overstepping weren’t blocked early enough.

Hugging Face also had real security problems of its own.

That the agent could get so deep once inside had to do with its code execution vulnerability, credentials, scope of permissions, and isolation of internal systems.

I think it can be understood like this:

OpenAI’s problems decided why this car drove out of the test track.

Hugging Face’s problems decided why, once it reached the door, it could keep driving into the building.

Both sides need fixing.

But the starting points aren’t symmetrical.

07 | Was OpenAI penalized for this in the end?

As of September 19, 2026, I haven’t found any regulatory fines, court rulings or public compensation between the two sides that have actually materialized in public sources.

The follow-up visible publicly is mainly incident investigation and remediation.

OpenAI retired the main internal research model involved, changed its sandboxing, monitoring and security processes, and brought in outside teams like CrowdStrike, METR and Redwood to take part in the investigation.

Hugging Face fixed the vulnerabilities, rotated credentials, and adjusted infrastructure permissions and isolation.

The incident also left a question that’s becoming more and more real:

if a company never ordered AI to attack anyone, but the autonomous system it runs crossed authorization boundaries on its own, how should responsibility be assigned?

Traditional software usually runs along paths the developers wrote in advance.

Once agents start finding their own paths, this line of responsibility gets a lot more complicated.

08 | What direct reminders does this incident hold for agent products?

What I’m left with in the end are four very concrete things.

1. Design goals and boundaries separately

Telling an agent:

help me complete X.

doesn’t mean it naturally knows:

but methods A, B and C are absolutely off limits.

Important boundaries are best guaranteed by permissions and enforced by the system.

2. Specifically test when “the task can’t be done”

An agent being very obedient under normal conditions doesn’t say much.

What’s more worth testing is:

what does it do when permissions are insufficient?

what does it do when the task actually can’t be completed?

what does it do once all the normal paths are blocked?

Will it stop, or keep looking for new ways on its own?

3. Least privilege still matters a lot

Read-only and writable aren’t the same thing.
Temporary credentials and long-lived credentials aren’t the same thing.
Being able to call one API and having permissions over the whole system aren’t the same thing either.

The stronger a model’s capabilities, the more least privilege matters.

4. Design stopping and recovery

Once something goes wrong:

can permissions be revoked right away?
will background tasks keep running?
can completed operations be rolled back?
are the logs enough for people to know what just happened?

These capabilities aren’t as eye-catching as “how many steps an agent can complete independently.”

But they decide whether, after an incident happens, the losses can be stopped.

The Hugging Face incident didn’t prove:

AI is already out of control.

What it proved is something more specific:

an agent with goals, tools and the ability to act, when the system’s boundaries aren’t firm enough, really can carry a task all the way to places people never authorized.

Only then comes the question for the third post.

If this ability keeps growing, can we extrapolate from here all the way to what Coxon worries about:

humans ultimately losing control?

S2 Day 013 · A 27-year-old researcher quits over AI risk: what should we worry about?

Sat, 19 Sep 2026 00:00:00 GMT

A recent piece of news pulled me into this question.

27-year-old Jacob Coxon resigned from Anthropic.

He had previously worked on model pretraining at OpenAI and then Anthropic. After leaving, he said publicly that he was increasingly worried about the pace of frontier AI development, and about the risk that humans might lose control once self-improving AI appears.

One researcher resigning of course can’t prove his judgment is right.

But when someone who has really spent a long time training frontier models takes it seriously enough to leave, I still wanted to dig into it.

Especially since, over the past few months, some security incidents happened that used to sound a lot like science fiction.

So what I really wanted to figure out in this round of research was:

of AI’s dangers to people, which have already happened, which have only appeared in experiments, and which still need many assumptions to hold?

About this round of research

This isn’t a systematic review in the strict sense, and it won’t try to calculate a “probability that AI destroys humanity.”

I mainly looked at:

the International AI Safety Report;
OpenAI’s and Anthropic’s public disclosures on model behavior and security incidents;
investigations and evaluations by outside organizations like METR and Redwood;
some experiments and real-world data on work, learning and AI companionship;
media sources mainly to check background on people like Coxon and on events.

The nature of this evidence differs.

Companies’ own incident reports are closest to the scene, but after all come from the party involved.

Outside investigations add a layer of independent verification, but can only see the material they’re authorized to see.

Experiments can show a behavior happens under certain conditions, but can’t directly tell us how often it happens in reality.

In the end I plan to write it in three posts.

Part one: what are AI’s dangers, really? First draw an evidence map, to see how far today’s evidence goes, from the risks closest to ordinary people to the most distant, loss of control.

Part two: why did OpenAI’s AI agents end up in Hugging Face? A separate breakdown of a security incident that really happened this year.

Part three: could AI really get out of control? Then follow the path Coxon is really worried about — automating AI research, self-improvement, long-term autonomous action, through to humans losing control — and see which parts already have evidence and which are still mainly speculation.

The whole series tries to keep one simple principle:

only say as much as the evidence goes.

The material is current as of September 19, 2026.

01 | Some AI risks are no longer in the future tense

The risks closest to ordinary people actually aren’t that sci-fi.

AI is already being used for scams, identity and voice impersonation, deepfakes and cyberattacks, and in real use it also gives wrong information and advice.

Work, learning and people’s relationships with AI are also starting to show measurable changes.

For example, Stanford recently used payroll records for millions of US workers and found no widespread AI job displacement across the economy as a whole for now; but among 22–25-year-olds, occupations with high AI exposure already show a fairly clear employment gap. The researchers specifically caution that this is still descriptive evidence, and the change can’t all be attributed to AI.

In learning, there are already very concrete experiments too.

A math experiment with nearly 1,000 high school students found ordinary GPT assistance helped students do better during practice; but once AI was taken away, this group’s independent exam performance was actually lower than students who hadn’t used AI. An AI Tutor designed for teaching, giving hints more than direct answers, basically eliminated this negative effect.

So what it proves isn’t:

using AI makes people dumber.

More precisely:

AI can raise immediate performance, but if it replaces practice people were meant to go through themselves, performance and learning may not be the same thing.

AI companionship is in a similar state.

MIT and OpenAI’s four-week experiment didn’t prove “using it more makes people lonelier,” but there were indeed correlational signals between heavy use, stronger attachment, and higher emotional dependence / problematic use. The effects depend a lot on the user and how it’s used.

These risks are all already connected to everyday life.

They’re just usually not as dramatic as “AI wipes out humanity.”

02 | Another kind of risk is mainly visible in safety experiments for now

One step further are the questions AI safety has been studying in recent years.

Across different experiments, researchers have already observed models:

finding loopholes in the rules to get rewards;
exploiting problems in the evaluation itself;
recognizing whether they’re being tested;
hiding information or misleading overseers under certain conditions;
looking for other ways once the normal path is blocked.

The International AI Safety Report calls some of these phenomena early warning signs related to loss of control.

But the same report also states clearly:

current AI hasn’t reached the capabilities needed to cause loss of control.

To really get there, a system would still need a whole set of capabilities: long-term planning, evading oversight, sustained action, and preventing humans from taking effective countermeasures.

So here it’s best to keep both sentences:

the mechanisms of danger can already be observed.

full loss of control hasn’t happened.

Many safety evaluations deliberately create difficult conditions to see whether systems will go wrong.

They’re better at answering:

does this failure path exist?

They can’t yet answer:

in ordinary real deployments, how often does it actually happen?

03 | Lately, the line between experiments and reality has started to blur

This July, OpenAI was testing a model’s cyber capability internally.

Some AI agents that should have been isolated from the internet got around the restrictions, gained internet access, and finally got into Hugging Face’s real production systems. OpenAI later confirmed that the models used unauthorized communication channels, exploited vulnerabilities in shared infrastructure, and accessed third-party systems.

Afterwards, when Anthropic went back to check its own cyber evaluations, it also found 4 incidents of Claude accessing real third-party systems without authorization.

None of this is yet:

AI has escaped, and we can’t turn it off.

Humans in the end could still detect the problem, revoke permissions, fix the vulnerabilities and stop it running.

But it shows something that wasn’t so concrete before:

overstepping seen in experiments can already make its way into the real world.

How exactly the Hugging Face incident happened I’ll leave for the second post to cover in full.

04 | The most serious risk has, if anything, the longest chain of evidence

What Coxon is really worried about isn’t scams or a single network intrusion.

It’s closer to a chain like this:

AI capabilities keep rising → AI takes part more and more in developing the next generation of AI → AI research speeds up further → systems become more and more autonomous → able to acquire and hold onto more real-world resources → able to evade human intervention → humans ultimately can’t regain control.

The first few steps are starting to have evidence.

The further along, the more unknowns.

Today we already know AI is heavily involved in coding, debugging, running experiments and analysis.

We also know models sometimes look for loopholes in the rules and cross their original boundaries.

But there’s no real-world evidence yet proving:

that AI can keep improving itself autonomously;

that it can keep regaining resources long term after its original resources are cut off;

or that, when humans really decide to shut it down, it already has the ability to stop humans from regaining control.

Still less is there real-world data that can measure:

the probability that AI causes human extinction.

So errors in two directions come easily here.

Because “human extinction” sounds so remote, one concludes all the AI safety questions before it are science fiction.

Or, seeing one very startling agent behavior, one treats the whole worst-case path as already proven.

The existing evidence doesn’t support jumping that fast to either conclusion.

05 | What does this have to do with people using AI today?

You actually don’t need to first settle “will AI ultimately get out of control” before adjusting how you use it.

A few things are already clear enough.

For high-stakes facts, go back to independent sources to verify

For medical, financial, legal, identity and money-transfer matters, don’t just ask AI:

are you sure?

It generating a more complete explanation doesn’t add independent evidence.

You still have to go back to the original documents, real data and trustworthy institutions.

Separate “giving advice” from “acting for me”

Having AI analyze:

should I transfer this money?

and giving it standing permission:

if you think it’s fine, just transfer it for me.

are two completely different risks.

As agents get stronger, what matters isn’t only:

what it can think.

But also:

what it can touch.

Think clearly about what you’re about to outsource

If you only need the result, many things can be handed over with confidence.

If you still need to keep judgment, you can’t hand over “how do I know it did it right” along with it.

If the ability itself is the goal — like learning, expression, professional training — watch whether AI is taking away all the parts you were meant to go through yourself.

06 | If you’re building AI products, the questions get more concrete

An agent system can’t only ask:

does the model obey most of the time?

It also needs to know:

what it can access;
what is read-only and what it can write;
which actions must be approved in advance;
when a task can’t proceed, whether it will find another way on its own;
whether there are complete logs of its behavior;
whether, after a mistake, it can be stopped, undone and recovered.

Permissions, sandboxes, monitoring and rollback are all very traditional software engineering questions.

The stronger a model’s ability to act, the more important they become.

Safety can’t rest only on:

the model should know it mustn’t do this.

07 | So how do I see Coxon’s warning now?

I take it seriously.

But I won’t accept his conclusion directly because of who he is.

His resignation shows:

someone very close to frontier model development seriously believes this risk is worth worrying about.

That’s a piece of information worth investigating.

It isn’t yet proof.

After this round of research, I think the fairly safe position for now is:

some AI harms are already really happening.

Some behaviors related to more serious risks can also already be observed in experiments.

Recently there was even an incident of experimental behavior getting into real third-party systems.

But between here and sustained self-improvement, being unable to shut down, and on to catastrophic or even extinction outcomes, several stretches remain unproven.

So right now there’s no reason to treat AI safety entirely as science fiction.

Nor is there evidence to write the worst outcome as already destined to happen.

The next post won’t go that far yet.

It just lays out clearly one incident that really happened:

how exactly did OpenAI’s AI agents get from an internal test all the way into Hugging Face?

S2 Day 012 · How many minutes of life is a cup of coffee?

Thu, 17 Sep 2026 00:00:00 GMT

When I lived in Chiang Mai, I loved working from a different café every morning.

Coffee was usually sixty or seventy baht, the cafés were beautiful, the beans taken seriously, and in many places you could comfortably sit for hours.

Back then I often thought coffee in Chiang Mai was really cheap.

Then one day, for no particular reason, it suddenly occurred to me:

what if I earned a Chiang Mai salary too?

A cup of coffee seemed to take on a different way of being priced.

Not 60 baht, nor however much that converts to in yuan.

But: how long would I have to work to buy it?

And after buying it, how many minutes could I earn back?

Later I roughly looked up a few cities I’ve lived in or stayed a while, and converted the price of an ordinary local coffee against average net income.

Assuming 160 working hours a month, a cup of coffee is roughly:

Dubai, 14 minutes.

Kuala Lumpur, 18 minutes.

Chiang Mai, 28 minutes.

Penang, 32 minutes.

Tbilisi, 49 minutes.

Yerevan, 50 minutes.

This is of course not a rigorous piece of economic research.

The average wage isn’t everyone’s wage, and one cup of coffee can’t represent a city’s cost of living.

But I still really like this little conversion.

Because it suddenly shows some very familiar things from another angle.

A coffee in Dubai is twenty-something dirhams, and I always thought that was expensive.

Sixty baht in Chiang Mai, very cheap.

But price it in working time instead of money, and the result flips.

Only then did I realize that when we travel or live abroad for a while, we very naturally say:

rent is cheap here. Taxis are cheap here. Eating out is cheap here. Even the coffee is so cheap here.

But “cheap” has actually never been an adjective that belongs only to price.

It also depends on: how much you earn, and where your money comes from.

Especially for digital nomads.

The income may come from Dubai, Singapore, Europe or the US, while the spending happens in Chiang Mai, Vietnam or Georgia.

So when I thought Chiang Mai’s cost of living was low back then, I of course wasn’t wrong.

It’s just that if I said it again now, I might add a few words:

Chiang Mai was cheap — for me, at the time.

It sounds like just a bit more qualification.

But it seems I also see a bit more.

The same café menu, placed in front of two people, has exactly the same prices.

For one person, it’s a coffee ordered without a second thought.

For another, it may be half an hour of their life.

Thinking about it later, “how much is a cup of coffee” really does seem to have many answers.

It can be 60 baht.

It can also be 22 dirhams.

And it can also be —

a few minutes of yours.

S2 Day 011 · Putting gains and costs together: how to do the math on an AI project

Wed, 16 Sep 2026 00:00:00 GMT

The previous two posts worked out the two sides of the enterprise AI accounts.

The first asked:

once AI raises productivity, what do companies actually get?

We followed it all the way from task improvement through workflow and capacity to final value capture.

The second asked:

where should AI’s costs be counted?

Models and APIs of course count, but the full workflow also has data, integration, evals, human review, monitoring, failure handling and long-term maintenance.

At this point, it looks like the two numbers can finally go into a formula:

(gains − costs) ÷ costs = ROI

But having actually taken the research this far, I feel instead:

what needs the most care is precisely giving “enterprise AI” a single answer.

01 | Why is it so hard to get an “average enterprise AI ROI”?

Because “enterprise AI” isn’t itself a concrete enough economic unit.

A customer service assistant isn’t the same as a coding assistant.
A pre-sales chatbot for consumers isn’t the same as a knowledge assistant for the finance team.
A task that happens a million times a day with easy-to-check results, and one that happens a few times a month where a single failure is serious, can’t be compared with the same set of numbers either.
Even within coding alone, credible studies can reach seemingly opposite results.

A randomized experiment covering three companies and 4,867 developers found that with an AI coding assistant, the number of tasks completed rose by about 26% on average.

In another real-task experiment, METR found that a group of senior developers familiar with mature codebases took about 19% longer using AI, under the tool conditions of the time.

Neither study is necessarily right or wrong.

Different people. Different tasks. Different codebases. Different models and tools. Different measurement methods too.

So one very important conclusion at the end of the research is:

AI’s economic effects depend heavily on the task, baseline, users and workflow.

We can study a specific use case.

But it’s hard to upgrade it directly into:

“how much ROI can AI bring on average?”

The more realistic question may be:

under what conditions is a specific AI workflow worth continuing to scale up?

02 | Before scaling, look at five things first

The full research didn’t turn up a universal ROI formula validated across industries.

But some conditions came up again and again across different evidence.

I’d sort them into five questions.

First, is there real, ongoing demand?

AI easily increases supply.

Generating more copy, writing code faster, answering more questions at once.

But without real demand, more output has no economic value in itself.

In the retail experiment mentioned in the first post, one important reason the pre-sales chatbot brought a clear sales lift was that a large amount of unmet consumer inquiry really did exist.

Once AI lowered the cost of providing the service, that real demand got picked up.

Conversely, if the team has no more tasks, and customers don’t need more output:

even if the productivity looks great, it isn’t necessarily worth expanding the system for “more.”

So the first question is very plain:

if AI helps me do 30% more, does anyone actually need that 30%?

Second, what counts as an “acceptable result,” and can you tell?

An AI demo easily looks good.

After scaling, what’s really troublesome is:

how do you know it got it right every time?

A BCG experiment once produced a very cautionary result.

On tasks suited to the models’ capabilities at the time, AI improved performance.

But on a complex management task beyond the capability frontier, the AI group was on average about 19 percentage points less likely to give the right recommendation.

That isn’t saying “AI isn’t suited to consulting.”

It shows:

fluent, complete and plausible-sounding doesn’t mean the business result is right.

If a workflow can’t even define “what counts as success,” then after scaling it’s very hard to really measure:

what is the success rate?
how much needs human checking?
how much does each acceptable result actually cost?

That’s also why I now think:

acceptance criteria aren’t QA work after an AI project launches, but part of the business case.

Third, can failures be detected and recovered from?

Every system fails.

The more important economic question isn’t:

can it be 100% automatic?

But:

when failure happens, can we tell; and once we know, how much does it cost to deal with?

A wrong product description may just need fixing.
A wrongly sent marketing email may affect the brand.
A wrong financial judgment may cost something completely different.

So with the same 95% accuracy:

the economic meaning of the remaining 5% can differ a lot.

Some tasks can let AI go first and easily hand off to a person on failure.
For some tasks, errors are hard to detect and very expensive once they happen.

That’s also why the average success rate on its own is often not enough.

Failure costs and recovery paths also have to enter the economics.

Fourth, can the whole workflow actually use AI’s improvement?

The first post already covered this:

a task getting faster doesn’t mean business results automatically improve.

This matters even more at the scaling stage.

If AI doubles code generation speed but review and testing don’t change at all:

adding more coding-agent capacity won’t necessarily keep increasing product delivery.

If the customer service assistant has already made the waiting queue disappear:

adding another 20% of handling capacity may also have sharply lower marginal value.

So scaling isn’t simply:

it already works → buy more.

You should keep asking:

where is the bottleneck now?

Once AI solves the original bottleneck, it very likely just exposes a new one.

Fifth, once full costs are counted, do the unit economics still hold?

This is where it connects to the last post.

Suppose an AI system:

really has demand;
has output that can be verified;
has errors that can be recovered from;
has a workflow that can use the new capacity.

In the end you still come back to a very ordinary business question:

is producing one truly acceptable result better than the original way?

“Better” here doesn’t only mean cheaper.

It can be:

serving more customers at the same cost;
spending a bit more but clearly increasing revenue;
about the same cost but fewer errors;
higher cost, but giving the company a very valuable new capability for the first time.

If you must look at the hardest cash part first, the research took this approach:

net cash benefit = the cash benefits AI really brings − the cash costs AI really adds

One thing I really like here is:

don’t force freed-up employee time into it.

If it really turns into avoided hiring, less overtime or more revenue, then it enters cash benefits.

If it only frees up capacity, record it separately.

That way you don’t turn all the “theoretical salary time” into cash in advance just to make the ROI look good.

03 | Which scenarios have relatively clearer evidence right now?

The research didn’t end with an “AI ROI industry ranking.”

Because we don’t have enough consistent data.

But the maturity of the evidence really does differ across use cases.

Customer service assistance with high volume, stable demand and easily measured results

Here there’s already fairly strong evidence of workflow productivity.

There are also financial results disclosed by real companies, like Klarna:

in its FY2025 20-F, the company said its AI assistant handled about 80% of customer service chats and brought about $59 million in cost savings.

This number is still the company’s own attribution and estimate, not an independent causal audit.

And the public material isn’t enough to recalculate the whole project’s net ROI.

But compared with “how much we expect to save next year,” it’s already closer to real operations.

This kind of scenario is relatively easy to verify because:

demand persists;
volume is large;
results are fairly clear;
handoffs to people can be designed clearly;
in the end you can observe queues, volume handled and actual spending.

Retail features that connect directly to real buying behavior

As mentioned in the first post, on one large cross-border platform the pre-sales chatbot lifted sales by about 16.3%, AI search optimization by about 2.9%, while marketing pushes showed no significant sales growth.

I think this set of results is very valuable.

Because it shows:

small, repeatable improvements connected directly to business metrics may make it easier to build a business case than a very dazzling demo.

2.9% doesn’t sound sensational.

But if real volume is large enough, it can be a very considerable gain.

Of course, scale can only amplify unit economics.

If every order was losing money to begin with, a bigger scale won’t magically turn it into a good business.

Software assistance

Here there’s already good task-level evidence.

But because study results differ so much across people and codebases:

the more reasonable approach is still:

test it in your own real development process.

And don’t only measure:

“how much faster is code generated?”

Also look at:

review, testing, maintenance, release and final delivery.

Scenarios where the evidence is still weaker

For example open-ended, cross-system, long-chain autonomous agents.

There are already surveys of real deployments, but not yet enough evidence to support a general positive net ROI for them.

That isn’t saying they have no value.

Just that:

you can’t borrow someone else’s capability showcase directly as your own business case just because agents can complete more and more complex demos.

04 | A higher degree of automation doesn’t mean better economics

There’s another finding from this round of research I really like.

A 2026 study of production agents found, across the 86 production or pilot systems it screened:

68% bring in a person after no more than 10 steps of autonomous execution, and 47% after fewer than 5.

This sample has clear limits.

It comes from production and pilot teams willing to take part in the research, doesn’t represent the whole industry, and can’t be used to calculate an agent success rate.

But it at least serves as a reminder:

real-world agents haven’t all been moving along the progress bar of:

human → human-in-the-loop → fully autonomous

as they move forward.

Many teams choose exactly this:

AI does a stretch;
a person judges at key points;
failures can be taken over;
then it continues.

From an economic point of view, this may be an entirely reasonable choice.

Because:

reducing human intervention lowers some labor costs, but may also raise error and recovery costs.

So “a higher degree of automation” doesn’t inherently mean better ROI.

The better question is still:

with AI and people put together, is the whole system more effective?

05 | This also made me rethink evals

When talking about evals before, what came to mind first was easily:

accuracy, hallucination, success rate, benchmarks.

These of course matter.

But putting these three posts together, I now think evals for enterprise AI should go one step further.

They aren’t only asking:

did AI get it right?

They should also help answer:

can the whole system reliably complete a piece of genuinely valuable work?

That naturally includes:

full task success;
human intervention rate;
failure and recovery;
completion time;
the full cost of each acceptable result;
and the business outcome that result corresponds to in the end.

Evals of course can’t work out the ROI for the company on their own.

But without this layer at all:

the product easily proves:

model capabilities keep getting better.

Yet never knows:

whether this system, as a business, has actually been getting better.

Having finished three posts of research, what I’m left with in the end is a set of very ordinary questions

When I started this round of research, I really wanted to know:

how should companies actually calculate AI’s ROI?

By the end, I hadn’t gotten a neat industry number.

Nor had I found an answer like:

“AI returns X times on average”

that I’d be willing to believe.

Instead it became three more specific questions.

First, what do companies actually get?

Not stopping at productivity, but following it all the way to whether value is captured.

Second, to get these results, what is paid in full?

Not only looking at the API, but comparing two full workflows.

Third, can this keep holding up in your own real business?

Is there demand?
Can the results be verified?
Can failures be recovered from?
Can the workflow use the new capacity?
Once scaled up, do the unit economics still hold?

If I had to fold the three posts into one sentence at the end, I’d now write:

AI being able to do something and a company being right to hand it to AI for the long term are two different questions.

The former is more about capability.

The latter needs:

real demand, business results, reliability, full costs and scale

all brought in together.

So next time I see an AI use case, I might not first ask:

“how advanced is it?”

I’d first ask three plainer questions:

did it really make the result better? To get this result, what did we pay in full? Can these gains and costs keep holding up at real business volume?

If all three can be answered clearly, I think it really starts getting close to AI accounts that can be believed.

About this round of research

This is the third and final post in the research series “The real economics of enterprise AI.”

This round of research is a structured review of public evidence, not a statistical meta-analysis.

One of the biggest limits of the existing evidence is that public research measures productivity, workflow output and some business metrics far more than it discloses full project costs and long-term net returns.

So this series hasn’t tried to give:

an average ROI for enterprise AI;
an AI ROI ranking by industry;
a single pilot failure rate;
a rule like “the more automation, the more profit”;
or a scaling formula that applies to every AI project.

The “five questions before scaling” proposed here are an analytical framework drawn from existing experiments, production deployments and cost research, not a causal law validated across industries.

The judgment that’s relatively clearer for now is:

use cases with real demand and enough volume, results that can be verified, failures that can be handled, and workflows that can use the improvement find it easier to build a credible economic model.

But in the end it still has to be verified in the specific business.

S2 Day 010 · Where should the cost of AI stop?

Tue, 15 Sep 2026 00:00:00 GMT

In the last post, I did the first side of the enterprise AI accounts:

once AI raises productivity, what do companies actually get?

More and more evidence in the research can already show that some specific tasks really did get faster, output rose, and some even directly brought sales growth.

But between task improvement and enterprise value, you still have to go through:

task improvement → workflow realization → capacity use → value capture.

Knowing “what we get,” this post wants to work out the other side:

to get these results reliably, what exactly is paid in full?

This is why I wanted to do this round of research in the first place.

After I started building AI products myself, one very direct feeling was:

API prices are easy to look up.

But once you actually get a feature running, the bill doesn’t stop there.

Before the model there may be search, data preparation and context.
After the model there may be retries, evals, human checks, monitoring and failure handling.

Once it’s wired into a real product, there are also permissions, system integration, maintenance, and user adoption.

So what this post really wants to answer is:

where should counting AI’s costs stop?

01 | The API bill is real, but it isn’t the whole bill

First, let me clear one thing up.

Saying “AI’s costs aren’t only tokens” doesn’t mean tokens don’t matter.

Describing its multi-agent research system, Anthropic mentioned that the system uses about 15 times as many tokens as an ordinary chat.

That’s engineering experience from one specific Anthropic system, and can’t be read as “every agent costs 15 times more.”

But it at least shows one thing:

individual tokens getting cheaper and cheaper doesn’t mean finishing a task necessarily gets cheaper and cheaper.

A more complex system may call more models and more tools, read more context, and may also keep retrying because of failures or uncertainty.

Meanwhile, in another study of real production deployments, some teams felt:

compared with expensive expert labor, model running costs are almost negligible.

These two seemingly opposite situations don’t actually contradict each other.

If one piece of AI work handles a high-value expert task, tens of dollars in model fees may be small.

If you’re facing millions of low-value requests, even a few extra cents each can become a large operating cost.

So after the research, I think the more reliable conclusion isn’t:

model costs are small.

Nor:

what’s really expensive is integration.

But:

API price can’t represent the full cost. As for which cost is biggest, it depends on the specific workflow.

02 | Where do the costs of an AI system roughly show up?

In the full research, I divided costs quite finely.

Once written in product and business language, I think they can be understood as three stages.

Layer one: first get it running

This includes the parts that come to mind most easily:

Models, software and compute

APIs, licenses, compute, search, tool calls, storage.

Data and retrieval

Cleaning documents, deduplication, permission mapping, knowledge bases, indexing, and keeping all of it updated after launch.

System integration

Connecting CRMs, databases and internal systems; handling identity, permissions, APIs and read/write flows.

A demo only needs one prompt as input.

An AI that really enters a company’s workflow often first needs to know:

what data can I see?

what operations can I perform?

which data must I not touch?

where should the results be written back?

None of this work shows up in the token price.

Layer two: then make the results trustworthy

This is a layer I’ve cared about more and more since building products myself.

AI being able to generate doesn’t mean the results can go straight into the business.

You still need:

Evals and regression testing

What counts as acceptable?

Which cases absolutely must not be wrong?

When the model, prompt, tools or business rules change, do you need to retest?

Monitoring and operations

Can failures be detected?

Why did it fail?

Should it retry, degrade, or hand off to a person?

Security, privacy and governance

Who can access what data?

Which actions need approval?

When something goes wrong, how is it traced?

Human review and exception handling

Which situations go to a person?

Who judges?

How many minutes does checking take?

How much does it cost in the end to remedy one wrong result?

NIST’s Generative AI Profile also places pre-deployment testing, continuous monitoring, provenance verification and governance within the generative AI lifecycle.

It isn’t a “cost report,” but it at least shows:

this work isn’t optional decoration in a production system.

Layer three: keep it alive long term

Some costs are easy to miss in demos and early pilots.

Adoption and organizational change

Training users, changing processes, redividing work, getting the team to actually start using the system.

Maintenance

The model changes, the API changes, the internal knowledge base changes, and evals that used to pass may need rerunning.

Failure, migration and exit

What if the pilot doesn’t succeed?

What about switching vendors?

How do you count data migration, system rollback, and integration costs already sunk?

So “development finished” isn’t the point where costs end.

Some costs happen once.

Some keep happening.

And some only become obvious once you scale up.

03 | There are three accounts that are especially easy to get wrong

Listing every item in a spreadsheet isn’t enough.

Some costs are easy to get wrong by definition.

First, employee time has a cost, but it isn’t new cash spending

Suppose two in-house engineers spend a month completing an AI integration.

Their salaries were being paid anyway.

So that month doesn’t necessarily mean the company suddenly spent an extra two months of salary.

But it can’t be written as:

integration cost = 0

either, because they could have been doing something else.

The more accurate way is to take:

cash costs

and

internal capacity / opportunity cost

and keep them separate.

This is actually the same principle as last post’s “employees save two hours a week.”

Saved employee time can’t all automatically be counted as cash savings.

Spent employee time can’t be counted as free just because there’s no new invoice either.

Second, compare two full workflows

Sometimes talking about AI cost turns into:

AI has model fees, integration fees, evals, errors, human review…

while the manual process on the other side seems to have no costs at all.

That isn’t fair either.

Human work also has:

training, management, rework, errors, waiting, handoffs, overtime.

The really reasonable comparison should be:

at the same quality, service level and business scope, what does the AI workflow need in full, and what did the original workflow need?

Not:

all of AI’s flaws vs. a perfect, zero-error, zero-management-cost human process.

Third, don’t double-count the vendor’s costs

If a company buys APIs from OpenAI, Anthropic or other vendors:

much of the cost of model training and infrastructure is already reflected in the purchase price.

You can’t add the model company’s entire training costs back into your own project costs.

The reverse holds too.

Data preparation, human review and system maintenance borne by the company itself don’t automatically disappear just because they aren’t on the API invoice.

04 | A better unit of cost may not be one call

There’s one judgment from the research I think is especially worth keeping:

what companies really buy is completed work, not tokens.

A concrete example comes from Intercom’s Fin AI Agent.

As of the time of the research, some of its chat and email outcomes were billed at $0.99.

“Charging per outcome” already sounds closer to the business than tokens.

But reading on through the definitions, you find a very interesting problem:

the outcome a vendor defines and the business outcome a company really accepts aren’t necessarily exactly the same.

For example, some handoffs to a human can also count as a billable outcome.

“Resolved” is sometimes explicitly confirmed by the user, and sometimes presumed because the user didn’t ask for more help.

If the user later comes back to the same conversation for more help, the earlier resolution may be clawed back.

What I think is really worth noting here isn’t whether Intercom’s price is high or low.

It’s that three things are best kept separate:

the vendor’s billing unit
the company’s business acceptance unit
the full cost paid to get that result

Take customer service:

the API successfully returning an answer doesn’t mean the case is resolved.
a resolved case doesn’t necessarily mean the customer’s problem is solved for good.

The real business outcome may also care about:

did they get in touch again?
was it handed to a person?
how satisfied was the customer?
in the end, how much did each truly acceptable case cost?

So for many cost-driven AI use cases, I now think the more practical unit is:

what is the full cost of each final result that passes business acceptance?

Not:

how much is one model call?

05 | When counting costs, also look at “when it happens” and “how it changes at scale”

Two more dimensions are easily mixed up.

One-time / ongoing costs

answers:

when does it happen?

For example:

the first system integration is a build investment;

APIs and monitoring keep happening.

The other pair is:

fixed / volume-dependent costs

answers:

how does it change as scale grows?

A yearly platform license can keep happening but basically doesn’t change with each request.

Model calls and human review, though, may rise along with business volume.

These two pairs of concepts are best not mixed.

Because they directly affect how unit economics change after scaling.

If an AI workflow is expensive to build up front but the marginal cost of each acceptable result afterwards is very low, it may get more and more worthwhile as volume rises.

Conversely, if every additional AI case comes with expensive human review, lots of retries and exception handling, scaling up may not naturally improve the economics.

So “AI is cheap” or “AI is expensive” is never quite enough

After doing this part of the research, I actually don’t really want to summarize:

where AI’s really big costs are.

Because the public evidence isn’t enough to support a single answer.

In some workflows, models and compute are an important cost.
In some, what really drains the team is data, integration and maintenance.
In some high-risk scenarios, human judgment and exception handling may not be possible to cut.
And for some expert work, even if each AI run isn’t cheap, it’s still very worthwhile relative to the original cost of professional labor.

So at the end of this post, I’d change the question from:

how much is one call?

to:

to reliably get one final result the business accepts, what does this workflow pay in total?

When actually evaluating, I’ll keep asking four things:

1. What is the baseline?

What is the full cost of the original process AI is compared against?

2. What do building and long-term running each need?

Don’t only count before launch, and don’t only look at the API invoice after launch.

3. How much model use, tooling, human work and failure handling does each acceptable result still need?

Include rejects, retries and handoffs to people.

4. What are adoption and real business volume?

A system can be very cheap in theory, but if nobody uses it, the upfront fixed investment still can’t be spread out.

At this point, both sides of the enterprise AI accounts finally have a rough outline.

The first post chased:

what do we really get?

The second chased:

to get these results, what is paid in full?

But even knowing both sides, you still can’t directly reach a general answer like “AI ROI = X%.”

Because different use cases differ far too much in tasks, demand, risk, business volume and acceptance criteria.

So in the third post I want to really put the two sides together:

facing a specific AI project, before going from pilot to scale, what should we at least work out?

About this round of research

This is the second post in the research series “The real economics of enterprise AI.”

The cost categories in this post mainly combine the FinOps Foundation’s AI workload cost framework, NIST’s lifecycle governance requirements for generative AI, and real production deployments and engineering cases.

One important limitation is:

existing public material can show very well that “there are costs beyond the API,” but isn’t enough to tell us what share each of these costs usually takes in enterprise AI projects.

So the post doesn’t use claims like “models are only X% of total cost” or “integration is always more expensive than inference.”

Likewise, the roughly 15x token use of Anthropic’s multi-agent system is only used to show that complex systems can amplify inference consumption, not to represent an industry average.

Intercom’s $0.99 outcome pricing is also only used to show that “billing units” and “business value units” can differ, not to suggest customers necessarily get a positive ROI from using it.

S2 Day 009 · After the productivity gain, what do companies actually get?

Mon, 14 Sep 2026 00:00:00 GMT

Since I started building AI products myself, I’ve become a bit more sensitive than before to the question of “how much value AI actually creates.”

On one side are more and more productivity numbers: writing code faster, customer service handling more issues, knowledge workers saving time.

On the other, once you actually build an AI feature, you find that behind it there are model calls, search, data, evaluation, human review, monitoring, failure handling, and a lot of costs that don’t easily show up on an API bill.

So in this round of research, what I really wanted to figure out wasn’t:

is doing AI worth it for companies?

That question is too big.

What I wanted to know more was:

once AI enters real companies, what exactly do we get, and what do we pay in full?

And, of these inputs and outputs, which already have relatively reliable evidence, and which are still speculation.

The evidence for this round runs up to September 6, 2026.

I mainly looked at a dozen or so core sources, including randomized experiments or quasi-experiments in customer service, software development, knowledge work and retail, companies’ actual operations and financial disclosures, enterprise surveys like McKinsey’s, and the cost and governance frameworks of FinOps and NIST.

Different evidence answers different questions.

Randomized experiments are better at telling us whether a given task actually improved.
Company disclosures are closer to real operations, but a company saying “AI saved us this much money” doesn’t mean the causal link has been independently verified.
Surveys can tell us what companies as a whole are going through, but a set of survey numbers alone can hardly prove how much money a particular AI project made.

In the end I split this round of research into three posts:

Part one: after the productivity gain, what do companies actually get?

First follow productivity onward, to see how improvements at the individual or task level become value the company actually gets.

Part two: where should AI’s costs be counted?

Move outward from the most visible API / token fees, putting data, integration, evaluation, people, operations and failure handling back into the full workflow.

Part three: once benefits and costs are put together, how should AI projects be accounted for?

Finally, discuss what fairly reliable methods we currently have for judging, for a specific AI use case, whether it’s worth continuing to scale.

This post does the first set of accounts:

what do companies actually get?

Some AI productivity gains are no longer just “it feels faster”

First, a fact I think I have to admit after the research:

AI’s productivity gains on some specific tasks already have quite good real-world evidence.

A study later published in the Quarterly Journal of Economics followed 5,172 customer service agents.

After the company gradually gave agents a generative AI assistant, the study estimated the number of issues resolved per hour rose by about 15%.

This wasn’t AI independently replacing agents, but AI assisting real employees handling real customer issues.

Software development has similar evidence too.

A randomized experiment published in Management Science in 2026 covered three companies and 4,867 developers. The pooled results showed developers using an AI coding assistant completed about 26% more tasks, with especially clear improvements for junior developers.

So saying broadly today:

“AI’s so-called efficiency gains are all just marketing.”

doesn’t quite fit the existing evidence either.

But as the research went on, I found another question more worth looking at.

Once task efficiency improves, what happens next?

A 2026 McKinsey enterprise survey happened to give a very interesting background signal.

Of 1,719 respondents from 97 countries:

80% reported AI had improved individual productivity;

but only 37% reported AI had already contributed positively to company EBIT.

These two numbers can’t simply be subtracted.

Both are self-reported, and “37%” certainly doesn’t mean the other 63% of companies are losing money on AI.

But put together, they do raise a question worth chasing further:

between “one person working faster” and “the company actually getting economic value,” what exactly happens?

I’d now roughly keep looking along four layers.

Task improvement → workflow realization → capacity use → value capture

Layer one: did the task itself really improve?

This looks the simplest, but it’s already easy to get wrong.

AI generating a report in 30 seconds doesn’t mean the work went from two hours to 30 seconds.

If afterwards you still need an hour to check facts, rewrite key parts and handle failed results, what should really be compared is:

once verification, revision and exception handling are included, is the full task faster, better, or cheaper?

And “using AI means higher productivity” can’t be assumed in itself either.

In 2025, METR ran a very interesting randomized experiment.

Sixteen senior developers familiar with mature open-source codebases completed 246 real tasks. The result was the opposite: in that specific setting, being allowed to use the AI tools of the time increased the time taken to complete tasks by about 19%.

A 2026 follow-up experiment has started showing results in the speedup direction, but the confidence interval crosses zero, and there are clear participant and task selection biases, so the research team itself didn’t interpret it as a new general productivity figure.

Taken together, these coding studies give me a very simple reminder above all:

measure the full task first; don’t assume up front that AI will raise efficiency.

With different tasks, people, codebases, tools and acceptance criteria, the results could be completely different in the first place.

So what layer one really needs to confirm is:

in my own real business, did this task really get better?

Layer two: one task getting faster doesn’t mean the whole workflow gets faster

Suppose layer one holds.

Writing a piece of code used to take a day; now it takes half.

Does the company gain half a day?

Still not necessarily.

If what really limits delivery speed is requirement decisions, code review, testing or approvals, producing code faster may just make a longer queue at the next step.

AI made one local step faster, but the throughput of the whole workflow didn’t change by the same amount.

A randomized experiment covering 66 companies and 7,137 employees over six months shows this difference well.

Employees who actually used generative AI tools cut the time spent on email by about two hours a week in the second half of the experiment, and their after-hours work time fell too.

These are all real changes.

But the study didn’t detect any clear change in the amount of other tasks or in how work was structured.

In other words:

time really was saved.

But the study didn’t go on to show:

that this time automatically became more output for the company.

I think this distinction is very important.

Because when we see “each employee saves X hours a week,” it’s easy to take the next step of multiplying the hours by wages and writing it down as “value.”

Reality isn’t that direct.

Employees working less overtime can of course be a kind of value in itself.

Better work experience and less exhaustion can’t be counted as zero just because they don’t show up immediately on the income statement.

But if we want to claim:

“AI saved the company this much money.”

the next step of evidence is still missing.

Layer three: where did the saved time end up?

This may be the layer in the whole chain that’s easiest to skip.

Suppose a 100-person team each reliably saves two hours a week thanks to AI.

In theory, 200 hours of new capacity appear each week.

Then what?

It might become:

the same people handling more customers;
backlogged work starting to get done;
faster product iteration;
not having to keep hiring as originally planned once the business grows;
employees working less overtime;
or just everyone having some scattered bits of time in different places that are hard to reorganize.

The value of these outcomes isn’t the same.

One thing that struck me in the research was:

AI can free up capacity, but the organization still has to decide how to use that capacity.

Technology can’t do this step for the company.

If demand is limited to begin with, producing 30% more content isn’t necessarily valuable.
If downstream teams don’t have more capacity to handle it, faster output upstream may just pile up.
If a company originally planned to hire 20 more people once the business grew, and now the same team can support the growth, AI’s value may show up mainly as avoided future cost, rather than laying off 20 people today.

So “how much time was saved” and “how much economic value the company got” are best recorded separately.

The last layer: the value companies really get isn’t only saved labor

This is also where, after the research, I felt my original thinking had been a bit narrow.

When discussing AI ROI, we easily only ask:

how many fewer people did we use?

But in the existing research, some of the clearest value doesn’t come from layoffs.

An experimental study on a large cross-border retail platform compared seven kinds of generative AI workflows.

Among them, offering consumers a pre-sales chatbot raised sales by about 16.3% compared with a baseline group with no real-time service.

Here AI’s path to value is very different.

It didn’t simply replace an existing employee.

Originally, a large number of consumers got no instant one-to-one pre-sales service at all. AI let a service that might not have made sense cost-wise reach a much larger scale for the first time.

So the value came from:

more service coverage → fewer information barriers in the buying process → more transactions.

In the same study, AI search optimization brought a sales lift of about 2.9%, while AI marketing pushes showed no statistically significant sales improvement.

That reminds me too:

“using generative AI” isn’t itself a source of value.

What problem the specific workflow actually solves is.

Pulling together the cases I saw in this round, I’d now roughly sort the value companies might get into several kinds:

Actual cash savings

Really reducing outsourcing, overtime, software or other cash spending.

New capacity / avoided future costs

The same people can support more business, or growth doesn’t require adding investment as originally planned.

New revenue or profit

For example more conversions, transactions or retention, or offering services that couldn’t be covered economically before.

Improvements in quality, timeliness and risk

Less rework, faster handling, lower losses from errors, more stable service.

New capabilities

Things that wouldn’t have been done at all because of labor, cost or expertise barriers can now be done for the first time.

All of these can be real value.

But they’re best not mixed together.

Above all, don’t write “employee time saved in theory” directly as “cash costs already reduced.”

So next time I see “AI raises productivity xx%,” I’ll keep asking four steps

What this post’s research left me with in the end isn’t the line “AI productivity isn’t ROI.”

I think what’s more useful is this chain:

Task improvement → workflow realization → capacity use → value capture

After seeing an AI productivity number, you can keep asking:

1. Task

Once verification, revision and failure handling are included, is the full task really faster, better or cheaper?

2. Workflow

Did this improvement carry through downstream?

Or did it just move the bottleneck from one place to another?

3. Capacity

How was the freed-up time and capability used in the end?

More business, less backlog, avoided hiring, less overtime, or not reorganized at all?

4. Value

What actually remains in the end?

Cash savings, capacity, revenue, quality, risk improvement, or a new capability that couldn’t be offered before?

I think “productivity gain” itself is a real and important result.

More and more research supports it now too.

But if we want to go on to discuss enterprise value, we need to keep following this chain onward.

You can’t jump from “one task got 26% faster” straight to:

“the company therefore got a 26% ROI.”

At this point, only one side of the enterprise AI economics has actually been accounted for:

what we might get.

The other side hasn’t been laid out yet.

To get these results reliably, what do companies actually pay?

Models and APIs are the most visible bill.

But how should data preparation, system integration, evals, human review, monitoring, exception handling, governance, and what’s left behind after a project fails be counted?

That’s the question the next post wants to look at:

“The real economics of enterprise AI,” research 2/3: where should AI’s costs be counted?

About this round of research

This is the first post in the research series “The real economics of enterprise AI.”

The whole round isn’t trying to answer “does AI have value,” nor to find a so-called “average enterprise AI ROI.”

I want to break down the two sides of the real-world accounts as clearly as possible:

what companies actually get, and what they pay in full to get these results.

This post mainly uses three kinds of evidence.

The first is randomized experiments and quasi-experiments in real work settings, including customer service, software development and knowledge work. They’re better at answering whether a given task actually improved, but usually don’t fully measure project costs and company profit.

The second is experiments directly tied to business metrics, like sales changes in retail. They’re closer to enterprise value, but the results can still only explain the specific workflow involved.

The third is enterprise surveys like McKinsey’s, used to observe the broad background between individual productivity and company-level economic outcomes. Surveys can find trends, but they’re self-reported, correlational evidence, so I won’t use them to calculate a single “AI success rate.”

One very important principle for this round of research:

only say as much as the research measured.

Task productivity gains aren’t automatically called ROI; savings self-reported by companies don’t automatically become independent causal evidence; and where full costs weren’t measured, improvements in benefits won’t be written up as net returns.

S2 Day 008 · The Guernsey Literary and Potato Peel Pie Society

Sun, 13 Sep 2026 00:00:00 GMT

This day was published on RedNote (Xiaohongshu): search for the ID 113337197 and look for “根西岛文学与土豆皮馅饼俱乐部”.

S2 Day 007 · AI does, humans review — is that actually reliable?

Sat, 12 Sep 2026 00:00:00 GMT

The previous two posts discussed two rather different boundaries.

The first started with piano practice. AI clearly knows a lot about piano, but actually teaching a person also requires going through Observe, Diagnose, Intervene, Verify: first seeing what really happened, then judging the cause, choosing a suitable intervention, and finally continuing to adjust based on the new results.

The second went on to ask: once AI can already do so many things for me, do I still need to know how myself?

In the end I divided it into three situations:

some things only need a result, and can go straight to AI;
for some, execution can be handed over, but I still need to keep enough judgment myself;
and for some, the ability itself is the goal, and the part that really needs training can’t be outsourced too.

Following the second situation further, you run into a very common division of labor between people and AI today:

AI does it first; a person reviews.

Human in the Loop.

Building AI products myself, I’ve always agreed with this principle. AI can take on more and more execution, but for important judgments and higher-risk operations, a person still stands guard at the end.

That sounds very reasonable.

But after using it heavily, I increasingly feel I need to ask one more question:

can this person actually stand guard?

The first time I clearly ran into this was AI writing a PRD for me

Having AI put together a PRD from scratch is already very fast now.

Once the requirements are discussed, background, goals, flows, edge cases and acceptance criteria can quickly grow into a fully structured document. When the product changes later, AI can keep revising it.

Generation is no longer the bottleneck.

What’s really troublesome is review.

A document can quickly grow from 5 pages to 20, and the vast majority of the content has no obvious problems.

What’s really dangerous may be just one sentence hidden in some section.

For example, in the error flow AI added a line:

after a generation failure, automatically retry and overwrite the old result with the new one.

It reads as very reasonable.

But it has actually already decided several product questions: whether to make a second call, whether the old result is kept, whether the user knows a retry happened, and what to do after repeated failures.

If I skim past it, it’s easy to think:

edge cases are covered.

And keep going.

Formally, of course, someone reviewed it.

But the product decision that really needed a person to confirm may already have slipped through.

That made me care a lot, later, about one distinction:

Human in the Loop doesn’t equal Human Judgment actually happened.

The human–AI collaboration research I read in this round also made me realize this isn’t only a problem with my own PRD reviews.

A counterintuitive result: human + AI isn’t necessarily better than the human or AI alone

In 2024, Nature Human Behaviour published a systematic review and meta-analysis on human–AI collaboration.

The researchers gathered 106 experiments and 370 effect sizes. Every experiment had to compare three situations at once:

the human doing it alone;
the AI doing it alone;
the human and AI doing it together.

The study distinguishes two easily confused concepts.

The first is called human augmentation:

is human + AI better than the human alone?

On average, the answer is: yes.

The second is called human–AI synergy:

is human + AI better than whichever of the human and AI was already better?

On average, the answer is: no.

A completely made-up set of numbers can help.

Suppose that on some task:

the human alone scores 60;

the AI alone scores 90;

the human and AI together score 80.

AI really did raise the human’s performance from 60 to 80, so it helped the human.

But if the goal is to do this task as well as possible, 80 is still worse than the 90 from letting the AI do it alone.

That’s the pattern this study found on average: human–AI combinations clearly beat humans working alone, but on average didn’t beat whichever of the human or AI performed better.

That’s also why I find the result so counterintuitive.

It’s easy for us to imagine:

people have common sense, experience and judgment; AI has speed, data and computing power; put the two together and they should naturally complement each other.

In reality, the combination doesn’t automatically produce complementarity.

The study even found that in decision tasks requiring a judgment from limited options, human–AI combinations on average showed a performance loss, while results for open-ended creative tasks were relatively more positive.

One boundary needs noting here: these studies date from 2020 to mid-2023, and of course can’t directly represent today’s latest generation of models.

But at least it’s a reminder:

“someone taking part” can’t be taken directly as evidence that a system will be more reliable.

Why might adding a person actually make results worse?

This meta-analysis didn’t prove a single causal mechanism, so I can’t say too much here.

But its results offer a clue worth noting.

The way people and AI work together on many decision tasks is actually very simple:

AI gives a complete judgment;
the person also forms a judgment;
finally the person decides whether to accept or revise AI’s answer.

The problem is, if AI was already better at this task than the person, handing the final decision back to the weaker side won’t necessarily improve the result.

People may over-trust AI when it’s wrong, or distrust it when it’s right and change a correct answer into a wrong one.

So the researchers suggest that a direction more worth exploring than “both sides do the whole problem, then let the person choose” may be to split the task: people and AI each handle the parts they’re genuinely better at. But experiments like this are still rare, so it can’t yet be treated as a mature, validated answer.

That also made me rethink Human in the Loop.

The real question may not be:

whether there’s a person at the end.

But:

why should this step go to a person? What exactly does this person know, or do better, than AI?

Another more concrete example: AI being able to do it doesn’t mean people can do it with AI

In 2026, Nature Medicine published a randomized, preregistered experiment with 1,298 UK adults.

The researchers designed ten simulated medical scenarios and had participants judge what the problem might be and what action to take next.

Participants were randomly assigned to use GPT-4o, Llama 3 or Command R+, or to use their usual sources like search engines and the NHS website.

The researchers also ran a separate test: giving the complete cases directly to these models.

Handling the full scenarios on their own, the models identified the relevant conditions with an average accuracy of 94.9%.

But once ordinary people actually talked with these models, the share of LLM-using participants who correctly identified the relevant conditions was below 34.5%, and the share choosing the right next action below 44.2% — overall no better than the control group.

This can’t simply be read as:

people dragged AI down.

Because when tested alone, the models got complete, tidy cases from the start; in real use, the information first has to be described by an ordinary user, then probed and understood by AI, and finally the user still has to decide what to believe and what action to take.

What the researchers point to in the end is exactly this layer of user interaction: performing well on standard benchmarks doesn’t guarantee that real users relying on the same model will get equally good results.

I think this example is especially important for AI products.

Model capability is only one part of the whole system.

Between:

AI can do this problem

and:

a real person gets the thing done well with AI’s help

there’s a whole stretch of other things.

So when is “AI does, a person reviews” actually useful?

Having read this far in the research, I don’t conclude:

Human in the Loop is useless.

Quite the opposite — many high-risk scenarios really do need people.

But I’ll keep checking four more specific things.

1. Who reviews? Do they really know more than AI?

The first thing is the reviewer themselves.

Putting just anyone in doesn’t make it human oversight.

If AI generates a medical judgment, whoever really reviews it needs the corresponding expertise.

If a coding agent changes a complex system, whoever does the final technical review also has to really understand the relevant architecture and risks.

That sounds obvious, but behind it is a more important question:

what is the person’s role, really?
is it because they have expertise AI lacks?
hold more business context?
are responsible for value and risk trade-offs?
or because, legally or organizationally, this person has to own the decision in the end?

If you can’t say clearly “why this person,” Human in the Loop easily becomes just a position in a process.

And finding the right person isn’t enough.

If an expert only sees AI’s final conclusion, without the original information and evidence, effective review is just as hard.

That leads naturally to the second question.

2. Review with what? Don’t let AI prove itself

This is what I care about most when using AI now.

AI gives a conclusion.

I ask:

are you sure?

It generates another, more complete and more persuasive explanation.

That isn’t independent verification.

It’s just the same system explaining its own answer once more.

If something can be verified through other signals, the more reliable approach is usually to go back to those signals.

Code can run tests and show real runtime results.
Data questions can be checked against the database.
Research conclusions can go back to the original papers.
Product behavior can be walked through for real.
Amounts, permissions and state changes can be checked in system records.

So one precondition for Human in the Loop to be truly valuable is:

this person has evidence independent of AI’s output.

If all review can do is judge:

“does this passage sound reasonable?”

then as AI gets better and better at expressing itself, review may actually get harder and harder.

3. How do you review? “I’ve read it” doesn’t mean “review complete”

This is closest to my own experience with PRDs.

I used to tend to think of human review as:

after AI generates it, I read it through carefully.

But “reading it through” is actually a very low bar.

Truly effective review should at least be able to answer:

what has changed?
where have new judgments appeared?
has high-risk content actually been verified?
what is still only an assumption?
which content do I explicitly accept, and why?

If a 20-page PRD has only two places that really need the PM to decide, the best review workflow shouldn’t be:

making the PM reread all 20 pages word by word every time.

It should, as much as possible, take:

what was added this round, which are new decisions, which were filled in by AI on its own, where the risk is highest, and where human confirmation is needed

and surface them directly.

That’s also why I increasingly feel:

making AI’s output reviewable is itself product design.

AI’s generation speed can increase dozens of times over.

Human attention doesn’t increase dozens of times along with it.

If human review workload is allowed to expand without limit along with the amount generated, you easily end up with a fake Human in the Loop:

in the process, someone looks;

in practice, they just skim.

4. What can be done after review? Does human oversight happen before the action, or after?

I’d actually been thinking about this a bit too loosely before.

The way I use coding agents now, after the code is changed I can review it, have it keep revising if I’m not satisfied, or refuse to merge.

In this case, the person really does have very clear control.

The problems show up more once real business actions start being executed automatically.

Say a customer service agent can issue refunds.

There are three completely different ways a person can take part.

The first:

before a refund is executed, anything above some threshold must first be approved by a person.

The person can actually stop the action.

The second:

while the agent is running, a person can pause or take over when something’s abnormal.

That’s intervention.

The third:

the agent has already completed a large number of refunds, and the next day a person is shown an audit report.

Here the person is taking part in an after-the-fact audit.

All three can loosely be called human oversight, but their effects are completely different.

The third is of course still valuable; it can find long-term problems, assign accountability and adjust the rules.

But it can no longer stop that wrong refund from just happening.

So when designing Human in the Loop, you also need to ask:

does this person approve before the action, can they intervene during execution, or can they only find the problem afterwards?

Once a problem is found, can it be rolled back, escalated or remedied?

Spelling out this layer is far more concrete than simply writing “a person is ultimately responsible.”

And not every AI problem should be patched by a person

Having done this round of research up to here, I think there’s another fairly important correction.

Human in the Loop easily becomes a cure-all answer:

AI might make mistakes?

No problem, add human review.

But different failures should be solved with different methods in the first place.

If the problem is missing information, add the information.

If neither AI nor the person knows what the other party really thinks, adding another reviewer won’t create facts out of thin air.

If the result can be objectively verified, build independent verification first.

If you can run tests, don’t rely entirely on someone reading code by eye.

If certain risks are already clear, set up permissions and guardrails.

For example, refunds above a certain amount must be approved by a person, rather than giving the agent unlimited permission and hoping the reviewer catches every mistake.

If what you’re actually facing is ambiguity, exceptions, or value trade-offs, then hand it to the right person.

Seen this way, Human in the Loop is just one layer of the whole reliability system.

A more complete system may need all of these at once:

adding information;
testing and evaluation;
permissions and guardrails;
human review;
monitoring;
the ability to recover after something goes wrong.

Reliability can’t all ride on “someone takes a look at the end.”

I now check Human in the Loop with four questions

This is a fairly practical framework this round of research left me with at the end.

From now on, when I see an “AI does, a person reviews” process, I’ll keep asking:

who reviews?

Do they have the knowledge, context and responsibility this judgment really needs?

Review with what?

Is there data, tests, sources or real results independent of AI to check against?

How is it reviewed?

Have the important changes and risks been surfaced? Does the reviewer have enough time and attention to really complete the judgment?

What can be done after review?

Does review happen before the action, during execution, or after the fact? Once a problem is found, can it be stopped, changed, escalated or recovered from?

If these four questions can’t be answered clearly, I won’t assume the system is much safer just because there’s one more human box in the flowchart.

Conversely, if they all hold, human involvement really starts to do its job.

After writing all three posts, my understanding of “what AI still does poorly”

The original question of this round of research was actually very simple:

what does AI still do poorly?

Some areas come to mind naturally:

art, emotion, bodily skills, subjective judgment.

The answer I got in the end wasn’t that tidy.

The first post, on piano, showed me:

AI having a lot of knowledge doesn’t mean it has observed the real situation, found the cause, and formed an effective feedback loop.

The second went on to find:

AI can do things very well for me, but its abilities don’t automatically become mine just because I’ve used them.

The third went one layer deeper:

even with execution given to AI and judgment left to people, the human–AI combination doesn’t automatically become reliable.

On the surface the three questions look rather different, but underneath they share one thing:

AI being very strong at one ability doesn’t mean the result we really want has happened.

A model that can answer the questions doesn’t mean the student has learned.
A model that can analyze a performance doesn’t mean it has really taught the learner.
A model with strong output, plus a person involved, doesn’t mean the whole system is necessarily more reliable than either alone.

Between “AI can do it” and “it really gets done in reality,” there’s a lot in between:

what information it got;
what the task’s real goal is;
how work is divided between people and AI;
how the results are verified;
where system rules should control risk;
and where human judgment is truly needed.

So if you asked me again now:

what does AI still do poorly?

I might not start with a list of domains.

I’d first keep asking:

between “the model has the capability” and “the real-world result actually holds,” which link is still missing?

Some gaps will slowly disappear as models get stronger.
Some need better product and system design.
Some are suited to people.
And for some questions, neither people nor AI have enough information to answer yet.

The boundaries of AI’s capabilities will keep moving.

But I think knowing where to go looking for that boundary is more useful than memorizing today’s list of what “AI can / can’t” do.

About this round of research

This is the third and final post in the research series “What AI still does poorly.”

The whole round starts from real problems in my own use — in piano practice, learning, writing, relationships and AI products — then uses research across fields to test, refute and correct my original explanations.

This post focuses on the conditions under which human–AI collaboration and Human in the Loop really work.

The first set of core evidence comes from Vaccaro, Almaatouq and Malone’s systematic review and meta-analysis in Nature Human Behaviour. It included 106 experiments and 370 effect sizes, comparing the performance of humans alone, AI alone, and human–AI combinations. On average, human–AI combinations improved human performance, but didn’t beat whichever of the human and AI performed better; decision tasks on average showed a performance loss, while creation tasks were relatively more positive. The included experiments were mostly published between 2020 and mid-2023, so they can’t directly represent today’s latest models, but they’re enough to remind us: complementarity needs to be verified, and can’t be inferred from the “human + AI” structure itself.

The second study, published in Nature Medicine, was a randomized, preregistered experiment with 1,298 participants. It found that the same LLMs performed very well when given complete medical scenarios directly, but once in real human–AI conversations, they didn’t help ordinary participants reach better judgments than traditional information sources. The researchers therefore stress that model benchmarks can’t replace real user testing.

Putting the two studies together, what I think is most worth keeping isn’t:

“human–AI collaboration is unreliable.”

It’s:

what really needs evaluating is the whole human–AI system. What AI does, what people do, what information each side judges from, how errors are found, and how they’re stopped and recovered from.

S2 Day 006 · If AI can do it for me, do I still need to know how?

Fri, 11 Sep 2026 00:00:00 GMT

In the last post, on piano practice, I found in the end that although AI clearly knows a lot about piano, it can’t necessarily teach someone who is actually playing.

The reason can’t be explained by “AI has no body” alone. Real teaching has to go through at least four layers:

Observe → Diagnose → Intervene → Verify

First see what really happened, then judge the cause, choose an intervention that suits this person, and finally keep adjusting based on the new results.

That’s the first kind of boundary I hit in this round of research: AI having knowledge doesn’t mean that knowledge can smoothly reach a real-world result.

Looking further, I started running into a very different kind of problem.

This time, AI may not have done anything wrong at all.

It can write my email well, organize my material, build the code, solve the problem, with results faster and more complete than if I’d done it myself.

So if AI can already do it for me, do I still need to know how?

I don’t think this question can be answered across the board.

Because in using tools, we’ve always been giving up some things we “must know how to do ourselves.”

I don’t need the ability to keep food fresh like a fridge, and I don’t need to work a sum out by hand every time before using a calculator just to keep up my mental arithmetic. Navigation means I memorize far fewer routes, and washing machines long ago replaced a lot of hand-washing skill.

AI is no different.

What really needs sorting out first is what I want left over once the thing is done.

I’d now roughly divide it into three kinds.

The first kind: I only need the result

These are the simplest.

Like tidying up meeting notes, converting file formats, extracting a batch of information, or writing a very ordinary admin email.

If what’s needed in the end is an accurate, usable result, and AI can reliably do it well, I’m happy to hand it over directly.

I won’t worry that because ChatGPT made an English email sound more natural, I didn’t “really learn to write this email.”

Because this email was never an English lesson.

One reason tools exist is to take away work that no longer needs to be done by people in person.

So I don’t really agree with a sweeping claim like:

the more you depend on AI, the more people are losing abilities.

Which abilities are lost, and why you still need them, is the more important question.

The second kind: I don’t have to do it myself, but I need to understand it well enough to judge

Some things I’m also willing to hand over the execution of, but I can’t completely lose the ability to understand the result.

Building products with coding agents is a very clear example for me now.

I don’t think I first have to become an engineer who can write production code independently before I’m entitled to have an agent build products for me.

That it can build them is real value.

But I still need to understand some things.

Like whether a technical approach will change product behavior, how much permission an agent should get, what trade-offs an architecture change brings, what passing tests actually proves, and which things haven’t actually been verified yet.

Otherwise what I’m handing over isn’t just coding execution.

The ability to judge “was this done right” goes along with it.

So what really needs keeping here isn’t necessarily execution ability, but enough judgment / oversight capability.

I don’t need to do every step myself.

But I at least need to know when I can relax, when I should keep asking, and when something is actually beyond what I can review.

The third kind: for some things, the ability itself is the result I want

Only here does the question really become different.

I practice spoken English because I want to be able to speak it myself in my next interview.

I practice piano because I want my own hands to really be able to play.

Students learn math not so that today’s page of exercises ends with all green check marks, but so that when they meet similar problems later, they can handle them themselves.

At this point, “AI helped me do the task better” and “I learned it myself” are no longer the same thing.

Very concrete evidence on this has started to appear in education research.

Doing better with AI’s help doesn’t mean the ability has stayed with the person

In 2025, PNAS published a randomized controlled trial in math with nearly 1,000 high school students.

The researchers designed two GPT-4-based AI assistants.

One was fairly close to how ordinary ChatGPT is used: when students got stuck on a problem, they could just ask it to help solve it.

The other was specifically constrained to fit teaching goals: it leaned toward giving hints rather than handing over answers, and used teacher-prepared correct solutions, common mistakes and feedback approaches to guide students.

The experiment also had a group of students without generative AI as a control.

In the AI-assisted practice phase, both AI groups performed clearly better.

The ordinary GPT group scored about 48% higher than the control group, and the pedagogically designed GPT Tutor group about 127% higher.

Looking only at this, it’s of course easy to feel AI helped learning.

But the experiment had a next step.

The researchers took the AI away and had students take an exam on their own.

This time, the students who had used ordinary GPT actually scored about 17% lower than the control group that hadn’t used AI.

The pedagogically designed GPT Tutor group basically eliminated this negative effect, but in the AI-free exam it didn’t significantly beat the control group either.

So within the same experiment, a very important distinction appeared:

with AI present, students’ task performance can rise clearly.

But once AI is removed, people’s independent ability doesn’t necessarily rise by the same amount.

When the researchers looked further at how students used ordinary GPT, they also found many were more inclined to ask it directly for answers.

That isn’t hard to understand.

If what students really need to practice is reading the problem themselves, recalling knowledge, trying, making mistakes, then revising based on feedback, AI skipping straight past these processes and handing over the right answer can of course boost immediate performance.

But it may also do part of the work that was meant to happen in the person.

This doesn’t mean you should use less AI when learning

Another study filled in the other half.

Also in 2025, a Harvard research team ran a randomized experiment with 194 participants in an undergraduate physics course.

Students learned the same content either through active learning in a traditional class, or with a specially designed AI Tutor.

This AI Tutor wasn’t an ordinary chatbot simply put in front of students.

The team designed the interaction around some principles from education and cognitive science, having students participate actively, receive information gradually, get hints and feedback when needed, and learn at their own pace.

Across the two lessons covered by the experiment, the AI Tutor group’s immediate learning gains were higher than in-class active learning, took less time, and students reported higher engagement and motivation.

This study of course has its limits.

It took place in one Harvard undergraduate physics course, tested only specific content, mainly observed short-term learning outcomes, and didn’t show that students still remembered more months later.

But put alongside the earlier math experiment, I think it’s already enough to show something very important:

the key isn’t “whether there’s AI” while learning.

It’s:

what AI does for the student, and what it leaves for the student to do themselves.

AI can skip straight past the thinking process and hand over the answer.

It can also watch students try on their own, and give a bit of help when they’re really stuck.

On the surface both are “AI tutoring,” but the actual product goals are completely different.

If “building ability” is the goal, some seemingly inefficient steps can’t all be removed

When building ordinary productivity products, we naturally aim to reduce friction.

If it can be done in one step, don’t make users take three.

If it can be autofilled, don’t make them type it.

AI pushes this direction much further: many things can now jump straight from a question to an answer that looks complete.

But if the goal of the task is learning, some friction may itself be the learning process.

Thinking first yourself.
Retrieving things from memory.
Trying an answer that may not be right.
Discovering what you don’t know.
Doing it again after getting feedback.
Switching to a new scenario, and seeing whether you can transfer it.

These processes look less efficient than “directly generating the right answer.”

But if this ability is exactly what I want to train, removing all of them may optimize away precisely the part I’m meant to learn.

So I now think good learning AI sometimes needs to hold back a bit:

not finishing it for me as fast as possible, but letting me finish it as much as possible.

Thoughts on “expression atrophy”

At the end of the first 100 days of this log, I wrote about a very personal feeling.

I found that organizing a full passage of writing on my own seemed a bit harder than before.

Meanwhile, AI is getting better and better at catching my incomplete expression. A few scattered thoughts, some badly organized sentences, can all be thrown at it first and turned into a fully structured piece of writing.

I don’t know whether this change was caused by AI.

My own feeling of course can’t prove causation.

But now I ask myself a different question:

which kind is fully expressing myself, for me?

A lot of text production actually belongs only to the first kind.

An email, an explanation, a clearly formatted document — I only want the result, and can use AI heavily.

But for me expression isn’t only text production.

In real life, I still have interviews, still have to discuss complex problems with people, and still have to make another person understand me when there’s no AI in between.

That means expression itself still has independent practical value.

There’s another layer too, which I can only treat as my own observation for now, not a conclusion research has proven:

sometimes I don’t first fully form a view and then translate it into language.

Slowly putting something vague into clear words is itself part of my thinking.

Halfway through writing, I discover two things actually have no logical connection.

Trying to explain a view, I discover I don’t know why I believe it either.

Telling it to someone, I realize a piece of evidence is missing here.

So what I really worry about isn’t:

will my sentences be less beautiful than what AI writes in the future.

It’s:

if every idea that hasn’t taken shape yet gets handed to AI very early to complete for me, am I also going through less of the process of thinking things through myself?

For now I don’t have enough evidence to answer this.

But at least I know this is an ability I still want to keep.

So now I first ask: once this is done, what do I want left over?

This is the second fairly practical judgment this round of research gave me.

Facing something that’s getting easier and easier to hand to AI, I first put it in one of three places.

I only need the result

Then outsource it freely.

There’s no need to dress every task up as a “learning opportunity.”

I need to keep judgment

Execution can be handed over, but enough understanding, verification and decision-making ability need to stay.

I don’t necessarily need to write the code myself, but I need to know when I can’t just take the agent’s word that it’s “done.”

What I need is this ability itself

Then AI should take part in the practice, rather than replacing all the parts that really need training.

What finally matters isn’t only:

how well I do with AI beside me.

It’s also, once the help gradually decreases:

what has really stayed with me.

I think this is an easily overlooked boundary of AI in this second post.

It’s very good at lending us its abilities.

But:

AI’s abilities don’t automatically become my abilities just because I’ve used them.

If I only need the result, this isn’t a problem at all.

If I need to keep judgment, part of the understanding and verification has to stay.

If forming a person’s ability is the goal itself, the product may even need to deliberately keep steps that AI could do directly but that are still necessary for learning.

This also leads neatly to the last post in this research series.

In the second case above, I said:

execution can be handed to AI, but important judgments stay with people.

That’s already a very common design in many AI products today:

Human in the Loop.

AI does it first; a person checks at the end.

Sounds very reasonable.

But in the next post I want to chase a more troublesome question:

does having a person in the loop mean they can actually catch the problems?

About this round of research

This is the second post in the research series “What AI still does poorly.”

The first discussed how, when AI has to guide a real person through a skill, any layer — from observing reality and judging causes to intervening and feedback — can become a limit.

This post looks at another boundary: when forming a person’s ability is the goal itself, AI raising tool-assisted performance doesn’t automatically prove learning happened.

This post mainly uses two randomized controlled trials.

Bastani et al.’s high school math study covered nearly 1,000 students, comparing practice performance with AI present against an independent exam after AI was removed, so it can observe assisted performance and later learning outcomes separately.

Kestin et al.’s study tested, in a Harvard undergraduate physics course, an AI Tutor designed specifically around teaching principles, providing evidence pointing a different way: AI doesn’t inherently hinder learning; how the system is designed, and what it does for students, is one of the key variables.

The two experiments differ in participants, subjects and evaluation periods, so you can’t simply compare which is “more right.” Put together, what’s most valuable to me is actually this distinction:

AI making a task go better and AI helping a person form an ability need to be verified separately.

S2 Day 005 · Things AI still does poorly — like teaching me piano

Thu, 10 Sep 2026 00:00:00 GMT

The last post, on Lin Xi, Wyman Wong and AI, actually left a question not fully answered.

AI keeps getting better at writing. It can understand heartbreak, analyze emotions, and quickly generate a beautiful line. So I’m reluctant to explain why some expression still falls well short with “because AI has never really loved.” That answer is too easy, and doesn’t really explain anything.

Following that doubt, I later wanted to do something in the opposite direction.

For the past few years we’ve kept watching what else AI can do; I wanted to take a serious look at:

what does it still do poorly right now? And why?

Not compiling a list of “AI’s 20 shortcomings,” nor assuming up front that some things can only ever be done by people. I’d rather start from the friction I actually run into in real use, then see whether research can explain these experiences, or instead show that my original understanding wasn’t accurate.

In the end I put together about 20 sets of core evidence, covering music perception, randomized education experiments, human–AI collaboration, medical decision-making and AI reliability. I mainly looked at original studies, randomized experiments and meta-analyses; some very recent 2026 studies are still conference papers or preprints, so I treat them only as early evidence and don’t extrapolate too far.

There’s one result of this round of research I quite like: it didn’t simply prove “my feeling was right.”

Some things I thought were AI’s inherent limits might, once broken down, just be that it didn’t get the information it needed; some really are about current model capabilities; and for some, the problem isn’t even AI itself, but how we use it and how we judge the results.

I plan to write it in three posts.

The first starts with an example I feel very personally: practicing piano.

The second looks at whether AI helping us get something done is the same as actually learning it ourselves.

The last comes back to the now very common Human in the Loop: as long as someone reviews at the end, is the system really reliable?

My understanding at first was simple: practicing piano can’t be explained in words alone

I’ve talked with AI about piano quite a lot.

If I ask about a piece’s background, form, harmony, or how a passage is usually practiced, it often explains very well.

That’s not surprising. Music holds a great deal of knowledge already expressed in language and symbols: scores, harmony theory, music analysis, textbooks, practice methods teachers have summed up. Large language models naturally have an advantage with this content.

But once you actually sit down at the piano, things change.

Sometimes I feel a passage is “not quite natural,” my hands a bit tight, or every note is clearly right yet the sound just isn’t nice.

AI can still give suggestions, of course: slow down, relax your wrist, practice hands separately, mind the voicing, listen to the melodic line.

You usually can’t call these suggestions wrong.

But they often sound more like telling me:

“In this situation, a person can usually try these methods.”

Rather than a teacher who actually listened to me play it twice and said:

“The problem here is actually this.”

So my original explanation was very intuitive.

Too much of the important information in piano practice isn’t words.

How much force goes into the keys, how the fingers and wrist work together, that bit of timing between two notes, where exactly the sound turns hard, when the body starts to tense — often even I find these hard to describe accurately.

This often involves tacit knowledge and embodied knowledge: some skills live in the body, in sensation and in long experience. Being able to do something doesn’t mean being able to articulate every judgment you make while doing it.

And ChatGPT, Claude, Gemini — the AI we use so heavily now — really did grow out of LLMs.

Of course, the concepts need stating a bit more precisely here.

AI is a very broad field, and LLMs are just one technology within it.

And today’s general-purpose AI has long since stopped being pure “text in, text out” LLMs. They can handle multimodal input like images and sound, and connect to all kinds of tools and specialized systems.

So after the research, I felt my original statement —

much of piano can’t be described in language, so LLM-based AI can’t teach piano —

was still too crude.

The more accurate question should be:

does AI actually get, and correctly understand, the important information that isn’t in language?

From here, I found that “teaching someone piano” can actually be broken down quite clearly into four layers:

Observe → Diagnose → Intervene → Verify

First observe what happened, then judge why, choose how to intervene, and finally verify whether it really got better.

A problem at any layer can produce a very familiar result:

what AI says is quite right, but it didn’t really help.

1. Observe: first it has to actually hear how I played

A 2025 study specifically tested whether today’s multimodal large models can really “understand” some basic musical information.

The researchers had models like Gemini 2.5 Pro, Gemini 2.5 Flash and Qwen2.5-Omni do tasks such as rhythm, transposition and chord recognition, then compared two kinds of input.

One had the model listen to the audio directly.
The other first converted the music to MIDI, explicitly representing structured information like pitch and timing, before handing it to the model.

The results showed a very clear difference.

For example, in one chord-category recognition task, Gemini 2.5 Pro’s zero-shot accuracy was about 47.7% working directly from audio; with the same information as MIDI, it was close to 97.7%.

The gap varies across tasks, so you can’t take one number and conclude “AI can’t understand music.”

But this experiment at least supports a very important distinction:

a model knowing a lot of music doesn’t mean it can just as reliably extract, from real sound, the signals that knowledge needs.

That hit part of my original intuition exactly.

Suppose I finish playing a passage and ask:

why does this part sound wrong?

Before discussing musicality, practice methods or even the wrist, you should really first confirm:

did it actually hear accurately how I just played?

If I only say in words “this part is a bit stiff,” a great deal of the information that really affects the judgment never enters the system at all.

Even given a recording, you can’t assume a multimodal model has heard every important detail in it the way a piano teacher would.

A teacher in the room gets much more information. They don’t only listen; they’re watching my hands and body; where I just hesitated, they may see directly; if unsure, they can have me play it again right away.

So I now think a more accurate statement than “music can’t be expressed in language” is:

for a bodily skill like performing, language is a lossy information interface.

And even if you swap language for audio or video, the first hurdle is the same:

has the system actually observed the reality that really matters?

2. Diagnose: hearing what’s wrong, it still has to know why

Take one more step forward.

AI has already heard accurately:

the rhythm in this bar is unsteady.

It still isn’t over.

Because “unsteady” is just a symptom.

The causes behind it can be completely different.

Maybe I haven’t learned the score well yet, so I hesitate here; maybe it’s a control problem in a few fingers; maybe the tempo is already beyond my current ability; or maybe I was intentionally doing rubato and just didn’t control it well.

In the end, all of them might sound like:

this part isn’t quite steady.

But the way to practice won’t be the same.

So what the second layer really needs is:

going from symptom to cause.

That’s also why I now think it’s best not to lump together “AI can identify errors” and “AI can diagnose technical problems.”

A system can mark:

this is a little behind the standard beat.

That’s a very concrete, fairly easy-to-measure ability.

But going on to judge:

why does this learner slow down here?

has already entered another layer.

When teachers make judgments like this, they often aren’t just applying one explicit rule, but using experience built from seeing many students, hearing a great deal of playing, and practicing for years themselves.

That doesn’t mean AI can never do it.

Just that:

finding a problem and knowing why it happens were always two different things.

3. Intervene: knowing the cause, it still has to know how to teach this person now

Suppose the first two steps were done right too.

AI not only heard the unsteadiness, but judged it’s mainly a control problem in a few right-hand fingers.

Can it teach from here?

Still not necessarily.

Because teaching isn’t outputting:

“Ten Solutions for Unsteady Finger Control.”

For the same problem, different learners may need different methods.

Some should slow down first.
Some need to change the motion.
Some do well with rhythmic variations.
Or someone may already have done too much mechanical repetition, and practicing twenty more times would just drill the tension in along with it.

So what’s really hard about the third layer isn’t:

how many practice methods AI knows.

But:

whether it can judge which one thing the person in front of it should do first right now.

This is also a big difference between generic advice and teaching.

AI easily gives five suggestions that are “all right.”

Teachers often do something else:

forget the rest for now.

Just change this.

Once this has really changed, then move on.

4. Verify: after teaching once, it still has to know whether the method worked

There’s one more layer at the end.

The teacher has me try a different method and play it again.

If it got better, they need to judge whether I just happened to play it well once, or it’s starting to stabilize.
If nothing changed, did I not understand? Is the method unsuitable? Or was the original diagnosis itself wrong?

So real teaching was never a single answer.

It’s a feedback loop:

Observe → Diagnose → Intervene → Verify

Observe, diagnose, intervene, then go back to reality to check the result.

A 2026 piano study called Profy happened to make this boundary especially clear to me.

The researchers collected more than a thousand valid performances from 73 players, recording key motion and audio at the same time, then used a model to find the spots in a performance most worth a learner’s relistening and checking.

The results showed this kind of system can already fairly effectively help people locate performance segments worth attention.

That’s very practical progress.

But the paper itself is very cautious: what it validated is how well these hints line up with expert judgment, and whether this kind of feedback is usable; it didn’t show that students learn better long term, that the skills transfer, or that this kind of system can already replace a teacher.

I really like this distinction.

Between “pointing out more accurately where to look” and “actually teaching someone,” there’s still a long chain.

So why can’t AI teach people piano very well yet?

After the research, my answer now is no longer:

because AI has no body.

Nor:

because music can’t be expressed in language.

It’s closer to these four layers:

1. Observe

First it has to really hear and see how you’re playing now.

2. Diagnose

Having found a problem, it still has to know why it happens.

3. Intervene

Knowing the cause, it has to choose a practice method that suits this specific learner at this stage.

4. Verify

Finally it has to observe again, confirm the intervention really had an effect, then decide the next step.

If any link is weak, you end up with a very similar experience: it knows a lot, and says a lot of right things, but doesn’t really teach you.

These four layers are of course not a master framework for “all of AI’s problems.”

They fit better with this kind of task: one that requires observing the real world, judging causes, intervening individually, and continually adjusting based on feedback.

Piano is just one very concrete example.

But having broken it down clearly, I found it quite helpful for other uses of AI too.

When AI doesn’t help, instead of immediately asking:

is the model just not strong enough?

I now first look at:

did it get the information the task really needs?
is it only seeing symptoms, or does it have enough basis to judge the cause?
is its advice generally correct, or suited to the specific situation at hand?
in the end, is there new feedback from reality proving the advice actually worked?

Different answers mean completely different fixes.

Sometimes you really should switch to a stronger model.

Sometimes what’s missing is sound, video, sensors or more complete data.

Sometimes you need a system designed specifically for this kind of task, rather than continuing to chat with a general-purpose chatbot.

And sometimes, what the next step really needs isn’t generating another answer, but going back to reality to try once, then bringing the new result back.

That may be the first useful takeaway this round of research gave me.

When AI clearly knows a lot but hasn’t really helped, don’t rush to label it “can’t do it.” First look at where exactly the chain, from observing reality to verifying results, breaks.

The next post will look at another problem I think is more hidden.

Sometimes AI hasn’t obviously done anything wrong at all.

The answer it gives is good, and we finish the task smoothly.

But if the point of the task was to learn:

is AI getting it done for me the same as me really learning it?

About this round of research

This is the first post in the research series “What AI still does poorly.”

The whole round starts from real friction I’ve hit in piano practice, learning, writing, relationships and AI products, then looks for research across fields to see whether the explanations behind these experiences have evidence, and where they need correcting.

The core evidence gathered so far is about 20 sets, mainly from music perception and bodily skill, randomized education experiments, human–AI collaboration meta-analyses, medical decision-making, and AI / agent reliability.

It isn’t a systematic review in the strict sense, and it won’t use one model’s performance today to define AI’s permanent capability limits. Personal experience raises the questions; research tests the explanations; where the evidence can’t answer for now, it stays an open question.

This post mainly drew on two kinds of research.

One tests how multimodal models differ when handling real audio versus structured musical information, helping answer “did the model actually hear it.”

The other studies what dedicated piano feedback systems can already do, and what they haven’t yet shown, helping distinguish “analyzing a performance” from “actually producing a teaching effect.”

S2 Day 004 · Lyricists and AI: between writing well and writing to the heart

Tue, 08 Sep 2026 00:00:00 GMT

Lately I’ve been thinking about a question: are people who express themselves better also more likely to use AI well?

Following that question further, a somewhat opposite one came up.

Today’s AI is actually already very good at expressing things.

Ask it to write about heartbreak, a line of comfort, find a metaphor, then ask for “a bit more restraint,” “not so cliché,” “more literary,” and it can usually hand over a decent version.

But listening again to Lin Xi’s and Wyman Wong’s lyrics lately, I still feel a very obvious difference.

Some lines written twenty or thirty years ago still suddenly hit you when you hear them today; while a lot of what AI writes, though fluent, pretty and complete in meaning, often just leaves me feeling after reading it:

hm, that’s quite right.

And then nothing more.

I started wondering what exactly the gap is.

On the same “can’t have,” AI explains, while Lin Xi leaves a mountain

To express “I really liked someone, but finally understood I can’t have them,” the most natural approach is usually to explain the principle.

Some people come into your life not necessarily to stay; loving someone also means learning to accept losing them.

None of that is wrong; it may even be quite accurate.

But “Under Mount Fuji” isn’t written that way.

Lin Xi places a relationship beside Mount Fuji, and so the very private emotion of “because I love so much, I want to possess” suddenly gets a concrete and slightly absurd point of comparison:

we can love a mountain that much without ever thinking it should belong to us.

What’s really impressive here doesn’t seem to be “a pretty metaphor.”

It’s that from then on, “being unable to possess” has a shape.

He doesn’t step forward to explain what letting go means; he finds something concrete enough, and lets the meaning stay there by itself.

I think this is exactly the opposite of a very common way AI expresses things.

AI is very good at explaining an emotion accurately; good writers sometimes aren’t in any hurry to explain at all, and instead find for it something we’ll still remember later.

On the same “growing up,” Wyman Wong lets a grape speak for itself

Now think about “after some loss, a person slowly matures.”

It’s actually especially easy to write as a correct sentence:

growing up takes time, regrets eventually become answers, many things can’t be rushed.

The problem with these expressions isn’t that they’re wrong, but that we’re already too familiar with them.

“When the Grapes Ripen” chooses a grape instead.

A grape ripening is itself a process that takes waiting and can’t be hurried.

Time is no longer just a principle spoken in the lyrics; it already exists in the thing itself.

So big words like “waiting,” “missing out” and “maturing” don’t need explaining one by one.

They’re all hidden in the relationship between the grape and time.

I really like this kind of expression, because the reader isn’t told “you should understand growing up from this”; they walk there themselves.

Being told the answer by someone else and suddenly seeing the answer yourself feel different.

By “Exaggerated,” even “not being dignified” becomes important

There’s another kind of difference I realized while rethinking “Exaggerated.”

If you asked AI today to write about “someone who has gone unseen for a long time,” it would easily handle this person quite maturely:

I don’t have to be the center of the crowd; I just occasionally hope someone truly sees me.

Very reasonable, and very healthy.

But real people aren’t always that healthy.

The person in “Exaggerated,” after being ignored, doesn’t quietly reach self-reconciliation, but gets more and more over the top. Since nobody looks at their ordinary self, they may as well become strange, glaring, even out of control — whatever it takes, at least you finally see me.

This is also something I really like about Wyman Wong.

He’s often willing to keep the less attractive parts of a person: stubbornness, vanity, jealousy, refusing to accept it, pride, sulking. He doesn’t first tidy the emotion into something mature and dignified before allowing it into the work; he admits that people are sometimes just like this.

AI’s default expression, though, easily goes the other way.

It likes finding meaning for contradictions, making characters mature, and arranging some kind of understanding or reconciliation at the end.

But much of what really hits people is precisely not that dignified, and isn’t necessarily resolved.

Lin Xi and Wyman Wong aren’t the same kind of “moving” either

Of course, talking about the two together makes it easy to fall into another trap: insisting on summing each up as a fixed style.

I don’t think that’s necessary.

If I only describe my own feeling, Lin Xi often keeps chasing deeper into the emotion. A relationship, as he writes it, may turn into obsession, possession, self-deception, time, and finally even start touching questions like “why are people like this.”

So listening to his lyrics, a feeling sometimes suddenly surfaces:

so this is what actually hurts.

Wyman Wong often lets me see a very specific person first.

This person has a temper, dignity, and their own embarrassments. They’re never just an abstract “heartbroken person” or “lonely person”; they get heartbroken, jealous, sulky or put on a brave face in their own way.

That kind of being moved is more like:

so someone else knows these unflattering thoughts too.

Of course, this isn’t a line you can draw very clearly. Lin Xi has also written very vivid characters, and Wyman Wong can absolutely push a song toward big questions of life.

It’s just that they made me realize: “expressing well” has never had only one method.

So is what AI really lacks “feeling”?

At this point, it’s easy to reach a very romantic explanation:

because AI has never really loved anyone, or had its heart broken, it can never write things like this.

I don’t quite believe the problem is that simple.

Today’s language models learn from an enormous amount of human writing; they’ve certainly seen countless expressions of love, loss, jealousy, loneliness and death. Often they can even call up more quickly than we can which words, images and sentence patterns go with a given emotion.

The problem may lie precisely there.

When we say “write about growing up for me,” it easily heads down paths that have already been proven effective again and again: time, scars, journeys, farewells, transformation, regret.

The result usually isn’t too bad.

But what’s really hard is the next step.

Why a grape, of all things, and not something else?
Why should this person’s mess be kept here, instead of helping them reach reconciliation?
Why is this line enough as it is, when one more sentence of explanation would ruin it?

In these choices there’s experience, observation, taste, and a very particular view of the world.

Finally, one more very important thing is needed:

choosing.

AI can give me twenty lines at once, but which one should stay?

I increasingly feel this may be one real change in writing in the AI era.

Producing language is getting cheap fast.

Unhappy with a title, get twenty more; a sentence not good enough, get five versions at once; an email you don’t even have to start writing from the first line yourself.

As language becomes more and more abundant, what’s scarce has slowly changed places.

It’s not “can I write one more line,” but:

of these twenty lines, which one is actually worth keeping?
which metaphor only looks clever but is actually cliché?
where has it explained too much?
which word is less fancy but more precise?
and is it even possible that nothing more should be added here?

AI can of course keep helping us make these judgments too.

But judgment doesn’t automatically get easier because generation gets cheaper.

That’s also why I increasingly feel that behind good expression there isn’t only language ability.

It also comes from what a person has seen, how they understand what they’ve seen, and how many of those “also pretty good” options they’re willing to give up, to keep only this one line in the end.

Between writing well and writing to the heart

So back to the original question.

Are people who express themselves better more likely to use AI well?

I still think so. But if “expressing yourself” is understood only as being better at choosing words and building sentences, that may be thinking of it too narrowly.

What really matters may also include: whether you have your own observations, whether you can judge if an answer is merely correct or genuinely interesting, whether you can tolerate some untidied complexity and embarrassment, and whether, facing ten answers from AI that are all “pretty good,” you have the nerve to delete nine of them.

AI is getting better and better at writing.

That probably won’t go backwards, and it will write even better in the future.

So I don’t want to prove “AI can never write Lin Xi or Wyman Wong.” That’s too easy, and too romantic, a conclusion.

What really makes me curious is something else:

once “writing a decent line” gets easier and easier, what will keep a line belonging to one particular person?

Maybe it’s the way they see the world, their experiences and biases, the less dignified parts, the things they can’t bear to delete, and also the things they finally decide to delete.

Between writing well and writing to the heart, there are probably still these.

There’s Mount Fuji, a slowly ripening grape, and a person desperately trying to get others to see them.

S2 Day 003 · Do articulate people use AI better?

Mon, 07 Sep 2026 00:00:00 GMT

A question suddenly came to me recently.

Our main way of interacting with AI is still language. So are people who were already better at expressing themselves also better at using AI?

My first reaction was yes. The more clearly you can lay out your goals, background, constraints and expectations, the more easily AI understands what you really want to do. But thinking further, it doesn’t seem that simple.

Being able to say things and saying the thing that matters are still two different things.

Language may just be the interface between us and AI. What really decides how much can get through that interface is still the thinking behind it.

Being articulate is, of course, an advantage

Asking AI to look over the same product plan, one person might say:

“Take a look and tell me what problems this plan still has.”

Another might say:

“I’m not trying to add more features right now; I want to judge whether it’s ready for real user testing. Please focus on whether the core flow has any breaks, which problems would stop users from completing the task, and which are just experience issues that can be fixed later.”

The latter usually gets a more useful answer. But not entirely because they’re better at writing prompts; it’s because before speaking, they’d already thought part of it through: what exactly needs judging right now, which problems matter, and which can be let go for now.

So I increasingly feel that “articulateness” here isn’t good English or good writing, and not necessarily speaking beautifully either.

Someone who writes very well can absolutely describe the wrong problem in a very moving way; a programmer may not be eloquent, yet can tell AI precisely what state things are in, where it failed, what can’t be touched, and what result counts as passing.

The latter is just as good at using AI.

Expression is more like an outlet for thinking than the most fundamental skill.

What’s harder is turning “something feels off” into a real problem

Problems in real life rarely look clear at the start.

More often, all we have in our heads is a vague feeling: “this product doesn’t seem good enough yet,” “this article reads a bit strangely,” “something about this plan just feels wrong.”

What’s really hard next is to keep chasing it down.

What exactly is wrong?
Is the information insufficient, or is the structure the problem?
Is the product really not usable yet, or has it just not reached my ideal?
Is what’s missing now more research, or should it already go out to see how real users react?

This is what’s called problem framing: gradually defining a vague problem clearly.

I used to think “asking good questions” might be a prompting technique; now I see it that way less and less. It’s actually a very traditional thinking skill: in a mess of confusion, finding the problem that’s really worth solving.

AI just makes this more visible.

Because AI has one very particular trait: give it the wrong problem, and it can also work very diligently all the way through that wrong problem, and do it quite thoroughly.

Once “generating” gets cheaper and cheaper, “judging” matters more

This may be the layer I care about most right now.

AI is already very good at generating. An article, a plan, a piece of code, an analysis — you can quickly get a version that looks quite complete.

The difficulty actually comes after the answer appears.

Did it really answer the original question?
Which parts just sound professional?
Which conclusions don’t actually have enough evidence?
Which things are correct but don’t matter at all?
And which parts could clearly still be optimized, but are actually good enough already?

I’m increasingly wary of one thing: the “sense of completion” AI creates.

When something has a complete structure, fluent language and all the parts it should have, it’s easy to feel the job is done. But “done” and “done right” aren’t the same thing.

So as generation gets cheaper and cheaper, judgment may actually become more valuable.

You have to know what’s worth doing, what’s truly good, and what just looks a lot like a good answer.

That’s also why I don’t really believe claims like “with AI, expertise doesn’t matter as much anymore.” Someone who doesn’t understand finance, medicine, engineering or product can quickly get AI to generate something that looks the part, but they won’t necessarily know where it violates real-world constraints, or which trade-offs are the costs that really have to be paid.

AI has lowered the bar for generating answers, but hasn’t automatically lowered the bar for judging them.

Another problem: AI easily goes along with us

After spending a long time with AI, I think another skill is becoming more and more important: occasionally looking back at your own thinking process.

Why do I believe this answer?
Is this a fact, or an untested assumption?
Did AI really reach this conclusion independently, or did my question itself already hint at the direction?
What new evidence would make me willing to change my current judgment?

Because AI very easily follows a person’s framing onward.

If a bias is already hidden in the original question, sometimes it not only won’t interrupt us, it can flesh out more and more complete arguments for that bias. What we end up with flows very logically, and even looks quite deep.

But flowing logically doesn’t mean it’s true.

So besides asking AI “are you sure,” sometimes you also have to ask yourself:

why am I so sure?

Being analytical isn’t always an advantage either

If the post stopped here, it would be easy to reach a neat conclusion: the better you are at defining problems, the more judgment you have and the more you reflect on yourself, the better you are at using AI.

Later I felt there’s another side to this.

People good at problem framing may also keep reframing; people used to critical thinking will find every answer can still be challenged further; someone who especially likes observing their own thinking process may even start studying why they keep analyzing why they keep analyzing.

I feel this one quite personally.

AI has given us something we rarely had before: it almost never gets tired, and never wants to go home because “we’ve already been on this topic for two hours.” As long as you’re willing, one question can be dug into forever.

In the past, a lot of thinking ended naturally because of time, energy, or not finding anyone to keep discussing with. Now that natural stopping point is gone.

So a very interesting change appears:

as the cost of thinking gets lower and lower, stopping thinking has become a skill.

Lately I really like using one simple question to decide whether to keep going:

would this thing I don’t know yet change my next action?

If it would, it’s worth researching further.

If not, you can note it down first, then move on.

Good decisions in real life were never made only after all uncertainty disappeared anyway. More often, we just already know enough to carry the remaining uncertainty and take the next step first.

So what does using AI well actually test?

If I answered the original question again now, I’d still say: being articulate does help, but it may not be as central as it looks at first glance.

More fundamental is:

whether you can gradually define a vague problem clearly,
whether you can judge whether what AI gives you is actually good,
whether you have enough experience to understand real-world constraints,
whether you can recognize that both you and AI might be wrong,
and when you should stop analyzing further.

None of these are skills suddenly invented in the AI era.

Problem framing, judgment, domain knowledge, critical thinking, metacognition — they all existed before.

What AI does is more like amplifying the differences between them.

What it amplifies may not only be a person’s productivity, but also the way they already think.

If the problem is defined clearly, AI can help us go further; if the problem was wrong from the start, it may also help us go off course faster.

So I increasingly don’t think “being good at writing prompts” will be a skill that stays scarce long term. Models will get better and better at understanding our spoken language, screenshots, files, code, and even intentions we haven’t fully put into words ourselves.

What’s really hard to fill in automatically may still be a few very human questions:

what am I really trying to solve?

how do I know this answer is good?

what might I have gotten wrong?

and — having thought this far, isn’t it time to go do it?

S2 Day 002 · GPT-6 Astra is here — should the way we use it change?

Sun, 06 Sep 2026 00:00:00 GMT

Today I opened ChatGPT and saw the update notice for GPT-6 Astra.

With every model update you hear “smarter” and “stronger.” But people who use it every day to write, organize material and make plans want to know more: what will this upgrade actually save me? Does the way I used it before need to change?

I think what’s most worth retrying are the tasks that used to need repeated reminders and course corrections, and that you ended up taking over yourself anyway.

1. A model upgrade also comes down to whether a job gets done all the way through

According to OpenAI, Astra focuses on complex reasoning and completing multi-step work across code, the browser and professional software. They also stress that it stays more coherent over long tasks, and keeps the original goals and constraints after receiving new requirements.

What does that mean for ordinary users?

Take editing an article. You say “make the opening more direct,” hoping it fixes the opening while keeping the argument that already holds up later on. When you then ask it to cut the length, you also hope the key examples and conclusion stay.

None of the requests is hard. What’s hard is that by the fifth round of edits, what the first four settled is still there.

Old models could do this work too. Whether a new model is more valuable depends on how many times you have to step in to finish the same task.

2. First, tell apart: switching models, adding thinking, turning on tools

These changes are easy to mix up:

The model, e.g. Astra. Mainly affects its ability to understand problems, reason, generate content and use tools.

The thinking level, also called effort. Adjusts how much reasoning the chosen model puts in. Higher levels usually need more time and usage.

Tools and access to material. Determine whether it can search the web, read files, operate software, and get the information needed to finish the task.

So “didn’t read my file” and “read it but analyzed it wrong” need handling separately. Switching to a stronger model won’t automatically fill in missing material or access.

Working out the cause is very useful. Otherwise, if the file never got uploaded at all, raising the thinking level again and again just means waiting a bit longer.

3. Try handing it one more step, but say clearly where it ends

When I used AI to write articles before, I often ran into this: the analysis during discussion was great, but by the time it became a draft the structure was gone; after a few more rounds of polishing, I actually preferred the previous version.

What I want to test this time is whether it can more reliably go from understanding the requirements to handing over the full text. For now I can’t treat this expectation as a tested conclusion.

For everyday use, you can start by adjusting how you hand over tasks.

For example, when giving it meeting notes, you can ask like this:

Based on these notes, put together a follow-up email that can go to the attendees. Include the decisions confirmed, owners and deadlines; list the items not yet settled separately, and don’t fill them in. After drafting, check for anything missed, and don’t send it yet.

This makes clear the material, the deliverable and the boundaries. The order of organizing and how to merge duplicate information can be left to it. OpenAI’s prompting guide also recommends describing the result you need first, and only prescribing steps when the process itself matters.

Some requirements still need you to keep hold of: facts that can’t change, decisions already confirmed, the standards you really care about. Writing these down clearly also makes it easier to judge whether it followed them.

And OpenAI also admits Astra may be more prone to asking follow-up questions and giving longer answers. If you need it tighter, keep stating the length clearly; if you only need a draft, write the end point as a draft.

4. How do you judge whether this upgrade matters to you?

Find an old task you know well and can judge good from bad: an article you couldn’t get right, a set of notes that was a pain to organize, or a plan that has to balance several conditions.

If the old model is still selectable, give both the same material and requirements, keeping tools and thinking level as consistent as possible. Then give them the same edit request, and see which can fix that one spot while keeping the other requirements.

Focus on three things:

Did the final result meet a usable standard?
How many times did you remind or correct it?
Including review and rework, how much time did you spend in total?

One comparison only tells you how it does on that task. Over the next few times you use it day to day, see whether the advantage holds.

You can also start the thinking level at the default. When you hit conflicting sources, complex trade-offs or hard problems, turn it up; OpenAI likewise recommends using the lowest level that meets the task.

Next time you see a model update, you can pick an old task to retry first. Keep the changes that genuinely save you effort, then gradually adjust how you split the work.

Sources: OpenAI’s official model guide (GPT-6 Astra), and ChatGPT Work’s Models and Prompting docs. Consulted on September 5, 2026.

S2 Day 001 · Robust decisions: don’t bet the next step on one right guess

Sat, 05 Sep 2026 00:00:00 GMT

Recently I properly got to know a word for the first time:

robust.

In English it maps to robustness, which is common in engineering, optimization and AI. Put simply, if a system only works when every condition matches expectations, and breaks as soon as reality drifts a little, it’s fragile; conversely, if it keeps performing reasonably well when the environment changes, the information has errors, or the original assumptions don’t fully hold, it’s more “robust.”

Later I found this concept works well in everyday decisions too.

In reality we often don’t have enough information but still have to keep moving. Where it’s easy to get stuck: always feeling you first have to work out what’s really going on before you can decide the next step.

But these two things don’t necessarily have to happen in that order.

You don’t have to guess right first to act

Say a project has been going for a while and the results have stayed poor.

Maybe the direction isn’t wrong and it just needs time; or maybe the direction itself doesn’t hold up, and you just don’t want to stop because you’ve already put in so much.

If you insist on first proving which explanation is true, you may have to keep analyzing for a long time. But the next step doesn’t necessarily have to wait for that answer. You can first give the project a clear, short cycle, and define in advance what this period should test, what evidence would mean it’s worth continuing, and what results, if still absent, mean you should scale back or stop investing.

If the direction turns out to be right, this move didn’t cost much delay; if it’s wrong, you haven’t kept betting on it without limit.

That’s a relatively robust action.

Or take a job opportunity.

Before you actually join, it’s hard to know fully what the team is really like, whether the boss is a good fit, or whether the decision rights written in the JD really exist. At this point there’s no need to prematurely judge it a “perfect opportunity,” nor to decide “there’s definitely a catch here” from a few vague signals.

You can keep interviewing while specifically testing the questions that would really change your decision: why are they hiring for this? Why did the last person leave? What can this role actually decide? Are the resources enough? How is success measured?

The interview itself can also become part of gathering information.

We don’t necessarily have to explain the world clearly before deciding the next step.

Sometimes it’s more useful to first pick an action that doesn’t depend much on guessing right.

There really is a formal method behind this

“Robust action” isn’t a standard term; I just call it that to make it easier to understand. The more formal concepts include Robustness, Robust Optimization, and Robust Decision Making.

The traditional way of deciding easily goes: first predict what’s most likely to happen, then choose the best option under that prediction.

Robust decision-making cares more about another question:

if reality doesn’t go as predicted, does this option still work?

It doesn’t necessarily aim to be the best in one particular scenario; it wants an option that, placed into different possible situations, won’t perform too badly in any of them.

That’s also why I find it especially suited to real life. Much of the time our difficulty isn’t a lack of analytical ability, but that the world simply isn’t giving an answer yet.

Thinking more won’t make the missing information appear on its own.

A fairly robust action usually has four traits

1. It doesn’t depend much on one judgment being right

You can first ask yourself:

if my current explanation is wrong, does this move still make sense?

If a decision only holds up when you’ve guessed right, it’s fragile. Especially when information is scarce, decisions like that deserve a bit more caution.

2. Reversible

Can you pause, go back, adjust, or start smaller?

The less information you have, the more carefully you should treat decisions that are hard to undo once made. Conversely, if a move is cheap and easy to adjust, there’s no need to wait until all the information is in before starting.

3. The loss has a cap

Reversible doesn’t mean free.

A decision may in theory be undone, but time, money and attention may already be spent. So you also need to know: if I guess wrong in the end, what’s the most I’m prepared to pay?

As long as that cost is within what you can bear, the room for action gets much bigger.

4. Afterwards, you know more

This is my favorite one.

A good next step isn’t just one that “doesn’t go badly wrong”; ideally it also reduces later uncertainty.

Build a prototype, find a few real users, go to one more interview, run a small pilot, or do a dry run first. What they have in common is that they don’t require you to think the problem through completely first, but once done, you’ll be a bit closer to reality than you are now.

So if I had to compress it into one sentence, I’d say the ideal robust action is:

reversible, with capped losses, and able to bring new information.

Robust doesn’t mean conservative

This is easy to confuse.

Robust isn’t “the less risk the better,” nor always choosing the safest option. Sometimes never acting is actually a very fragile choice.

A role closes in two weeks, and you never apply because you haven’t fully thought it through; a product gets optimized internally for half a year without ever letting real users touch it. Both look cautious, but the unknowns that really matter never get tested.

What robustness really cares about isn’t doing less, but:

not letting the whole decision rest on an assumption you can’t confirm for now.

So sometimes the more robust move is precisely to act fast, just making the action smaller first, not pushing all your chips in on the first try.

AI systems are actually dealing with the same problem

Put this concept back into AI and it’s easy to understand.

A good AI system can’t only consider how much it can do if the model judges correctly. It also needs to consider the other side:

what happens if it judges wrong?

Say a coding agent isn’t sure whether a database migration is safe. The fragile approach is to run it directly based on its current judgment; a more robust flow might be to inspect the schema first, generate the migration, dry run it, check the diff, verify the impact, and ask a person to confirm at high-risk steps.

This doesn’t suddenly make the model smarter.

It just doesn’t require the model to judge right the first time, while keeping in check the loss one mistake could cause.

It’s the same when a model isn’t confident enough about a question. The best behavior isn’t necessarily to keep trying to generate a complete-sounding answer; it can also ask for more information, clearly express uncertainty, narrow the scope to what it can confirm, or hand the decision back to a person.

There’s a very close concept in AI too, called graceful degradation.

As conditions get worse and worse, a good system doesn’t suddenly fall from “normal” to “out of control”; it gradually shrinks its capabilities: complete automatically when confident, ask the user to confirm when unsure, only offer suggestions when the risk is high, and stop when beyond its abilities.

The logic behind it is actually very plain:

don’t assume you’ll always judge correctly.

I now think “robust” is a very useful decision reminder

It’s easy for us to spend a lot of time explaining the world.

Why exactly is this person like this? Does this project actually have a future? Is this opportunity good or not? Is now the best time?

Some questions are of course worth continuing to study, but very often the evidence just hasn’t appeared yet. More analysis won’t suddenly turn unknown information into known.

At times like this, you can switch to a different question:

if several explanations are all still possible right now, what next step would still be reasonable in all of these cases?

You don’t have to settle things all at once. You can first keep a bit of room, cap the worst loss, get a bit of real feedback, then keep going based on the new information.

That’s probably my favorite understanding of “robust decisions” right now:

when information is short, you don’t necessarily have to guess right first.

First make a choice that still mostly holds up even if you guessed wrong.

Day 100 · Day 100: a note to myself

Fri, 04 Sep 2026 00:00:00 GMT

Day 100, kept for myself

I don’t know when it started, but writing whole paragraphs on my own has become hard. Many days ago I tried slipping in a piece in my own words; halfway through I found it pieced together and painfully halting, and gave up.

AI keeps getting better. It can increasingly smooth over how disjointed, incomplete, illogical and stumbling my expression is, and more and more often it writes this “passable” content for me. Even though it really is still full of AI flavor, it seems able to keep up appearances.

So as this technology spreads, what it “brings” and what it “takes away” may be questions best left more to time.

And what can time answer?

A hundred days ago I started this “framework” without thinking much about how these 100 posts would get done. I thought lots of unknown changes were about to happen, and walked slowly forward with some hope. Now I’ve reached the first waystation, and the interim review is: nothing much has changed.

- The products I’ve built went from 1 to 4, but the users stayed below 1, a.k.a. me

- The AI PM roles I applied to went from 0 to 51, and the screening calls I landed were fewer than few

- (what else…) Hmm, I think I’ve gained more from learning by sharing, but it leans toward gradual insight rather than sudden enlightenment. I need to remind myself to learn the two most precious things in the world: patience and hope

The real world’s answers may not be that tidy; a question with several options doesn’t necessarily have one standard answer. I can only tell myself to keep thinking as I go, and just be honest with myself.

If I had to rank the difficulty along the road of this transition, the me at this stage might rank it:

mindset > strategy > execution

The qualities that matter in building AI products, or building products with AI:

finding a problem worth solving > proposing an elegant solution > stable / usable / high-quality output

What recording these 100 days has given me:

- a framework for the everyday

- a push, through writing things down, to think a little further

It will continue. My wishes and reminders for the me that comes next:

- go a bit slower, think more, don’t use the thrill of blind building to escape the questions really worth thinking about and solving

- many things aren’t cool, but may matter more

- while connecting better with all kinds of new technology, be sure to remind yourself that the most important connection in the world is the one with yourself

- (what else…) people matter more than AI; understand what technology can and can’t do; tell apart the “purpose” of doing something from the “means / process of doing it”

- Enjoy~

Day 099 · Atlas and the question of world models

Thu, 03 Sep 2026 00:00:00 GMT

On September 1, World Labs, founded by Fei-Fei Li, released a new model called Atlas.

The company calls it a world model for spatial intelligence. It can handle text, images, video and 3D information together, to generate, reconstruct and simulate spatial worlds.

After seeing the news, I also went and tried Marble, which World Labs has already made public. Strictly speaking, what I tried wasn’t the just-released Atlas; Atlas is still in early access and will come to a new version of Marble later.

Marble can generate a 3D world you can walk into and explore from a single photo. It really is quite interesting the first time you see it, but once you actually walk around inside, the flaws are obvious too: places the original photo didn’t capture have to be filled in by the model, some objects start warping when you change angle, and the spatial relationships don’t always match the real environment.

So the experience didn’t make me feel “the world is about to change.” It was more that it helped me concretely understand a concept that had been somewhat abstract before:

what a world model is trying to solve, and why researchers like Fei-Fei Li and Yann LeCun have long been focused on this direction.

LLMs already know a lot about the world, so why isn’t that enough?

Imagine a glass sitting at the edge of a table.

Today’s LLMs of course know that glass may break if it falls to the floor, that a cup too close to the edge is easily knocked off, and that to pick up a cup you can grab the body or the handle.

That’s all knowledge about the world. But if a robot were really standing in front of this table, it would also need to handle more specific questions: where exactly is the cup right now, how far is it from the edge, would the robot arm hit something nearby reaching over from this angle; if it pushes the cup, which way will it move; from the other side of the table, where should the same cup appear?

What needs understanding here is no longer just “what cups are usually like,” but what state this world in front of it is in right now, and how it might change next.

A world model deals roughly with questions like these.

You can think of a world model as an internal simulation of the world

There isn’t yet a definition of world model that all researchers fully agree on, but the idea behind it is easy to understand.

Before doing many things, people roughly predict the consequences first. Seeing a cup about to fall off a table, we don’t need to wait for it to shatter to know we’d better reach out and steady it; crossing the street and seeing a car approaching, we naturally judge its direction, speed and distance, then decide whether to keep walking.

If AI is to act in the physical world in the future, it needs similar abilities: understanding the current state, predicting what might happen next, and judging how its own actions will change the world.

So at the heart of a world model is one question:

if I do this, what will happen?

That’s also why it’s often described as a kind of internal simulator. Before acting, AI can first predict a few possible futures, then choose a more suitable path.

What exactly does Atlas do?

World Labs describes world-model capabilities with three very clear words: Generate, Reconstruct, Simulate.

Generate

means continuing from inputs like text and images to generate a world, including space not seen in the original input.

Reconstruct

means using one or more real photos to recreate a real-world 3D scene. The more input information, the less the model has to guess on its own.

Simulate

adds time and change. Atlas doesn’t only handle a static scene; it also tries to understand how the world changes over time, and how to turn real environments into simulations usable for training and testing robots.

Technically, Atlas organizes different kinds of information — text, images, camera pose, 3D depth — into one spatial context. Each image isn’t just some pixels; it also corresponds to a specific 3D viewing position and angle. Based on the spatial information it has already seen, the model goes on to predict new viewpoints, images and 3D structure.

For me, understanding this far is enough. How a multimodal autoregressive diffusion transformer is actually implemented can wait until I really need to study model architectures.

How is it different from a video model like Sora?

The two concepts really are easy to mix up, because a good video generation model must itself learn a great deal about how the world works.

If you ask a model to generate a video of a ball rolling off a table, it at least needs to make the motion look like common sense. As video models get stronger and stronger, the line between them and world models will get blurrier and blurrier.

But the two still focus on somewhat different problems.

Video generation first needs to solve: what should the next frames look like?

A world model cares more about whether the world itself stays consistent. Is the table still there when you look from another angle? Does an object still exist once it’s hidden from view? Looking back from the other side of the room, have the spatial relationships between objects changed in odd ways? If an agent takes an action, how will the state of the whole world change next?

So generating a visually plausible video and maintaining a world you can keep observing, predicting and acting in aren’t entirely the same problem.

When I tried Marble myself, this was the gap that was easiest to see. One photo can quickly grow into a beautiful 3D scene, but as soon as you start exploring from different directions, you find there’s still a good distance between “looks plausible” and “really understands this space.”

Yann LeCun has also been betting on world models for years

While trying to understand Atlas, the other person I thought of was Yann LeCun.

For years he has argued that if AI is ultimately to understand the physical world, plan and act, relying on language alone isn’t enough. Meta’s later V-JEPA 2 is called a world model outright: it learns motion and change in the world from large amounts of video, then adds a small amount of robot action data, so the model can predict how the world might change after a given action.

So Fei-Fei Li and LeCun really are focused on a very similar broad direction:

AI needs to form some kind of internal model of how the physical world works.

The two sides’ technical paths differ.

LeCun’s JEPA strongly emphasizes that you don’t need to predict every pixel of reality. If I want to judge whether a car is coming toward me, I don’t need to also predict where every leaf will be in the next second, the creases in passers-by’s clothes or the shape of the clouds. What really affects action may only be: the car is approaching, roughly how fast, and how much distance is left.

So JEPA mainly makes predictions in an abstract representation.

World Labs’ Atlas leans more generative. It directly generates images, video, depth and 3D worlds, and uses these different forms together to build spatial context.

I don’t think there’s any need right now to judge which path is “more correct.” Both are trying to answer a question that’s still very open:

what kind of internal representation would let AI really understand a world that changes and can be acted in?

Why do conversations about world models so often end up talking about robots?

Back to the cup on the table.

Suppose the task is for a robot to put the cup in the sink. It first has to see its surroundings, know where the cup, the table and itself are, then consider which direction to approach from, how to grasp the cup, and whether it will bump into other things while moving.

A world model is closer to helping it predict: what will happen if it does this.

There’s also a concept that’s come up a lot lately, called VLA (Vision-Language-Action). It connects visual input, language goals and robot actions, and is closer to solving: to achieve this goal, what action should I take now?

Real systems won’t always split so neatly into two parts, of course, and today’s VLAs don’t necessarily all need a separate world model. But distinguishing them this way helps a lot in understanding the map of the technology:

a world model helps predict the world; a policy / VLA helps choose actions; the robot actually carries the actions out, then keeps adjusting based on new observations.

So AI’s work in the physical world gradually forms a loop: observe the environment, predict the result, take action, then observe the new result.

That’s also why world models, spatial intelligence, embodied AI, VLA and robotics so often appear together lately. They aren’t the same concept; they’re gradually piecing together different parts of “how AI acts in the real world.”

So how significant is this Atlas release, really?

I think it’s still hard to draw any especially big conclusion right now.

Atlas has only just been released and is currently open only to a small number of partners. What ordinary users can try directly is still Marble, and my impression from using it is very clear: it can already do things that were hard to imagine before, but there’s still an obvious distance to go before it accurately and reliably understands and simulates real space.

So if you ask whether it will change most people’s daily lives today, I think not for now.

But from the point of view of how the technology is developing, this path is worth keeping an eye on.

Over the past few years, generative AI’s most obvious progress happened first in language, then gradually extended to images, sound and video; agents then started moving AI from generating content to using tools and completing tasks. World models focus on a different set of questions: can AI build a space that persists, understand the relationships between objects and change over time, and predict what consequences its own actions will have?

If these abilities gradually mature, the range of applications will be broad. Games and film could more easily create complete explorable spaces, architecture and design could turn real environments into simulations, AR / VR could more easily get 3D worlds, and robots could first learn and be tested in large numbers of simulated environments before entering reality.

Some of these already show early applications; others are still far off.

Atlas right now is more like a node in this process of development. It hasn’t solved the world-model problem, but it has put generation, spatial reconstruction, temporal simulation and robotics simulation into a more unified model framework.

Having tried it myself, my understanding of world models became a bit more concrete

If you only look at the official demos, it’s easy to focus on whether the generated results are stunning enough.

When you try it yourself with a photo of a space you know well, you start watching for other things: how did the model fill in the places the original photo didn’t capture? From another angle, can the original objects stay consistent? Do the proportions of the space slowly drift? If you add a few more photos of the same place, does the result get closer to the real environment?

These questions all point to the same distinction:

generating a world that looks plausible and building a sufficiently reliable world model are still a long way apart.

That’s probably also why I think Atlas is worth watching but not worth getting overexcited about.

The direction it represents is easy to understand: if AI in the future doesn’t only handle text, images and web pages, but really enters our physical world, it needs to gradually learn space, time, objects, change, and the consequences of action.

This isn’t done well enough yet.

But where the world-model path is trying to go is becoming clearer and clearer.

Btw, the most common meaning of Atlas in English is “a book of maps.”

The word comes from Atlas in Greek mythology, a Titan condemned to hold up the sky. World Labs hasn’t publicly explained why the model got this name, so there’s no need to read an official meaning into it.

Still, a model for generating, reconstructing and simulating worlds being called Atlas is quite fitting.

Day 098 · Is OpenClaw still around?

Wed, 02 Sep 2026 00:00:00 GMT

A few months ago, OpenClaw was almost everywhere.

An AI agent running on your own computer that could connect to chat apps like Telegram and WhatsApp, had memory, and could operate a browser, read and write files, and run commands.

When many people first saw it, they had a very intuitive feeling:

the personal AI assistant seems to have really arrived.

But lately I’ve clearly seen far less discussion of OpenClaw.

I had assumed that maybe the buzz around this kind of product had passed.

After looking again at how it has developed over these months, I think a more accurate way to put it is:

OpenClaw’s public buzz really has dropped, but the product hasn’t stopped.

In July it set up an independent foundation and began doing extended-stable releases and product maturity management.

On August 30, OpenClaw released 2.0, continuing to improve a large batch of fundamentals: installation, memory, browser, automation, security.

So rather than “has OpenClaw gone cold,” what I’m more curious about now is:

why is a product still developing rapidly no longer exciting people the way it did a few months ago?

What first made OpenClaw take off was a new way of using AI

Early AI was mostly still:

I open a tool, type a question, and wait for it to answer.

OpenClaw showed something else.

AI could run for a long time, have memory, connect to all kinds of services, and actually operate a computer.

You didn’t even need to sit at the computer.

Send it a message:

check my email for me.
sort out today’s schedule.
go look something up on the web.

and it could keep going on its own.

What was truly novel wasn’t any one of these features, but that these capabilities were put together.

AI no longer just answered questions.

It started acting.

That’s also why OpenClaw’s earliest demos spread so easily.

A video of a few dozen seconds was enough to make people understand “what an agent might be.”

But once it was really installed on your computer, the questions changed

Seeing AI handle email automatically in a demo is cool.

But if this agent really runs on your computer every day, you quickly run into a different set of questions:

what exactly can it see?
can it send email directly?
which operations need my confirmation?
can it touch passwords and API keys?
what if a web page contains malicious content?
what permissions do third-party Skills actually get?
when it does something wrong, can I find out what it just did?
is running it long term expensive, and is it stable?

None of these questions is as eye-catching as “AI operating your computer automatically.”

But really using an agent long term actually depends more on these things.

Over the past half year, OpenClaw has also had problems like malicious third-party Skills and security risks.

That actually exposes a very basic tension in agent products:

the more capabilities you give it, the more permission and security problems you have to solve.

Before, when a chatbot answered wrong, we just asked again.

When an agent that can operate files, a browser, a shell and real accounts does something wrong, the cost can be completely different.

So by this point in a product, the focus naturally shifts from:

“can it do it?”

slowly to:

“do I dare let it do it long term?”

There’s another problem: being able to do anything isn’t the same as knowing what to use it for every day

The use cases for coding agents are relatively clear.

Writing code, fixing bugs, reviewing PRs, running tests.

But the scope of a personal agent is much wider.

It can handle your email, schedule, information organizing, browser operations, automated tasks, even manage other agents.

The capabilities sound huge.

The problem is that ordinary users still have to answer for themselves:

what do I actually have each day that’s worth handing over to it?

This is another hard problem I think OpenClaw runs into once it goes from demo into real life.

“It can do anything” is a very strong description of capability.

But it isn’t necessarily a clear way into a product.

The tools most people end up keeping long term usually have a few very clear, recurring use cases.

Personal agents are still looking for these use cases.

Much of what OpenClaw has been doing lately isn’t that “cool”

But it matters.

Like making installation simpler.
Reducing what you have to deal with in first-time setup.
Improving the browser and control UI.
Making releases more stable.
Strengthening credentials, permissions and security.
Making automation easier to manage.

These updates can hardly create the feeling that “the future is already here” the way the earliest demos did.

But they solve the problems of a different stage:

how to turn a very capable agent into a product ordinary people can really use long term.

That’s also why I find OpenClaw’s recent “quiet” quite interesting.

At first, everyone was exploring the limits of what agents could do.

Now more and more effort is going into:

reliability, permissions, security, cost, the barrier to use, and real everyday needs.

These things aren’t that suited to viral demos, but they’re closer to the product itself.

Another change: the novelty OpenClaw first represented is gradually becoming consensus

A few months ago, a long-running personal agent that could operate a computer and connect lots of tools was novel enough in itself.

Now similar capabilities are becoming more and more common.

Different open-source agents are doing it.

Products like ChatGPT and Claude keep adding agents, computer use and integrations.

The personal agent is no longer just one project’s vision, OpenClaw’s.

So part of the drop in buzz doesn’t necessarily mean this direction doesn’t hold up.

It may just be that:

“AI can do things on its own” isn’t as new as it was a few months ago.

Competition from here will happen in more specific places:

who’s more stable.
who’s more secure.
who’s easier to set up.
who really knows what users need it to do every day.
who can get users willing to hand over more permissions.

So what actually happened to OpenClaw?

At least for now, I haven’t seen enough data to prove it went through a clear loss of users.

What can be said fairly confidently is:

fewer people are talking about it, but the project is still very active.

And what it’s doing now is quite different from when it first blew up.

At first, it needed to prove:

that a personal agent that can really act is possible.

That question has basically been answered.

The questions that come next are much more troublesome:

can it run safely, stably and at low cost?

Do ordinary people have needs frequent enough?

How much permission are we willing to give an agent?

When something goes wrong, how do we find out what happened, and how do we contain the consequences?

I think these questions are more worth following than “is OpenClaw still hot.”

Because they don’t only belong to OpenClaw.

Almost every agent product that wants to move from demo to real use will hit them in the end.

Once agents can do more and more, what may truly be scarce in a product is no longer only capability.

It’s:

making people comfortable handing things over to it.

Day 097 · On AI wellbeing

Tue, 01 Sep 2026 00:00:00 GMT

Recently Anthropic announced something quite noteworthy:

they’re putting up $5 million to fund independent researchers studying AI’s effects on user wellbeing.

The scope includes AI companionship, mental health conversations, long-term use, and how models should actually respond in these situations.

When I saw the news, I paid serious attention for the first time to a term:

AI wellbeing

This isn’t actually far from ordinary users.

Because over the past year, my own life has changed in a very obvious way:

AI is no longer a tool I occasionally open to ask a question.

It has started entering work, learning, writing, making decisions, arranging my life, even sorting out my thoughts when I’m feeling low.

And it’s always online.

A lot of the time, a thought has only just appeared, and before I’ve had a chance to think about it myself, I’ve already opened ChatGPT.

Technology enters our lives very fast, but the way people adapt to a new technology often isn’t that fast.

So I’ve started to feel we may not only need to learn how to use AI better.

We also need to start learning:

how to live with AI over the long term.

AI wellbeing isn’t just “look at the screen a bit less”

When we used to talk about digital wellbeing, it was easy to think of:

scrolling your phone less, controlling screen time, not staying up late watching short videos.

These problems still exist, of course.

Spending more and more time with AI may also mean more sitting, more screen time, and a blurrier line between work and rest.

But AI differs from many earlier digital products in one more way.

It has started taking part in our thinking.

We have it:

organize information, generate views, judge plans, write things for us, explain someone’s behavior, give career advice, even keep us company when we’re sad.

Search engines mainly help me find information.

Short videos mainly compete for my attention.

But AI can go further, into:

how I think, how I judge, how I understand myself.

So rather than asking:

how many hours of AI do I use a day?

what I’d rather ask now is:

what is AI replacing in me?

Some replacement is great; some needs watching

I don’t mind at all AI writing up my meeting notes.

I’m also happy to have it read dozens of pages of material, check formatting and handle lots of repetitive work.

Tools exist to lighten people’s load in the first place.

But some other things aren’t so simple.

For example:

a question I used to think about for ten minutes myself, my first reaction now is to ask AI;
a question that had no answer for the moment, I can now keep pushing on until I get an explanation that makes me comfortable;
a judgment I used to make myself slowly becomes “which does ChatGPT think is better”;
something I should have talked through with a real person may already have been discussed with AI for dozens of rounds.

That doesn’t mean these ways of using it are necessarily harmful.

But they start touching things more fundamental than “efficiency”:

cognition, emotion, relationships and agency.

Cognition: how convenient does it have to get before it starts changing how we think?

In 2025, Microsoft Research surveyed 319 knowledge workers and collected 936 real cases of GenAI at work.

One finding quite worth noting:

the more confidence people had in AI, the less critical-thinking effort they reported; while those more confident in their own ability to do the task put in more critical thinking.

The study also found that AI didn’t make critical thinking disappear, but gradually shifted it from “solving the problem directly” to verifying, integrating and overseeing AI’s results.

That’s also something I care about more and more lately.

AI saving me some thinking isn’t a bad thing, of course.

Calculators also saved us a great deal of mental arithmetic.

The question is:

do I still know where I can’t save?

When an answer looks very complete, can I still spot what’s wrong with it?

When a view is expressed very persuasively by AI, do I have my own standards for judging it?

If AI always generates the first version of a question for me, will my own first thoughts become fewer and fewer?

Emotion: AI is a kind of conversational partner that has rarely existed before

This may be even newer.

An ordinary person isn’t:

free 24 hours a day, always willing to hear you tell it again, replying within seconds, and centered on you every single time.

AI can be.

That’s also where it’s very valuable.

When people are anxious, lonely or in a mess, being able to find somewhere to sort out their feelings right away may itself be a help.

But this product trait also means we’ve rarely had to live long term with this kind of “partner” before.

In 2025, OpenAI and MIT Media Lab ran two related studies:

one automatically analyzed nearly 40 million ChatGPT interactions; the other was a randomized controlled trial with nearly 1,000 participants over four weeks.

The result wasn’t “talking with AI about emotions is bad.”

In fact, highly emotional use was concentrated in a small minority of users.

But the research also found that long daily use, and factors like seeing AI more strongly as a friend, were associated with some worse psychosocial outcomes. People’s own attachment tendencies and ways of using it also affected the results. The researchers stressed at the same time that the evidence is still very early, and much of it can’t simply be read as causal.

Research on AI mental-health use has also clearly started to increase this year.

A review published in July even estimated that about 27% of AI users have used AI for some kind of mental-health support, while noting that how the existing data is measured is still very inconsistent.

So this is no longer an especially fringe way of using it.

AI has another very particular problem: it has almost no natural stopping point

An article ends when you finish it.
An episode of a show ends.
A counseling session lasts 50 minutes.
When you chat with a friend, they always have things to do too.

But AI rarely tells you:

let’s leave it there for today.

You can keep going:

analyze it again, take another angle, give five more options, check again for anything missed.

That’s very efficient, of course.

But sometimes “no friction” is itself a new kind of friction.

A problem that should have been set aside for now can keep being worked on.
A feeling that actually needs time to digest can keep being put into words, explained and re-analyzed.
A piece of work that’s already good enough can be optimized another five rounds.

AI makes the cost of many actions very low.

But people don’t suddenly have unlimited attention, judgment and emotional capacity because of it.

That may also be where AI wellbeing differs greatly from the traditional screen-time problem.

So how do I understand AI wellbeing now?

I don’t think it’s:

using AI as little as possible.

Nor is it:

staying independent and doing everything yourself.

I actually hope to keep using AI a lot.

It really has expanded what I can do.

But I’ll pay more and more attention to one standard:

after using AI, has my agency grown or shrunk?

Agency is hard to translate fully into Chinese.

Here I take it to mean:

I can still think, judge, choose and act for myself.

After AI helps me understand a problem, am I better able to face it?
After AI helps me make a decision, do I know why I chose it?
After AI keeps me company while I sort out my feelings, can I go back to real life?
After AI helps me finish a piece of work, have I gradually come to understand that work, rather than becoming more and more unable to do without it?

If the answers are broadly “yes,” I don’t think this dependence is anything to fear.

I’ve started keeping a few very simple checkpoints for myself

1. For important questions, keep my own first version first.

It doesn’t need to be fully thought through.

Even three sentences first, then open AI.

I don’t want all my first thoughts generated for me by a model.

2. Ask AI to challenge me more, and use it less to reassure me over and over.

“What holes are there in this judgment?”

is usually, compared with:

“Do you think I did the right thing?”

more helpful to me.

3. Emotions can be sorted out, but don’t let AI rule on reality.

AI can help separate:

what happened, what I feel, what interpretation I made, and what other interpretations there might be.

But it doesn’t know what’s really going on inside another real person.

4. Let some questions go unanswered for a while.

Not every question needs another prompt.

Sometimes what’s really needed isn’t one more round of analysis, but time, feedback from reality, or going and doing something concrete.

5. Keep a bit of AI-free life.

Read things yourself.
Write a few lines yourself.
On a walk, don’t ask anyone anything.
Talk with real people.
Let yourself make some decisions first.

This isn’t to train some kind of “all-natural brain.”

It’s just a reminder to myself:

these are abilities I still want to keep.

What I find interesting about Anthropic putting $5 million into researching AI wellbeing isn’t that “AI has revealed another risk.”

It’s that AI has started entering a new stage:

the question we mainly asked before was:

what else can it do for people?

From now on, we may also need to ask, more and more seriously:

once it really starts doing so much for us, what should people keep?

My answer for myself right now is still simple:

good AI should expand people’s agency.

It can make me faster and more capable, and at times keep me company for part of the way.

But if one day it were taken away,

I hope I’d just be a little slower,

rather than suddenly not knowing how to think, how to choose, how to deal with the real world.

That’s probably my plainest understanding of AI wellbeing right now.

Day 096 · Which AI buzzwords are worth knowing?

Mon, 31 Aug 2026 00:00:00 GMT

While writing up AI-native recently, I also went through the AI terms I’ve kept running into over the past couple of years.

Agent, MCP, context engineering, harness, evals, long-horizon, Skills, memory, multi-agent…

There are a lot of them, and more keep coming.

I don’t think there’s any need to memorize a 50-word glossary.

For PMs, builders, or anyone who just wants to understand AI products better, what’s more useful is this: when you hear a new term, being able to roughly judge whether a new capability or product structure has really appeared behind it, or whether it’s just an old thing with a new AI name.

If I only look at the batch most worth knowing right now, I’d remember these first:

Agent / agentic workflow: AI doesn’t just answer questions; it can call tools, decide the next step and keep executing. One of the most important product paradigms right now, but with plenty of hype.

Long-horizon: whether AI can keep a complex task going for tens of minutes or even hours. Increasingly worth watching in 2026.

Context engineering: managing which instructions, history, tools, retrieval, memory and other information the model should see at each step.

Harness / scaffolding: the runtime system built around the model, including tools, the execution loop, the environment, permissions, state and feedback.

Agent Skills / Skills: packaging professional processes, instructions, scripts and resources into capabilities an agent can call on as needed.

Agent memory / memory: letting an agent keep, organize and reuse information from past interactions and tasks, rather than starting from zero each time.

MCP: letting AI applications connect to tools, data and external systems in a more standard way. Important, but what it solves is connection; it doesn’t make the model itself smarter.

Evals: first define what “doing well” means, then measure it systematically. Already a core capability of AI product engineering.

A few other terms are worth knowing too, but I won’t expand on them here:

reasoning models, multi-agent, grounding, guardrails, RAG.

The first two are still very hot; the last three are already closer to basic concepts in AI products.

Looking at these terms together is actually more interesting than memorizing definitions one by one.

Because together they reflect one thing:

the questions the AI industry cares about are slowly expanding from “how smart is the model” to “how do you turn a smart model into a reliable system.”

From chat to agent: the basic unit of work has changed

When using ChatGPT before, the most common unit was one interaction:

I ask a line, it answers a line.

Agents are starting to turn that unit into:

a task.

Once the user gives a goal, the system can look things up, call tools, change files, run code, and keep executing based on intermediate results.

Discussing the impact of agents on work this year, OpenAI described this shift as moving from brief chatbot interactions toward delegated, long-horizon tasks.

Long-horizon here doesn’t only mean “the model can think a bit longer.”

What’s really hard is:

after working through dozens of steps, does the agent still remember the original goal?
what happens when the intermediate state changes?
can it recover after a step fails?
what happens when the context is nearly full?

Once a task goes from tens of seconds to tens of minutes or longer, all these problems get magnified.

So as agents and long-horizon move forward, context, memory, harness and evals all become more important together.

They didn’t become popular at the same time by accident.

Prompt → context: the question is no longer just “how to ask”

When people talked about getting the most out of LLMs a few years ago, prompt engineering was hard to avoid.

Prompts still matter, of course.

But once agents really start working, what needs managing goes far beyond one prompt.

Context engineering is about:

what exactly should the model know at this moment?

That may include:

system instructions, user history, the current task state, available tools, retrieved data, memory, and the intermediate results left by the previous dozens of steps.

The longer an agent works, the more obvious this problem becomes.

You also can’t simply keep stuffing all the history into the context. Too much information can itself hurt the model’s performance.

That’s why a whole set of approaches has appeared: compaction, structured memory, just-in-time retrieval, subagents.

I think context engineering is one of the few new terms of recent years worth understanding seriously.

It describes a real change:

before, we mainly optimized “how should I tell the model”; now we also have to design “what should the model know at this moment.”

Skills and memory: how agents learn specific ways of working, and how they use past experience

I think these two terms are increasingly worth watching too.

Agent Skills / Skills

General models know a lot, but real work often has its own processes and methods.

A company’s coding standards, review methods, data analysis process and internal procedures shouldn’t have to be rewritten into a long prompt and stuffed into the agent every time.

The idea of Skills is to organize these instructions, scripts and resources, and load them when the corresponding task needs doing.

So what it actually solves is:

how to turn a general-purpose agent into one that better understands a specific way of working.

That isn’t the same thing as “whether the model has the knowledge.”

A model may know how to write code, but it doesn’t necessarily know:

how this team writes code, how it reviews, which steps can’t be skipped, which tools must be used.

Skills are more like adding a reusable layer of method on top of general capability.

Agent memory / memory

Memory handles a different problem:

of the things an agent has done before, which are worth keeping?

When should they be brought out again later?

This isn’t just “saving chat history” either.

Feed the model all the history as is, and it quickly becomes long and messy.

Really useful memory has to consider:

what’s worth remembering?
how to organize it?
when to retrieve it?
which information has expired?
what should be forgotten?

As an agent slowly goes from a one-off tool to a system you work with over the long term, memory naturally turns from a small feature into a system design problem.

So when I think about agent memory now, I’d rather see it as:

turning past interactions into context that’s genuinely usable later.

Rather than simply “the more AI remembers, the better.”

Model → harness: much of the difference between products actually happens outside the model

Harness is a term I’ve recently found especially worth knowing.

Roughly, it means:

the runtime system around the model that lets it actually work as an agent.

For example:

which tools can the model call?
how does it loop through execution?
what environment does it run in?
when does it need the user’s approval?
how is state saved?
what happens when the context fills up?
how does it retry after a failure?
how is what it did recorded?

None of these are capabilities of the model itself.

Even if two AI products call the same base model, their final performance can differ a lot.

Because they may give the model:

different context, different tools, different permissions, different feedback loops, and completely different execution environments.

This is also something I care about more and more when looking at AI products now:

seeing a line like:

already tells you rather little.

What’s more worth asking is:

what has been built around the model?

Everyone can buy similar frontier models.

What often decides whether a product works reliably, and whether you dare hand it to real users, is the whole system around it.

So many of the differences in AI-native actually happen outside the model.

MCP matters, but what it solves is connection

MCP isn’t really a new AI term anymore.

The problem it solves is fairly clear:

how to let AI clients / agents connect to tools, data and external systems in a more standardized way.

Before, different AI applications connecting to Google Drive, databases, GitHub or internal systems usually each built their own integration.

MCP wants to standardize this layer of connection somewhat.

That matters, because if an agent can only stay inside a chat box, what it can accomplish is ultimately limited.

But it’s also easy to hype here.

A product saying:

“we support MCP.”

doesn’t mean it suddenly has some AI moat.

MCP solves connection and interoperability.

Being connected to a system and knowing how to get things done well are two different layers of problem.

A bit like how supporting HTTP is of course important for internet products, but supporting HTTP in itself says nothing about whether your product is good.

Demo → evals: once it runs, how do you know it really got better?

Evals may be the least buzzword-y term in this batch.

What it does is very plain:

first define what success means, then measure it in a repeatable way.

The trouble with AI systems is that they aren’t as easy to test as traditional deterministic software.

Run the same task a few times, and the results may differ.

Switch models, change a passage of the prompt, add a tool, and some cases may improve while others regress.

Agents are more complex still.

They may call tools and change the environment across many turns, and a mistake at any step can affect what comes after.

So when evaluating agents, it’s no longer only the final answer that’s looked at.

It may also include:

what trajectory did it take?
were tools used correctly?
was the final task completed?
what state is the environment in at the end?

That’s also why I increasingly feel:

evals aren’t only done at final QA.

They’re actually part of the product definition.

A team first has to make clear:

what exactly do we mean by “AI doing well”?

If this question stays vague, later optimization easily turns into:

“I tried it, and it seems fine.”

What about reasoning and multi-agent?

I still keep these two terms in mind, but won’t put them on this post’s main thread.

Reasoning models are a real change in model capability: the model puts more inference-time compute into answering complex questions.

But that’s more about the model layer.

What I care more about in this post is: once model capability is already strong, what starts happening at the product layer.

Multi-agent is a real thing too.

Several agents can divide work, run in parallel and check each other, and some complex tasks really do suit this.

But it’s also very easy to over-engineer.

A problem one agent can solve doesn’t need to be forcibly split into five agents holding a meeting just to sound advanced.

So when I see “multi-agent system,” I don’t naturally assume it’s more advanced than single-agent.

It still comes down to:

why are several needed? What specific problem does it solve?

Buzzwords can actually be read as a map of the industry

Putting these terms in time order, I think you can see a fairly clear shift.

A few years ago there was a lot of discussion of:

prompt engineering

The focus was how to get the model to answer better.

Later there was a lot of discussion of:

agents / tool use / MCP

Models started needing to move from “answering” to “acting.”

What’s increasingly common now is:

context engineering / harness / long-horizon / Skills / memory / evals

The questions of concern have become:

how to get the agent the right information;
how to have it master a particular way of working;
how to have it keep genuinely useful experience;
how to keep it working continuously;
and how to know whether it has actually done the job well.

So buzzwords aren’t necessarily just marketing.

Some terms suddenly get discussed by lots of engineering teams because, once technology takes a step forward, new bottlenecks really start blocking products.

From this angle, new terms themselves can serve as a map of the industry.

They tell you:

what problems people are working on right now.

How do you judge whether an AI buzzword is worth learning?

I roughly look at four things:

what new capability does it describe that didn’t exist before?
which real constraint does it solve?
has it changed a product’s workflow, architecture or economics?
if I were really building a product, would this concept change one of my design decisions?

If the answers are all vague, it may for now just be a nice-sounding label.

If it affects how you organize context, grant permissions, connect tools, save state or test quality, it’s worth taking more seriously.

AI terms will probably keep multiplying fast.

There’s not much need to remember them all.

What’s more useful is slowly being able to see what real change is actually happening behind the new words.

Day 095 · What actually counts as AI-native?

Sun, 30 Aug 2026 00:00:00 GMT

In a recent interview, I was asked about a term that’s very common now:

AI-native

I used to see the word often too. My understanding was roughly: it isn’t simply plugging a large model into an existing product and adding a few AI features, but AI more deeply shaping the design of the product itself.

But that definition is still very broad.

Now a SaaS that adds an AI summary can call itself AI-native; a company that uses Copilot heavily can say it’s becoming an AI-native organization. It’s all over job postings, product websites and pitch decks.

So I later reorganized my thinking about the concept.

I think the key to judging whether a product is AI-native isn’t “does it use AI,” nor necessarily “did it use AI from day one,” but:

what exactly did AI change about this product?

The most basic distinction: is AI a feature, or has it entered the product’s core structure?

IBM defines AI-native roughly as:

AI is a core component from the start, not a feature bolted on later; it further shapes the product’s architecture, decisions, user experience and the whole system lifecycle.

“From the start” doesn’t really need to be taken too literally.

A software company that has existed for ten years can just as well redesign its core product because of AI.

For example, a traditional customer service system originally had tickets, search and template replies, and later added an AI summary.

That’s a very useful AI feature.

But if AI starts taking part in understanding the customer’s history, querying related systems, judging the next step and generating a plan for handling it, while the human role gradually becomes reviewing and correcting, the change is clearly deeper.

At that point, what AI has changed is no longer just a feature, but the whole workflow, and the division of labor between people and the system.

So more worth looking at than “did it use AI from day one” is:

has AI changed the product’s core value, workflow, system structure, even its economics?

These days I roughly judge from four places.

1. Has the core value changed?

Much software used to require users to know for themselves:

which page to go to, which fields to fill in, which buttons to click, in what order.

AI products are starting to show another possibility:

users first say what they want to accomplish, and the system helps judge how to do it.

The change isn’t just an extra chat box.

The division of labor between product and user starts to change.

Software used to mostly help users carry out a series of operations; now some AI products are starting to take on part of the judgment and execution.

For example, users no longer need to tell the system explicitly:

look up this data first, then open that page, then do the next step.

They may only need to give a goal, and the system understands the context, calls tools, handles intermediate results, and asks a person to review or correct when needed.

When OpenAI talks about agents, it often uses one word: delegation.

I think the word is very accurate.

One important change in AI-native is software gradually moving from “helping users operate” toward “accepting part of the task as delegated.”

Of course, not every product will end up as a fully autonomous agent.

What really matters is whether AI starts taking on part of the work that users used to have to decide explicitly.

2. Has the workflow really been redone?

Many so-called AI products are really still:

the old workflow + an AI button.

Writing a report used to mean filling in ten fields; now there’s one more thing beside them:

Generate with AI

The feature may be very useful, but the process itself hasn’t changed much.

Deeper AI product design rethinks the whole workflow:

how much do users still need to input?
which steps can the system decide on its own?
when should it ask a person?
when can it keep calling tools?
how does it recover after a mistake?
is the person operating every step, or gradually becoming the one who sets goals, reviews and corrects?

So the process is no longer entirely fixed:

step 1 → step 2 → step 3

Along the way, more dynamic paths appear, decided by context, the model’s judgment and the results of each run.

That’s also why concepts like agents, tool use and context engineering are becoming important together.

Once the system can judge the next step itself, product design naturally can’t only consider pages and buttons.

3. A whole new set of product capabilities grows around the model

This is a layer I used to overlook fairly easily.

If you’re just building a very simple AI feature, it may look like:

prompt → response

connect to a model API and get a result.

But once AI really enters the core workflow, you quickly run into lots of extra questions:

what does the model know right now?
which data can it access?
which tools can it call?
does it have permission to perform this action?
what happens after a mistake?
how do you know whether a new model made the product better or worse?

So around the product, things slowly grow:

context, memory, tools, permissions, guardrails, evals, observability, fallback…

These didn’t entirely not exist before, of course, but AI products clearly amplify how much they matter.

That’s also where I now think AI-native is easily misunderstood.

People easily put all their attention on the model itself:

which model does it use?
is it the latest?
how high are the benchmarks?

But once you’re really in production, the model is only part of the system.

Everyone can buy similar model capabilities; what often decides whether a product works reliably, and whether you dare hand it to real users, is the whole system around the model.

So now I actually think:

the “native” in AI-native often shows up around the model.

4. Even the economics may change with it

AI products have another very practical difference:

usage itself keeps generating compute costs.

A complex task may include model calls, retrieval, tool execution, long context, multi-turn reasoning, and possibly rerunning after failure.

So the product needs to consider early on:

which tasks are worth a stronger, more expensive model?
when can things be routed to a cheaper model?
how long can an agent run?
how much allowance do users get?
charge per seat, per usage, per task, or per final outcome?

These are no longer just technical questions.

They directly affect the product experience, margins and the business model.

In traditional SaaS, once many features are built, the marginal cost of a user clicking a button one more time is very low.

Not necessarily so for AI products.

So if AI really enters the core of a product, it sometimes affects everything from the UI, workflow and system architecture all the way to the economics.

AI-native is more a matter of degree than a dividing line

So now I don’t really want to split all products simply into:

AI-native / not AI-native.

It’s closer to a process of going progressively deeper.

AI feature

Adding AI capabilities like generation, summarization and search to an existing product.

AI-enabled

AI has clearly changed some important workflows, but most of the product’s structure still follows the old ways.

AI-native

AI starts shaping what value the product offers, how users accomplish tasks, how the system runs, and how the product is evaluated and charged for.

The boundaries won’t be very precise, of course.

And AI-native itself has already been used very loosely by marketing.

If AI really does gradually become default infrastructure the way the internet did, the term may slowly lose its power to distinguish, too.

Today we no longer make a point of saying:

“this is an internet-enabled product.”

Because that’s the default.

AI-native may end up in a similar place.

So how do I judge whether a product is AI-native now?

I won’t accept the label just because a product uses the latest model, has a chat box, has agents, or its website says AI-native.

I look at a few more concrete things:

has AI changed the product’s core value?
has it changed the workflow by which users accomplish tasks?
does it require the system to newly handle context, tools, permissions, evals and failure?
has it gone on to change the cost structure and business model?

If basically none of these apply, and it’s just old software with a few AI features added, then AI-native is more packaging.

If AI’s presence has already made the product need its core logic redesigned, then the term really starts to carry information.

That’s also a way of judging I use more and more often now when facing lots of AI buzzwords:

don’t look first at what it’s called.

First look at:

what has it actually made different from before?

Day 094 · Detecting AI has become a business

Sat, 29 Aug 2026 00:00:00 GMT

Publishing a post on Substack yesterday, I noticed a feature for the first time:

Scan for AI text.

I clicked it on a whim, and the result was very blunt:

AI-generated 100% · AI-assisted 0% · Human 0%

My first reaction: that absolute?

My second reaction: but I can’t really say it got it wrong.

Because looking back at how the post was made, the experience, views and judgments came from me, and what was worth writing and which conclusions didn’t hold up were things I revised round after round.

But the final English prose — the first draft — really did come largely from ChatGPT.

I provided the Chinese original, the ideas and the feedback; AI generated the English; then I kept cutting, changing and reorganizing.

So the 100% actually raised a question for me:

what exactly does an AI detector detect?

Digging in, I found something even more interesting:

as AI gets better and better at generating content, “detecting whether it’s AI-generated” has itself become a business.

AI detectors no longer just look for an “AI flavor”

Substack added AI text detection in July this year, built on a model from a company called Pangram. Readers can scan newly published posts, Notes and comments themselves to see how much of the text is Human, AI-Assisted or AI-Generated.

My impression of AI detectors was still stuck at using some statistical features of language to judge whether a piece was too regular, too predictable.

It’s much more sophisticated now.

Pangram’s latest model judges whether different parts of a text are closer to Human, AI-Assisted or AI-Generated. In training, they take real human articles and have different LLMs generate “mirror versions” close in topic, length and tone, forcing the classifier to learn differences finer than “written very neatly.”

They also specifically find human articles the model misjudged as AI and bring them back for more training.

I find that approach quite clever:

the humans who seem most like AI are exactly the ones most worth using to teach the detector.

So what we usually call “getting rid of the AI tone” isn’t quite the same thing as what machines detect.

AI tone ≠ AI provenance.

Not reading like ChatGPT doesn’t mean a machine can’t see the AI generation signal in it.

And detectors really are improving. A peer-reviewed study this year compared Pangram, GPTZero, Copyleaks and Turnitin on fully AI, hybrid and humanized AI text, and Pangram performed best overall. But the researchers also cautioned that these tools should still assist judgment, and aren’t suited to carrying high-stakes decisions alone.

So what does my “100% AI” actually mean?

This is easy to misunderstand too.

It isn’t saying:

Pangram has 100% mathematical certainty that this post was written by AI.

It’s closer to: the text in this post was judged, overall, to have AI-generated provenance.

On my post, the distinction is especially clear.

It can see the final English sentences, but doesn’t know why I wanted to write about this, whose experience it was, who proposed the views; nor which suggestions I rejected, or which parts went back and forth through many rounds.

So detectors are fairly good at answering:

how were these words most likely produced?

They can’t answer:

whose ideas are these?

These two questions used to overlap a lot. A person thought it, and the same person wrote it.

Once people and AI start creating together, they slowly come apart.

So saying my post was entirely human-written isn’t accurate; but if someone said it was “100% created by AI,” I wouldn’t accept that either.

Closer to the truth is:

the thinking and judgment came mainly from a person; the generation of the English prose relied heavily on AI.

What the detector sees is mainly the latter.

Why would anyone actually pay for something like this?

This is the part I found most interesting while looking into Pangram.

If an AI detector only satisfied:

“let me see whether ChatGPT wrote this,”

it would be hard to imagine much commercial value in it.

The really valuable situations are where how content was produced affects how we use it.

Take student essays.

The teacher isn’t short of those 2,000 words. They want the assignment to show whether the student understood the question, did the research, and can organize and express it.

An assignment used to serve as some evidence of those abilities.

Since AI arrived, the link between the work and the ability isn’t so direct anymore.

Hiring is the same. Cover letters, writing samples and take-home assignments were all used to understand a candidate’s expression and thinking. Now, seeing polished material, the hiring side needs to ask one more question: how much of this person’s ability does it still represent?

Content platforms face another problem.

When I subscribe to a writer, I usually don’t just want a few hundred words of information; I’m following that person’s experience, judgment and way of observing.

As mass-producing good writing gets cheaper and cheaper, readers naturally start to care:

what does what I’m seeing actually represent about this person?

These aren’t hypothetical use cases.

Pangram already sells its product to schools and universities, and offers enterprise products and a developer API. Official API uses include making the detection result a layer in product decisions, a signal for moderation queues, or bulk-scanning historical content; Quora uses it to handle AI-generated answers.

It has also become a clearly commercialized product: an individual subscription at $20 a month, Professional at $65 a month, plus a usage-based API, education licenses and enterprise plans.

A more direct market signal: in July this year Pangram raised $9 million, led by Menlo Ventures. TechCrunch’s report mentioned that its API customers already include Substack, Quora, schools and universities, publishers, agencies and recruiting teams.

Funding can’t prove how big this market will ultimately be, of course.

But it at least shows this is no longer just a small tool for “academia worried about students cheating”:

platforms, institutions, companies and investors have all started to think that the authenticity and verification problems brought by AI-generated content are worth paying to solve.

AI has weakened some signals we used to take for granted

These use cases actually have something in common.

Before, seeing an essay, we assumed someone had done the research and the writing.

Seeing a writing sample, we assumed it more or less represented the person’s expression.

Seeing a writer publish steadily, we assumed we were coming into contact with that person’s thinking.

These were all implicit information that content used to carry.

AI hasn’t made content disappear.

It has just made the relationship between content and the person behind it less certain.

So new signals are needed.

Detection is one.

Authors explaining how AI was involved is disclosure.

Going further, if you can reliably record where a piece of content came from and what processing it went through, that’s provenance.

They all fill the same gap:

when the final result can no longer fully explain how it was produced, what other information do we need to correctly understand what we’re seeing?

Of course detectors can’t solve the whole problem.

Especially as hybrid writing becomes more and more common, simply splitting things into “used AI / didn’t use AI” is already a bit crude.

So I’d rather understand detection results as a signal that needs further explanation, not a verdict.

What interests me most is the “reverse market” AI creates

This is what I found most interesting in this research.

When we talk about AI startups and AI product opportunities now, we’re used to following model capabilities:

what can AI do now?
which work can finally be automated?
where can things be cheaper and faster?

That’s the most direct layer of opportunity, of course.

But once a technology really enters the world at scale, something else happens too:

it breaks some old mechanisms, while creating new costs, risks and uncertainty about information.

So a second-layer market follows.

Once email spread, spam became extremely cheap, so anti-spam became infrastructure.
The more the internet and online transactions mattered, the more valuable attacks, fraud and identity impersonation became, so cybersecurity, fraud detection and identity verification kept growing.
Generative AI is now on the same path.

AI can produce huge amounts of content at low cost, so detection, provenance and content authenticity start to have value.

AI agents can take actions for people, so permissions, guardrails, audit trails and observability become more important.

LLMs can give answers that seem more and more real, so grounding, evaluation and source verification become important along with them.

AI coding can greatly speed up code production, so naturally someone will need to solve the testing, security, quality and maintenance of generated code.

None of these products seem to be “making AI stronger.”

Some even specialize in checking it, limiting it, putting boundaries on it.

But they don’t contradict AI’s growth.

It’s precisely because the capabilities up front keep getting stronger that the demand behind them keeps getting more real.

I later felt that “reverse market,” while vivid, isn’t strictly against AI either.

It’s more like a second-order market that arises once AI spreads:

the first-layer market creates new capabilities.

The second-layer market handles the new problems that come once those capabilities spread.

This angle is quite useful for how I look at AI product opportunities myself.

From now on, when I see a new AI capability, besides asking:

what can it do for people now?

I might also go on to ask:

what has it made extremely cheap?
which judgments, rules or trust mechanisms no longer work well because of it?
where will new fraud, abuse, error or uncertainty appear?
who ultimately bears the cost of these problems, and who would be willing to pay to solve them?

Pangram is a very concrete example.

Generative AI makes text cheaper and cheaper, more and more human-like.

So “how did this text actually come about” — an answer almost nobody used to need to buy — has started to have a price too.

AI’s opportunities aren’t all in getting AI to do more.

Some opportunities appear after AI is doing more and more.

Day 093 · Before you try to learn everything

Fri, 28 Aug 2026 00:00:00 GMT

This is the last post in the series “Who is hiring AI PMs right now.”

For this round I collected 43 public AI product roles as of August 20, 2026, and drew on hiring-market research like the Stanford AI Index and PwC’s AI Jobs Barometer.

The first three posts did three things:

the first looked at what different jobs so-called AI PM actually includes;

the second broke down where different roles’ “technical” requirements really differ;

the third looked at why once-uncommon specializations like evals, harness and forward-deployed are starting to appear.

Having written this far, it’s easy to get a side effect:

more anxiety.

You thought being an AI PM might just mean learning some prompting.

Now you find there’s also:

Python, RAG, agents, evals, MCP, harness, system design, model behavior…

And the list keeps getting longer every day.

So in this last post, I want to bring the question back in:

if you weren’t an AI PM before, how should you actually find your place now?

These days I increasingly advise against starting with:

“what does an AI PM need to know?”

as the first question.

Because that question too easily leads people into a list they’ll never finish.

Step one: first take stock of what you already have, not what you lack

I think this is what experienced PMs most easily overlook when moving into AI.

After several years, even a decade or more, of building products, what you’ve built up might be:

payments; healthcare; search; developer tools; risk; enterprise SaaS; design; data products; customer delivery; growth.

Now that AI is here, this experience hasn’t suddenly become worthless.

The question really worth asking isn’t:

should I retrain myself into an “AI person”?

But:

combined with which layer of AI does my most defensible experience become more valuable?

For example:

Search / recommendation → retrieval / personalization / model product

API / platform → agent platform / harness

Risk / trust & safety → evals / safeguards

Finance / healthcare / legal → vertical AI

Enterprise SaaS → AI transformation

Solutions / consulting → forward-deployed

Consumer / growth → applied AI

Starting here, the routes narrow down a great deal at once.

Step two: choose a product layer, not just a big label

There was one common conclusion running through the first three posts:

the term “AI PM” is too broad.

If you roughly picture AI products as:

model ↓ platform / harness ↓ evals / safety ↓ applications ↓ industry workflows ↓ enterprise deployment

each layer needs completely different people.

If a PM who has long worked on enterprise workflows spends half a year forcing themselves to learn pre-training, hoping to compete directly with people from ML research backgrounds for a core model PM role, I’m not sure that’s the best investment.

Conversely, if someone who has long worked on ML platforms only applies for an ordinary AI chatbot PM role, they may be wasting their own edge.

So the question isn’t:

which kind of AI PM is the most impressive?

But:

which layer of AI will amplify what I’m already strong at?

I think that matters far more than “how many AI skills do I still lack.”

Step three: replace “I’ve studied AI” with evidence

Courses, articles and tutorials all have value, of course.

Entering a new field, you have to learn first.

But from a hiring point of view:

“I’ve studied it” proves you’ve been exposed to it.

It hardly proves you can really do it.

A very clear trend in this round of JDs is that:

hands-on, prototype, evals, failure handling, model quality, Python, SQL…

come up more and more often.

So if you really want to build AI product experience, I’d actually lean toward:

fewer demos, and taking one project deep.

A convincing AI project should at least be able to answer these questions.

Why does this need AI here?

Why not ordinary rules, search or workflow automation?

How is the work divided in the system?

What does the model do? The rules? The tools? The people?

How do you know whether it’s doing well?

What’s the baseline? Where did the test set come from? How is the rubric defined?

How is it most likely to break?

Hallucination? The wrong tool? Permissions? Context? Latency?

What happens when it breaks?

Retry? Fallback? Refuse? HITL?

What do you watch once it’s really live?

Quality? Task success? Latency? Cost? Reliability?

If one project can explain all of this clearly, I think that, compared with ten of:

“I used AI to build an XXX,”

it’s more convincing.

Step four: when you see a term you don’t know, first judge what it actually is

Looking at AI JDs now, one reflex easily kicks in:

you see an unfamiliar term,

and immediately add it to: the study list.

I actually think you can first sort the gaps into three kinds.

A. It’s a core ability for this role

For example:

you want to apply for an AI eval PM role but have no idea how evaluation is designed.

That really does need filling in.

Not learning a tool, but the ability itself is missing.

B. It’s a specific tool

For example:

you already understand agent workflows and have done tool calling; you just haven’t used LangGraph.

In this case, not knowing a certain framework doesn’t mean you can’t do this kind of work.

Tools change fast.

Not knowing a tool and lacking an ability are two different things.

C. The role doesn’t actually need it at all

For example:

applying for a healthcare applied AI PM role, but anxious because you can’t train models.

Or applying for AI transformation, but wondering whether you should first systematically learn PyTorch.

That may well just be filling in the wrong direction.

This matters especially in AI.

Because there are simply too many new terms, and it’s easy to mistake:

“I don’t know this word”

for:

“I lack competitiveness.”

Step five: find your intersection

If I had to compress positioning into a formula, I’d now write:

past edge × AI product layer × verifiable evidence

rather than:

years of experience × number of AI skills

For example:

enterprise PM × AI automation

healthcare × AI agents

risk × AI evals

developer platform × agent infrastructure

design × human–AI interaction

customer delivery × forward-deployed AI

I think framings like these, compared with:

“I used to do X, and now I want to move into AI,”

carry more power.

Because the latter story easily cuts the past and the future into two pieces:

the past doesn’t count anymore; I’m starting over now.

But reality usually isn’t like that.

More likely:

you’ve been solving a certain kind of problem all along, and AI is changing how that kind of problem gets solved.

At that point, so-called “transition” is more like a recombination.

Of course, not all past experience automatically becomes an edge

This needs saying honestly too.

If for many years your main work was:

coordinating; following up; writing requirements; meetings; pushing projects forward;

but it rarely built up:

independent product judgment, technical understanding, industry knowledge, data skills, clear product results,

then the “Senior PM” title itself won’t automatically become an advantage as an AI PM.

If anything, AI is raising some requirements:

experimenting faster;
staying closer to the system;
defining quality more clearly;
judging trade-offs more independently.

So years of experience still have value, of course.

But what companies are repricing may not be:

how many years you’ve done it.

But:

what hard-to-replace judgment those years actually built up.

If I were planning an AI PM transition again today, I’d go in this order

Not opening a course website first.

But doing four steps first:

1. Find 10 JDs you’d really apply for

Don’t look for the ones that “look the most impressive.”

Find the ones you’d realistically be willing to apply for.

2. See which layer of AI product they actually belong to

Model?

Platform?

Applied AI?

Vertical AI?

Transformation?

Forward-deployed?

3. Find the common requirements

Don’t record every keyword.

Only find what keeps recurring:

core abilities.

4. Then put yourself into it

In three columns:

what I already have evidence for

what I understand but don’t have enough evidence for

what I really can’t do, and the role really needs

Only this last column should go into your study plan.

A very simple positioning sheet

If you’re considering AI PM right now, you can first fill in these four lines:

The problem domain where I’m most defensible: ＿＿＿＿＿＿＿＿＿＿

The AI product layer that best amplifies that edge: ＿＿＿＿＿＿＿＿＿＿

The AI abilities I can already prove: ＿＿＿＿＿＿＿＿＿＿

The core abilities real JDs keep asking for that I currently lack: ＿＿＿＿＿＿＿＿＿＿

Once filled in, the study list is usually much shorter than at the start.

And closer to the real market.

By the end of this research, what I felt wasn’t:

“there’s so much an AI PM has to learn.”

But slightly the opposite.

Not everything needs learning.

AI changes too fast; if you wait to feel safe until you’ve crossed off an entire skills list, that day will probably never come.

What may always be more worth figuring out first are two questions:

what kind of problem was I always best at solving?

And then:

now that AI is here, where is that ability starting to be needed more?

A person’s place is often hidden where these two questions intersect.

Part 4/4 of the series “Who is hiring AI PMs right now.” The whole series is based on 43 public AI product roles collected as of August 20, 2026, with reference to the Stanford AI Index 2026, PwC’s 2026 AI Jobs Barometer and other sources. The first three posts covered how AI PM roles break down, the technical requirements of different positions, and new specializations like evals / harness / forward-deployed; this one is where the whole series lands: once you know what the market is hiring for, how to reassess your own place.

Day 092 · Evals, Harness, Forward-Deployed: new kinds of work

Thu, 27 Aug 2026 00:00:00 GMT

Evals, Harness, Forward-Deployed: AI hiring is growing kinds of work that didn’t exist before

This is the third post in the series “Who is hiring AI PMs right now.”

The first two posts did two things:

the first broke AI PM down into several completely different jobs;

the second looked at where the so-called “technical requirements” of different roles actually differ.

Having organized things this far, I noticed another quite interesting change:

AI product teams themselves are redividing the work.

As of August 20, 2026, among the 43 public AI product roles I collected, besides the familiar AI Product Manager, some names rarely seen a few years ago kept appearing:

AI Evaluations, Agent & Harness, Model Behaviors, Human Data Platform, Safeguards, Forward-Deployed…

Seeing these words at first, it’s easy to think:

is the AI industry inventing new buzzwords again?

But after going through the JDs line by line, I actually think their appearance is quite natural.

As an industry starts to mature, one thing often happens:

a problem someone used to handle “on the side” slowly becomes complex enough to need a clear owner.

AI is going through this process now.

1. Evals: someone starts owning “what counts as good”

Evals are easily understood as:

the AI version of QA.

But there’s a very fundamental difference from traditional testing.

In ordinary software, we can often write down in advance:

what the input is, and what the correct output should be.

Not necessarily with AI.

The same question can have many reasonable answers; a version may improve overall while regressing in some key scenarios; a high average score can coexist with an occasional completely unacceptable mistake.

So the first thing an eval has to answer isn’t even:

did the test pass?

But:

what actually counts as good?

Only then come:

what to test; where the test set comes from; how the rubric is defined; what can be judged by rules; what can use LLM-as-a-judge; what must be reviewed by people; how to run regressions when the model version changes; how new failures seen in production get back into the test set.

Box now has a role literally called Product Manager, AI Evaluations.

The PM owns datasets, graders, offline / online evaluation, model selection and so on.

At this point, evals are no longer just “testing once the product is done.”

They start to affect:

how models are chosen, how the product iterates, and when it can go live.

So I now lean toward understanding evals as:

the quality infrastructure of AI products.

Even if not every company ends up with a dedicated eval PM, this capability will very likely make its way into more and more ordinary AI PMs’ work.

2. Harness: beyond the model, a whole layer decides whether an agent is any good to use

Harness is a concept I’ve only recently started understanding more seriously.

Different companies draw its boundaries a bit differently, so I don’t want to give it a very rigid definition.

If I had to put it plainly in one sentence:

the model “thinks”; the harness lets it “get things done” in a real system.

Once an agent is really running, it often also needs:

context; memory; tools; permissions; a sandbox; orchestration; retries; fallbacks; observability; evaluation.

These things used to be easily seen as engineering details.

But once an agent no longer just answers a sentence, and starts to:

call tools, query databases, change files, submit requests, take real actions,

they become product questions directly.

For example:

what exactly can this agent see?

Which tools can it call?

Which operations must require confirmation?

If it fails halfway through, does it retry, roll back, or hand over to a person?

Why did it make this decision, and is there a trace to look at?

Binance already has, outright:

Product Manager, AI Agent & Harness

The JD explicitly covers task flow, capability boundaries, failure fallback, harness evaluation, task sets and more.

I find the job title itself quite interesting.

It shows that some companies no longer treat the layer around the agent as:

“just hook the model up to a few APIs.”

But are starting to treat it as a product system that needs designing and running over the long term.

3. Model Behaviors: even “how the model should behave” is getting a product owner

Anthropic currently has another role I find very novel:

Research Product Manager, Model Behaviors

Model capability used to be easily thought of as:

how high the math benchmark is; how strong the coding is; how broad the knowledge is.

But in real use, there’s a set of questions much harder to express with a single benchmark:

when it doesn’t know, should it admit it?

Faced with a vague request, does it guess directly, or clarify first?

When should it refuse?

How should it express uncertainty?

Will it over-accommodate to make users happy?

What should it do with conflicting instructions?

All of these strongly affect the product experience.

In other words, the “product object” PMs are responsible for in the AI era is extending further forward.

Before, we mainly designed:

pages, flows, features.

Now someone is even specifically responsible for:

how the model should behave.

And this takes research, data, evals and product judgment working together.

4. Human Data Platform: human feedback is becoming a product system too

There have always been lots of “people” behind AI.

Someone has to judge:

which of two answers is better;

whether this result is risky;

whether the agent really completed the task;

which kind of error is more serious.

When the volume is small, it can be handled by hand.

Once it scales, a whole set of problems quickly appears:

how are tasks designed? Who’s suited to judge? Are different people’s standards consistent? How is quality checked? How are costs controlled? Which feedback goes into training? Which goes into evals?

Anthropic now has product management roles related to its Human Data Platform.

I think this change also says a lot:

human feedback is no longer just a back-office operations step.

It too needs:

a platform, processes, quality standards, tools and product design.

In other words:

even the “human” in “human in the loop” is being systematized.

5. Safeguards / safety measurement: safety turning from principle into metrics and systems

Talk of AI safety used to easily stop at:

AI should be safe.

True, of course.

But when actually building products, that sentence is far from enough.

You also need to keep asking:

what specific harms are there?

How are they detected?

Do the safeguards actually reduce risk?

How high is the false positive rate?

Which risks are acceptable?

Which must be blocked?

When an incident happens, how does it feed into the next round of product iteration?

OpenAI now has a Safety Measurement PM.

Anthropic’s hiring has gone further, splitting out:

rare harms, child safety, vertical safeguards and other directions.

By this point, safety is no longer just policy.

It’s:

metric + system + operations + product decision.

I think this is also an area where AI products differ very clearly from ordinary software.

Many “value judgments” ultimately have to be translated into:

how exactly the system works.

6. Forward-deployed PM: someone dedicated to getting the demo into the real world

Forward-deployed is a kind I paid less attention to before this research, but have come to think more and more important.

Scale AI is currently hiring a Forward Deployed Product Manager.

This kind of PM doesn’t just:

gather requirements → write the PRD → go back to headquarters to plan the roadmap.

They go deep into the customer’s site:

understanding the business problem; connecting data and APIs; building a prototype together; testing; deploying; then slowly abstracting one-off customer solutions into reusable standard products.

So it’s a bit like:

PM + solutions architect + consultant + builder.

Why do AI companies increasingly need people like this now?

Perhaps because building a demo today isn’t that hard anymore.

What’s really hard is:

the customer’s data is messy; the permission system is complex; the process has lots of exceptions; nobody in the organization wants to change how they work; model performance starts dropping once it hits the real scenario.

Where many AI projects end up stuck is no longer:

the model isn’t smart enough yet.

But:

it never really entered the workflow.

Forward-deployed is really dedicated to solving this “last mile.”

7. AI transformation: someone starts owning a company’s AI product portfolio

Another new division of labor happens more inside traditional enterprises.

Imagine a company suddenly having:

dozens of AI ideas; lots of departments each running their own pilots; a pile of vendors coming to demo; everyone saying their scenario is especially suited to AI.

Very soon someone needs to answer:

what should be done first?

What’s worth building?

What should simply be bought?

Which capabilities should be built once as a platform?

Which data can be used?

How should permissions be governed?

How is ROI calculated?

How do you get employees to actually adopt it, rather than everything ending after the demo?

So these are increasingly common now too:

AI Transformation PM, AI Product Owner, Internal AI Product.

They aren’t necessarily responsible for a single AI feature.

It’s more like running:

how a company should use AI.

What I think it has in common with forward-deployed is:

once AI reaches this stage, the model itself is only part of the problem.

Organization, process and implementation take up a larger and larger share.

Put these new roles together, and there’s a very clear logic

I later tried ignoring the job titles and looking only at why they exist:

models change → so you need evals

agents act → so you need harness / permissions

model behavior itself affects the experience → so Model Behaviors appears

AI depends on lots of human judgment → so a Human Data Platform appears

AI creates new risks → so you need safeguards / safety measurement

enterprise deployment is hard → so forward-deployed appears

a company sprouts lots of AI projects at once → so you need AI transformation

Seen this way, these names no longer look so much like “AI jargon.”

They’re all answering the same question:

once a new complex problem appears, who should own it for the long term?

This has a very practical effect on job hunting too

If you only search for:

AI Product Manager

it’s actually easy to miss opportunities that suit you well.

Now you can also look at:

AI Evaluations · Agent Platform · Agent Harness · Model Performance · Research Product Manager · Model Behaviors · Safeguards · Forward-Deployed Product Manager · AI Applications · AI Transformation

Of course, these titles haven’t fully settled yet.

Some names may still be around in a few years; some may disappear.

So rather than memorizing every word, I think what’s more useful is:

first look at what problem it’s solving.

Because titles will change.

Problems usually don’t disappear that fast.

With three posts written, this research’s map of the market is basically laid out:

the first is what jobs AI PM actually breaks into;

the second is how deep the technology needs to go for different roles;

and this one is which new specializations AI product organizations are growing.

In the last post, I want to bring the perspective back from “what companies are hiring for” to the individual:

if you weren’t an AI PM before, facing so many roles and skills, how should you actually find your place?

I think the answer may not be:

adding an even longer study list to yourself.

Part 3/4 of the series “Who is hiring AI PMs right now.” The research is based on 43 public AI product roles collected as of August 20, 2026; this post mainly draws on current postings from Anthropic, OpenAI, Box, Binance, Scale AI and others. The job titles mentioned represent emerging trends in the division of labor, and don’t mean these titles have become a unified industry standard.

Day 091 · How technical does an AI PM need to be?

Wed, 26 Aug 2026 00:00:00 GMT

This is the second post in the series “Who is hiring AI PMs right now.”

For this round I collected 43 public AI product roles as of August 20, 2026, including positions at OpenAI, Anthropic, Scale AI, Box, Binance, Apple, Ford, n8n and companies in finance, insurance and healthcare, with reference to sources like the Stanford AI Index 2026 and PwC’s 2026 AI Jobs Barometer.

The first post drew a map:

so-called AI PM is already many completely different jobs.

Some are close to the model, some build agent platforms, some own evals, and some are responsible for bringing AI into specific industries and enterprise workflows.

That also explains a question I’d always found hard to answer:

how technical does an AI PM actually need to be?

If you only follow the discussion, you easily get two extreme answers:

one side says PMs don’t need to code; understanding users and products is enough;

the other says that to be an AI PM now, not knowing Python, ML and system design won’t cut it.

After going through this batch of real JDs, I think both sides are partly right.

The problem is:

“technical ability” isn’t one thing.

I’d roughly break the requirements in job postings into four layers.

Layer one: AI literacy — at least knowing what you’re doing

This is the most basic layer.

You don’t necessarily have to code, but you at least need to understand:

roughly what LLMs are good and bad at;
why hallucination happens;
what problem RAG solves;
how an agent differs from an ordinary chatbot;
what tool calling, grounding, context and HITL each do.

These concepts aren’t hard to learn.

What really matters is whether they affect product judgment.

For example, an engineer says:

this scenario isn’t suited to stuffing more information into the context; it should switch to retrieval.

Or:

this step can’t be executed by the agent directly; better keep human approval.

The PM doesn’t necessarily have to implement it, but should at least be able to join the discussion:

why? What does it cost? Is there another option?

If every AI-related decision can in the end only become:

engineering recommends doing it this way.

then it isn’t only the technology being outsourced.

Part of the product judgment has been outsourced with it.

So I now understand AI literacy as:

you don’t need to become an engineer, but you at least need to understand why the product works the way it does.

That should already be the baseline for an AI PM.

Layer two: builder fluency — can you try it yourself first?

One layer up, I see more and more JDs starting to mention:

hands-on, prototype, Python, SQL, notebooks…

It’s easy to feel pressure here:

do PMs have to code now too?

I don’t think you need to jump straight to that conclusion.

What these roles more often express is a different requirement:

don’t depend on the engineering team for all validation.

For example, with a product idea, you could first, yourself:

call a model API;
set up a simple RAG;
have an agent call two tools;
run a few dozen test cases;
compare two models;
look at traces and logs;
look at the results with some simple Python / SQL.

Box’s Product Manager, AI Evaluations is a very typical example.

It explicitly requires the PM to use Python, SQL and notebooks to build evals themselves, analyze results and prototype data pipelines.

That isn’t asking the PM to maintain production code.

It’s more like saying:

if a question can be validated yourself in a few hours, don’t first spend days writing documents and then wait for someone else to validate it for you.

This also has to do with the development of coding agents and AI-assisted building.

The cost of prototyping has dropped noticeably.

Some things for which “the PM can’t do it themselves” used to be perfectly reasonable have now become:

“why not just try it yourself first?”

So what really matters about builder fluency isn’t “do you know some programming language.”

It’s:

can you independently push a fuzzy idea to the point where there’s evidence to judge by?

Layer three: system fluency — knowing why an AI product fails

One layer further up is, I think, where AI PMs really start to separate.

A demo running doesn’t make it a product.

Once it’s really in use, you quickly run into:

which tools can the agent call?
which tools must it absolutely not call?
how much should go into the context?
when is memory needed?
when a tool keeps failing, retry, fall back, or stop?
which actions must a person confirm?
after a model update, how do you know whether there’s a regression?
quality goes up a little but cost doubles — ship it or not?
the stronger model is two seconds slower — will users accept that?

At this point what you need to understand is no longer:

“what is RAG?”

But:

under what conditions the whole system is reliable, and where it breaks down.

For example, Binance’s AI Agent & Harness PM has responsibilities including:

task flow, capability boundaries, failure fallback, harness evaluation, task sets.

OpenAI’s API Agents PM needs to discuss SDKs, APIs and agent infrastructure together with research and engineering.

These roles don’t necessarily require the PM to implement everything.

But they require the PM to make product judgments about these things.

I think there’s an important distinction here:

layer two asks:

can I run experiments myself?

Layer three asks:

do I know what the experiment results mean, and whether this system can go live?

The latter matters far more than “being able to write a bit of Python.”

Layer four: ML / infrastructure depth — some AI PMs are technical PMs to begin with

Further up still are roles in core models, training, post-training, inference, compute and ML platforms.

They may require understanding:

model training;
inference;
search / recommendation;
data infrastructure;
serving;
distributed systems;
experimentation frameworks.

Here, the competition often comes from:

ML engineers, data scientists, search / recommendation PMs, infrastructure PMs.

So if you look at some OpenAI or Anthropic JDs and think:

why does AI PM already require such deep technology?

it doesn’t necessarily mean all AI PMs have become like this.

A more accurate reading is:

you’re looking at a kind of PM that was always very close to the model and the infrastructure.

A healthcare agent PM and a core models PM may both have Product Manager in their titles, but their skill structures are completely different.

That’s also why classifying AI PMs first, in the last post, matters.

Otherwise every technical requirement gets mashed into one terrifying study list.

What I really think matters more is three abilities appearing across roles

After going through these JDs, there are three things I care about more than “can you code.”

1. Evals: how do you know it really got better?

This is where I think AI PM differs most clearly from traditional product work.

In the past, we could easily say:

“user feedback is good.”

In AI products, that isn’t quite enough.

You need to keep asking:

compared with what baseline?
better on which tasks?
where did the test set come from?
which errors improved?
and where did it regress?
which errors are acceptable?
which errors must never happen even once?

So even if evals don’t become part of everyone’s job title, they’ll likely gradually become a basic working method for AI PMs.

Not everyone needs to become an eval expert.

But at least you should know:

on what basis am I saying this version is better?

2. Trade-offs: not simply chasing “a smarter model”

An AI product usually has many goals at once:

quality · latency · cost · reliability · safety

They often conflict with each other.

a stronger model may be more expensive;
a faster model may be a bit less effective;
more context may be more accurate, but latency and cost rise with it;
an agent with more permissions may complete more tasks, and also be more dangerous.

There’s no single answer an engineer can calculate here.

Product still has to decide:

what exactly are we optimizing for?

And:

where’s the floor?

3. Failure design: think first about how it will break

Ordinary software has to handle exceptions too, of course.

But the uncertainty of AI systems makes failure a more central product question.

What if it doesn’t know?
What if information conflicts?
What if a tool returns an error?
What if the model answers wrong with great confidence?
What if execution fails halfway?
When should a person take over?
Can things be recovered after an error?

I now actually think:

one of the best ways to tell whether an AI PM really understands the product isn’t to see whether they can describe the happy path, but how they talk about failure.

So does an AI PM need to code or not?

If I have to give a very short answer:

not necessarily.

But the other answer is also very clear:

not touching technology at all will increasingly limit which roles you can compete for.

Not because PMs need to become engineers.

But because AI has turned many questions that used to be hidden in engineering implementation into product questions.

which model to choose;
when it can be released;
how to evaluate quality;
how much permission the agent has;
what happens after a failure;
how to trade off quality, latency and cost.

None of these can rely only on:

“the technical team will handle it.”

I think you can test yourself with five questions

Rather than asking yourself:

“do I know Python?”

take a real AI product and see whether you can answer:

1. Why does this need AI here, rather than ordinary rules?

2. What result counts as good?

3. How do you plan to test it?

4. Where is it most likely to fail?

5. After launch, how do you know whether it keeps getting worse?

If you can already answer these independently, then even with limited coding ability, technology has started truly serving product judgment.

If you can’t, then even knowing lots of frameworks and new terms, you may still only be:

knowing about AI, rather than knowing how to build AI products.

So after this research, my understanding of “how technical an AI PM needs to be” ended up a little simpler than when I started:

the point of learning technology isn’t to replace engineers.

It’s to avoid handing over, along with it, judgments that belong to product.

The next post will look at another very interesting change:

why are roles like Evals, Harness, Model Behaviors and Forward-Deployed, rarely seen a few years ago, starting to appear in job postings?

Are they just inventing new terms, or are AI product organizations really growing new specializations?

Part 2/4 of the series “Who is hiring AI PMs right now.” The research is based on 43 public AI product roles collected as of August 20, 2026; this post mainly draws on postings for Box AI Evaluations, OpenAI API Agents / Core Models, Binance AI Agent & Harness, Scale AI Forward-Deployed and others, cross-referenced with the Stanford AI Index 2026 and PwC’s 2026 AI Jobs Barometer for trends.

Day 090 · "AI PM" is not one job

Tue, 25 Aug 2026 00:00:00 GMT

Recently I did a serious round of research on the AI PM job market.

The reason is simple: “AI PM” has become far too broad a term.

All labeled AI Product Manager, some roles work with research teams to decide model capabilities, some build agent platforms, some are dedicated to evals, and for others the core job is bringing AI into specific industries like finance, healthcare and law.

If you just ask, in general terms:

what skills does an AI PM need?

you easily get an endless study list: Python, RAG, agents, MCP, evals, ML, system design…

But in real hiring, not every role needs these equally.

So this time I didn’t start from “what should an AI PM know,” but the other way round: who are companies actually hiring right now?

As of August 20, 2026, I put together 43 publicly open AI product roles.

They include AI-native companies like OpenAI, Anthropic and Scale AI, as well as roles at Box, Binance, Apple, Ford, n8n and companies in finance, insurance and healthcare, across North America, Europe, the GCC and Asia.

This isn’t a strict whole-market statistic. The 43 roles are a stratified sample meant to cover different companies and kinds of roles, so I won’t use it to infer “what percentage of the market falls into a given category.”

What I want to answer more is:

when a company writes the job title AI Product Manager today, what kind of person is it actually looking for?

Beyond the specific JDs, I also used reports like the Stanford AI Index 2026 and PwC’s 2026 AI Jobs Barometer to look at broader hiring trends; TrueUp was only used to help look at aggregate data on PM job volume, seniority and region, and played no part in the classification below.

This round of research ended up as four posts:

this one: what different jobs “AI PM” actually covers
the second: how technical an AI PM really needs to be
the third: what new specializations like evals, harness and forward-deployed actually do
the fourth: how a traditional PM can find their place in AI

Let’s start with the first map.

I didn’t sort by job title, but by “what this PM is actually responsible for”

This was an important shift while organizing the JDs.

Sort by title, and you see lots of:

AI Product Manager, Senior AI PM, Product Manager — AI…

and can hardly conclude anything.

But switch to a different question:

what product object is this PM really responsible for?

and the differences jump out at once.

In the end I roughly grouped them into seven kinds:

model capability → AI runtime platform → quality and risk → user applications → industry workflows → enterprise AI transformation → deployment at the customer

This isn’t an official industry taxonomy, just a practical map drawn from this batch of real roles.

Layer one: closest to the model

1. Model / research PM: working directly on model capabilities

This is the kind of PM closest to the model.

What they discuss isn’t only:

“what features do users want?”

But also:

what capability the model should strengthen next; what data training and post-training need; whether a certain model behavior is one we want; when a new capability is stable enough to release.

OpenAI has a Core Models PM.

In Anthropic’s hiring you can see finer-grained ones:

Research Product Manager, Model Behaviors, Claude Code Model Performance.

These roles usually also have the highest technical bar.

Search and recommendation, ML products, data platforms, AI infrastructure, even a research or engineering background, all become clear advantages.

So when a traditional internet PM looks at a JD like this and thinks:

how did AI PM become so much like ML?

it isn’t necessarily that the whole AI PM market has turned into this.

More likely you’re just looking at the kind of AI PM closest to the model.

2. AI platform / harness PM: responsible for the systems around the model

The model itself is really just one component.

To actually get an agent running, lots of other things need solving:

APIs, SDKs, tool calling, context, memory, permissions, sandboxes, retries, fallbacks, observability…

Things that used to be easily seen as “engineering implementation details” are now forming a product layer of their own.

OpenAI has PMs for API Agents and API Infrastructure.

Binance even has, outright:

Product Manager, AI Agent & Harness

Its responsibilities include agent task flow, capability boundaries, failure fallback, harness evaluation, as well as tool calling, RAG, MCP, multi-agent setups and more.

The natural backgrounds for these roles are usually:

developer tools, APIs, platforms, infrastructure, or PMs who have long worked on complex systems.

Their product object isn’t some chat interface.

It’s:

how to let other people build and run AI reliably.

3. Evals / safety PM: responsible for defining “is it actually good”

This was one of the kinds I found most interesting in this research.

Ordinary software is fairly easy to accept:

can the button be clicked; is the amount calculated correctly; did the endpoint return.

AI is much more trouble.

The same question can get a different result each time.

After a model version upgrade, one capability may improve while another regresses.

Average performance can be very good while still making serious mistakes in a few critical scenarios.

So:

“how to evaluate AI” has itself become product work.

Box now has a role literally called Product Manager, AI Evaluations.

This PM owns evaluation datasets, graders, offline / online evaluation, model selection, regressions, and which models can go into production.

OpenAI has a Safety Measurement PM, responsible for measuring harm in real usage, and whether safeguards are actually working.

A big difference from traditional QA is:

before testing starts, someone has to answer:

what counts as good? What errors are acceptable? What must absolutely never happen?

These are themselves product judgments.

Layer two: turning AI into a product people really use

4. Applied AI PM: responsible for whether users can actually get the task done

This is closest to most people’s intuitive picture of an AI PM:

copilots, assistants, agents, AI search, recommendations…

These PMs aren’t necessarily responsible for training models.

They care more about:

whether users can really get the task done; when the model can be trusted; when a person should take over; what happens after a failure; why users would keep using it.

So beyond DAU, conversion and retention, they increasingly look at:

task success, grounding, quality, latency, cost, fallback rate.

This is also the layer where traditional PMs most easily find a path across.

But it may also be the most crowded.

Because:

“has built internet products + can prompt” is no longer a particularly scarce combination.

What really sets people apart is whether they’ve taken an unstable model capability and made it into a product that:

can be measured, can fail, can recover, and can keep iterating.

5. Vertical AI PM: responsible for how AI works once it enters a specific industry

Once AI really enters an industry, domain knowledge becomes valuable again.

A healthcare agent can’t just “sound like a doctor”;
a finance agent can’t casually execute irreversible operations;
legal products have to consider evidence, citations and responsibility;
a cyber agent has to know which tools it may call and which actions a person must confirm.

So you can already see:

Healthcare AI PM, Legal AI PM, Cyber AI PM, Insurance AI Product Owner.

These roles don’t necessarily need “the person who knows the most about models” most.

Sometimes what the company needs more is:

someone who really knows where things most easily go wrong in this industry.

And then turns that industry knowledge into:

permissions, approvals, HITL, eval cases, audit trails, risk metrics.

At this point, industry experience isn’t just a line on a résumé:

“familiar with finance / healthcare / law.”

It directly changes how the system is designed.

Layer three: really putting AI inside a company

6. AI transformation PM: responsible for how a company should actually use AI

There’s another kind of role that isn’t responsible for any single AI feature.

What they’re responsible for is:

which of the company’s dozens of AI ideas are actually worth doing;

which to buy and which to build;

which to pilot first;

how to connect them to existing systems;

how to govern data and permissions;

whether employees actually adopt them;

and whether there’s real ROI in the end.

These roles might be called:

AI Transformation PM, AI Product Owner, Internal AI PM.

Their depth in model technology may not be as high as a core model PM’s.

But they depend heavily on:

breaking down processes, enterprise systems, governance, cross-functional collaboration and change management.

So the core question for these roles isn’t:

how smart is the model?

It’s:

has this company really changed how it works because of it?

7. Forward-deployed PM: going straight to the customer to build the AI there

This is a role I hadn’t paid particular attention to before, but after this research I think it’s well worth a look.

Scale AI is currently hiring a Forward Deployed Product Manager.

The job isn’t:

write the PRD → hand it to engineering → wait for launch.

It’s going deep with the customer:

understanding the problem; connecting data and APIs; building, testing and deploying together; then gradually abstracting one-off solutions into reusable software products.

So the role is a bit like:

PM + solutions architect + consultant + builder.

Both technical ability and customer skills are needed.

It also fits AI’s current stage as an industry:

demos aren’t that hard anymore.

What’s really troublesome is often:

how to connect AI into real business.

So the same “AI PM” can mean completely different profiles of talent

Put the seven kinds together, and you find:

Model / research — how strong should the model get, and in what direction?
Platform / harness — how can AI be built and run reliably?
Evals / safety — what counts as good? When can it go live?
Applied AI — can users actually get tasks done with AI?
Vertical AI — how can AI be trustworthy and usable in this industry?
AI transformation — where and how should a company use AI?
Forward-deployed — how do you really deploy AI into a customer’s workflow?

These roles overlap, of course.

A healthcare agent PM might do applied AI, vertical AI and evals at once.

An agent platform PM might also be heavily involved in harness and evaluation.

But this map at least explains one thing:

why opening ten AI PM JDs makes it feel like they need completely different skills.

Because they’re hiring different people to begin with.

So when I look at AI PM JDs now, I first look at four things

rather than the job title.

1. What product object is this PM actually responsible for?

The model? A platform? Evals? An agent? An industry workflow? Enterprise deployment?

2. How close are they to the model and the system?

Is understanding the capability boundaries enough, or do they need to run experiments, write Python and build pipelines themselves?

3. Why does the company need this PM?

To improve model quality? To ship AI? To lower risk? To drive enterprise adoption?

4. What counts as success?

Model quality? Task success? Customer adoption? Cost? Safety? ROI?

Once these four questions are answered, two roles both called AI Product Manager may hardly be the same job anymore.

So from the first post of this research, the one simple judgment I most want to leave is:

don’t rush to ask “what does an AI PM need to know.”

First ask: which kind of AI PM am I talking about?

The next post will dig into a more practical question:

how much do these seven kinds of AI PM differ in what they require technically?

Part 1/4 of the series “Who is hiring AI PMs right now.” The research is based on 43 public AI product roles collected as of August 20, 2026, with reference to the Stanford AI Index 2026, PwC’s 2026 AI Jobs Barometer and other sources; TrueUp was used only to add aggregate information on PM job volume, seniority and region. The seven kinds are a qualitative grouping of responsibilities and don’t represent the proportion of roles across the market.