Detecting AI has become a business

Publishing a post on Substack yesterday, I noticed a feature for the first time:

Scan for AI text.

I clicked it on a whim, and the result was very blunt:

AI-generated 100% · AI-assisted 0% · Human 0%

My first reaction: that absolute?

My second reaction: but I can’t really say it got it wrong.

Because looking back at how the post was made, the experience, views and judgments came from me, and what was worth writing and which conclusions didn’t hold up were things I revised round after round.

But the final English prose — the first draft — really did come largely from ChatGPT.

I provided the Chinese original, the ideas and the feedback; AI generated the English; then I kept cutting, changing and reorganizing.

So the 100% actually raised a question for me:

what exactly does an AI detector detect?

Digging in, I found something even more interesting:

as AI gets better and better at generating content, “detecting whether it’s AI-generated” has itself become a business.

AI detectors no longer just look for an “AI flavor”

Substack added AI text detection in July this year, built on a model from a company called Pangram. Readers can scan newly published posts, Notes and comments themselves to see how much of the text is Human, AI-Assisted or AI-Generated.

My impression of AI detectors was still stuck at using some statistical features of language to judge whether a piece was too regular, too predictable.

It’s much more sophisticated now.

Pangram’s latest model judges whether different parts of a text are closer to Human, AI-Assisted or AI-Generated. In training, they take real human articles and have different LLMs generate “mirror versions” close in topic, length and tone, forcing the classifier to learn differences finer than “written very neatly.”

They also specifically find human articles the model misjudged as AI and bring them back for more training.

I find that approach quite clever:

the humans who seem most like AI are exactly the ones most worth using to teach the detector.

So what we usually call “getting rid of the AI tone” isn’t quite the same thing as what machines detect.

AI tone ≠ AI provenance.

Not reading like ChatGPT doesn’t mean a machine can’t see the AI generation signal in it.

And detectors really are improving. A peer-reviewed study this year compared Pangram, GPTZero, Copyleaks and Turnitin on fully AI, hybrid and humanized AI text, and Pangram performed best overall. But the researchers also cautioned that these tools should still assist judgment, and aren’t suited to carrying high-stakes decisions alone.

So what does my “100% AI” actually mean?

This is easy to misunderstand too.

It isn’t saying:

Pangram has 100% mathematical certainty that this post was written by AI.

It’s closer to: the text in this post was judged, overall, to have AI-generated provenance.

On my post, the distinction is especially clear.

It can see the final English sentences, but doesn’t know why I wanted to write about this, whose experience it was, who proposed the views; nor which suggestions I rejected, or which parts went back and forth through many rounds.

So detectors are fairly good at answering:

how were these words most likely produced?

They can’t answer:

whose ideas are these?

These two questions used to overlap a lot. A person thought it, and the same person wrote it.

Once people and AI start creating together, they slowly come apart.

So saying my post was entirely human-written isn’t accurate; but if someone said it was “100% created by AI,” I wouldn’t accept that either.

Closer to the truth is:

the thinking and judgment came mainly from a person; the generation of the English prose relied heavily on AI.

What the detector sees is mainly the latter.

Why would anyone actually pay for something like this?

This is the part I found most interesting while looking into Pangram.

If an AI detector only satisfied:

“let me see whether ChatGPT wrote this,”

it would be hard to imagine much commercial value in it.

The really valuable situations are where how content was produced affects how we use it.

Take student essays.

The teacher isn’t short of those 2,000 words. They want the assignment to show whether the student understood the question, did the research, and can organize and express it.

An assignment used to serve as some evidence of those abilities.

Since AI arrived, the link between the work and the ability isn’t so direct anymore.

Hiring is the same. Cover letters, writing samples and take-home assignments were all used to understand a candidate’s expression and thinking. Now, seeing polished material, the hiring side needs to ask one more question: how much of this person’s ability does it still represent?

Content platforms face another problem.

When I subscribe to a writer, I usually don’t just want a few hundred words of information; I’m following that person’s experience, judgment and way of observing.

As mass-producing good writing gets cheaper and cheaper, readers naturally start to care:

what does what I’m seeing actually represent about this person?

These aren’t hypothetical use cases.

Pangram already sells its product to schools and universities, and offers enterprise products and a developer API. Official API uses include making the detection result a layer in product decisions, a signal for moderation queues, or bulk-scanning historical content; Quora uses it to handle AI-generated answers.

It has also become a clearly commercialized product: an individual subscription at $20 a month, Professional at $65 a month, plus a usage-based API, education licenses and enterprise plans.

A more direct market signal: in July this year Pangram raised $9 million, led by Menlo Ventures. TechCrunch’s report mentioned that its API customers already include Substack, Quora, schools and universities, publishers, agencies and recruiting teams.

Funding can’t prove how big this market will ultimately be, of course.

But it at least shows this is no longer just a small tool for “academia worried about students cheating”:

platforms, institutions, companies and investors have all started to think that the authenticity and verification problems brought by AI-generated content are worth paying to solve.

AI has weakened some signals we used to take for granted

These use cases actually have something in common.

Before, seeing an essay, we assumed someone had done the research and the writing.

Seeing a writing sample, we assumed it more or less represented the person’s expression.

Seeing a writer publish steadily, we assumed we were coming into contact with that person’s thinking.

These were all implicit information that content used to carry.

AI hasn’t made content disappear.

It has just made the relationship between content and the person behind it less certain.

So new signals are needed.

Detection is one.

Authors explaining how AI was involved is disclosure.

Going further, if you can reliably record where a piece of content came from and what processing it went through, that’s provenance.

They all fill the same gap:

when the final result can no longer fully explain how it was produced, what other information do we need to correctly understand what we’re seeing?

Of course detectors can’t solve the whole problem.

Especially as hybrid writing becomes more and more common, simply splitting things into “used AI / didn’t use AI” is already a bit crude.

So I’d rather understand detection results as a signal that needs further explanation, not a verdict.

What interests me most is the “reverse market” AI creates

This is what I found most interesting in this research.

When we talk about AI startups and AI product opportunities now, we’re used to following model capabilities:

• what can AI do now?
• which work can finally be automated?
• where can things be cheaper and faster?

That’s the most direct layer of opportunity, of course.

But once a technology really enters the world at scale, something else happens too:

it breaks some old mechanisms, while creating new costs, risks and uncertainty about information.

So a second-layer market follows.

• Once email spread, spam became extremely cheap, so anti-spam became infrastructure.
• The more the internet and online transactions mattered, the more valuable attacks, fraud and identity impersonation became, so cybersecurity, fraud detection and identity verification kept growing.
• Generative AI is now on the same path.

AI can produce huge amounts of content at low cost, so detection, provenance and content authenticity start to have value.

AI agents can take actions for people, so permissions, guardrails, audit trails and observability become more important.

LLMs can give answers that seem more and more real, so grounding, evaluation and source verification become important along with them.

AI coding can greatly speed up code production, so naturally someone will need to solve the testing, security, quality and maintenance of generated code.

None of these products seem to be “making AI stronger.”

Some even specialize in checking it, limiting it, putting boundaries on it.

But they don’t contradict AI’s growth.

It’s precisely because the capabilities up front keep getting stronger that the demand behind them keeps getting more real.

I later felt that “reverse market,” while vivid, isn’t strictly against AI either.

It’s more like a second-order market that arises once AI spreads:

the first-layer market creates new capabilities.

The second-layer market handles the new problems that come once those capabilities spread.

This angle is quite useful for how I look at AI product opportunities myself.

From now on, when I see a new AI capability, besides asking:

what can it do for people now?

I might also go on to ask:

• what has it made extremely cheap?
• which judgments, rules or trust mechanisms no longer work well because of it?
• where will new fraud, abuse, error or uncertainty appear?
• who ultimately bears the cost of these problems, and who would be willing to pay to solve them?

Pangram is a very concrete example.

Generative AI makes text cheaper and cheaper, more and more human-like.

So “how did this text actually come about” — an answer almost nobody used to need to buy — has started to have a price too.

AI’s opportunities aren’t all in getting AI to do more.

Some opportunities appear after AI is doing more and more.