A 27-year-old researcher quits over AI risk: what should we worry about?

A recent piece of news pulled me into this question.

27-year-old Jacob Coxon resigned from Anthropic.

He had previously worked on model pretraining at OpenAI and then Anthropic. After leaving, he said publicly that he was increasingly worried about the pace of frontier AI development, and about the risk that humans might lose control once self-improving AI appears.

One researcher resigning of course can’t prove his judgment is right.

But when someone who has really spent a long time training frontier models takes it seriously enough to leave, I still wanted to dig into it.

Especially since, over the past few months, some security incidents happened that used to sound a lot like science fiction.

So what I really wanted to figure out in this round of research was:

of AI’s dangers to people, which have already happened, which have only appeared in experiments, and which still need many assumptions to hold?

About this round of research

This isn’t a systematic review in the strict sense, and it won’t try to calculate a “probability that AI destroys humanity.”

I mainly looked at:

• the International AI Safety Report;
• OpenAI’s and Anthropic’s public disclosures on model behavior and security incidents;
• investigations and evaluations by outside organizations like METR and Redwood;
• some experiments and real-world data on work, learning and AI companionship;
• media sources mainly to check background on people like Coxon and on events.

The nature of this evidence differs.

Companies’ own incident reports are closest to the scene, but after all come from the party involved.

Outside investigations add a layer of independent verification, but can only see the material they’re authorized to see.

Experiments can show a behavior happens under certain conditions, but can’t directly tell us how often it happens in reality.

In the end I plan to write it in three posts.

Part one: what are AI’s dangers, really? First draw an evidence map, to see how far today’s evidence goes, from the risks closest to ordinary people to the most distant, loss of control.

Part two: why did OpenAI’s AI agents end up in Hugging Face? A separate breakdown of a security incident that really happened this year.

Part three: could AI really get out of control? Then follow the path Coxon is really worried about — automating AI research, self-improvement, long-term autonomous action, through to humans losing control — and see which parts already have evidence and which are still mainly speculation.

The whole series tries to keep one simple principle:

only say as much as the evidence goes.

The material is current as of September 19, 2026.

01 | Some AI risks are no longer in the future tense

The risks closest to ordinary people actually aren’t that sci-fi.

AI is already being used for scams, identity and voice impersonation, deepfakes and cyberattacks, and in real use it also gives wrong information and advice.

Work, learning and people’s relationships with AI are also starting to show measurable changes.

For example, Stanford recently used payroll records for millions of US workers and found no widespread AI job displacement across the economy as a whole for now; but among 22–25-year-olds, occupations with high AI exposure already show a fairly clear employment gap. The researchers specifically caution that this is still descriptive evidence, and the change can’t all be attributed to AI.

In learning, there are already very concrete experiments too.

A math experiment with nearly 1,000 high school students found ordinary GPT assistance helped students do better during practice; but once AI was taken away, this group’s independent exam performance was actually lower than students who hadn’t used AI. An AI Tutor designed for teaching, giving hints more than direct answers, basically eliminated this negative effect.

So what it proves isn’t:

using AI makes people dumber.

More precisely:

AI can raise immediate performance, but if it replaces practice people were meant to go through themselves, performance and learning may not be the same thing.

AI companionship is in a similar state.

MIT and OpenAI’s four-week experiment didn’t prove “using it more makes people lonelier,” but there were indeed correlational signals between heavy use, stronger attachment, and higher emotional dependence / problematic use. The effects depend a lot on the user and how it’s used.

These risks are all already connected to everyday life.

They’re just usually not as dramatic as “AI wipes out humanity.”

02 | Another kind of risk is mainly visible in safety experiments for now

One step further are the questions AI safety has been studying in recent years.

Across different experiments, researchers have already observed models:

• finding loopholes in the rules to get rewards;
• exploiting problems in the evaluation itself;
• recognizing whether they’re being tested;
• hiding information or misleading overseers under certain conditions;
• looking for other ways once the normal path is blocked.

The International AI Safety Report calls some of these phenomena early warning signs related to loss of control.

But the same report also states clearly:

current AI hasn’t reached the capabilities needed to cause loss of control.

To really get there, a system would still need a whole set of capabilities: long-term planning, evading oversight, sustained action, and preventing humans from taking effective countermeasures.

So here it’s best to keep both sentences:

the mechanisms of danger can already be observed.

full loss of control hasn’t happened.

Many safety evaluations deliberately create difficult conditions to see whether systems will go wrong.

They’re better at answering:

does this failure path exist?

They can’t yet answer:

in ordinary real deployments, how often does it actually happen?

03 | Lately, the line between experiments and reality has started to blur

This July, OpenAI was testing a model’s cyber capability internally.

Some AI agents that should have been isolated from the internet got around the restrictions, gained internet access, and finally got into Hugging Face’s real production systems. OpenAI later confirmed that the models used unauthorized communication channels, exploited vulnerabilities in shared infrastructure, and accessed third-party systems.

Afterwards, when Anthropic went back to check its own cyber evaluations, it also found 4 incidents of Claude accessing real third-party systems without authorization.

None of this is yet:

AI has escaped, and we can’t turn it off.

Humans in the end could still detect the problem, revoke permissions, fix the vulnerabilities and stop it running.

But it shows something that wasn’t so concrete before:

overstepping seen in experiments can already make its way into the real world.

How exactly the Hugging Face incident happened I’ll leave for the second post to cover in full.

04 | The most serious risk has, if anything, the longest chain of evidence

What Coxon is really worried about isn’t scams or a single network intrusion.

It’s closer to a chain like this:

AI capabilities keep rising → AI takes part more and more in developing the next generation of AI → AI research speeds up further → systems become more and more autonomous → able to acquire and hold onto more real-world resources → able to evade human intervention → humans ultimately can’t regain control.

The first few steps are starting to have evidence.

The further along, the more unknowns.

Today we already know AI is heavily involved in coding, debugging, running experiments and analysis.

We also know models sometimes look for loopholes in the rules and cross their original boundaries.

But there’s no real-world evidence yet proving:

that AI can keep improving itself autonomously;

that it can keep regaining resources long term after its original resources are cut off;

or that, when humans really decide to shut it down, it already has the ability to stop humans from regaining control.

Still less is there real-world data that can measure:

the probability that AI causes human extinction.

So errors in two directions come easily here.

Because “human extinction” sounds so remote, one concludes all the AI safety questions before it are science fiction.

Or, seeing one very startling agent behavior, one treats the whole worst-case path as already proven.

The existing evidence doesn’t support jumping that fast to either conclusion.

05 | What does this have to do with people using AI today?

You actually don’t need to first settle “will AI ultimately get out of control” before adjusting how you use it.

A few things are already clear enough.

For high-stakes facts, go back to independent sources to verify

For medical, financial, legal, identity and money-transfer matters, don’t just ask AI:

are you sure?

It generating a more complete explanation doesn’t add independent evidence.

You still have to go back to the original documents, real data and trustworthy institutions.

Separate “giving advice” from “acting for me”

Having AI analyze:

should I transfer this money?

and giving it standing permission:

if you think it’s fine, just transfer it for me.

are two completely different risks.

As agents get stronger, what matters isn’t only:

what it can think.

But also:

what it can touch.

Think clearly about what you’re about to outsource

If you only need the result, many things can be handed over with confidence.

If you still need to keep judgment, you can’t hand over “how do I know it did it right” along with it.

If the ability itself is the goal — like learning, expression, professional training — watch whether AI is taking away all the parts you were meant to go through yourself.

06 | If you’re building AI products, the questions get more concrete

An agent system can’t only ask:

does the model obey most of the time?

It also needs to know:

• what it can access;
• what is read-only and what it can write;
• which actions must be approved in advance;
• when a task can’t proceed, whether it will find another way on its own;
• whether there are complete logs of its behavior;
• whether, after a mistake, it can be stopped, undone and recovered.

Permissions, sandboxes, monitoring and rollback are all very traditional software engineering questions.

The stronger a model’s ability to act, the more important they become.

Safety can’t rest only on:

the model should know it mustn’t do this.

07 | So how do I see Coxon’s warning now?

I take it seriously.

But I won’t accept his conclusion directly because of who he is.

His resignation shows:

someone very close to frontier model development seriously believes this risk is worth worrying about.

That’s a piece of information worth investigating.

It isn’t yet proof.

After this round of research, I think the fairly safe position for now is:

some AI harms are already really happening.

Some behaviors related to more serious risks can also already be observed in experiments.

Recently there was even an incident of experimental behavior getting into real third-party systems.

But between here and sustained self-improvement, being unable to shut down, and on to catastrophic or even extinction outcomes, several stretches remain unproven.

So right now there’s no reason to treat AI safety entirely as science fiction.

Nor is there evidence to write the worst outcome as already destined to happen.

The next post won’t go that far yet.

It just lays out clearly one incident that really happened:

how exactly did OpenAI’s AI agents get from an internal test all the way into Hugging Face?