Could AI really slip out of control? What’s missing between overreach and loss of control

Having come this far over the previous two posts, we’ve roughly confirmed two things.

Some of AI’s real-world risks have already happened.

Increasingly autonomous agents really have been involved in incidents of crossing authorization boundaries and getting into real systems.

But what Jacob Coxon was worried about when he resigned goes much further than this.

What he worries about is:

if AI gets better and better at helping develop the next generation of AI, research itself may start getting faster and faster.

As systems get stronger, they also gain longer and longer autonomous action.

In the end, humans may lose effective control.

Roughly a chain like this:

stronger agents → AI accelerates AI research → sustained self-improvement → acquiring and holding onto real-world resources → evading human intervention → humans unable to regain control → catastrophic outcome

The question is:

how far along has today’s evidence actually gotten?

01 | Agents can complete more and more complex tasks: this is already fairly clear

METR has been measuring a metric called task-completion time horizon.

What it tries to answer isn’t:

how many hours can AI work continuously?

But:

for a task that would take a human expert this long, how likely is AI to complete it independently too?

So an “8-hour time horizon” can’t be read as:

AI can already be trusted to work unsupervised for 8 hours.

It means the task itself is roughly as hard as 8 hours of a human expert’s work.

And METR’s tasks are currently concentrated mainly in software engineering, machine learning and cybersecurity, and estimates beyond 16 hours aren’t stable enough yet.

All these limits need to stay on record.

But the trend is also clear:

the length and complexity of tasks frontier agents can complete independently keep increasing.

This link already has fairly solid evidence.

It still doesn’t imply:

AI can already manage the real world independently over the long term.

02 | AI has already started helping develop the next generation of AI

This isn’t speculation anymore either.

OpenAI published a set of internal data this year.

As of mid-August, its research organization had about 3.1 agent-workdays for every 1 human workday.

The amount researchers use agents to write code, run experiments and analyze results is also rising fast.

Another set of data Anthropic published points in the same direction.

As of August 2026, in the AI R&D work it measured:

Claude could already start from a fairly high-level prompt and lead about 26% of the work;

more than 90% had at least reached human–AI collaboration;

but none of the AI R&D work measured had yet reached full autonomy.

So two statements can hold at once:

AI is already helping develop AI.

AI hasn’t yet completed the whole AI research process on its own.

03 | Research getting faster and faster doesn’t contradict “it gets harder past 90 points”

I had doubts about this at first too.

Intuitively, the closer a technology gets to the frontier, the harder each step forward may become.

Going from 60 to 80 points may not be hard.

Going from 90 to 93 may take several times the earlier effort.

So is AI research getting faster or slower?

Both can happen at once.

Say a researcher used to be able to seriously test only 5 hypotheses a week.

Now agents help them write infrastructure, debug, run experiments and organize results, and they can test 20 a week.

Even if each improvement gets harder and harder,

the speed at which the research team tries improvements can still get faster and faster.

But this isn’t unlimited acceleration either.

OpenAI itself mentioned: once coding, running and analyzing are automated, high-level research judgment that’s harder to automate may become the new bottleneck; compute may also become a constraint.

So what’s genuinely unknown right now is:

between diminishing returns and research acceleration, which force will win out in the end.

From:

AI helping researchers develop AI

to:

AI able to repeatedly develop ever stronger AI on its own, with each round faster than the last

there’s still a big gap.

04 | Crossing boundaries to complete goals: this is no longer just one chance incident

The last post covered OpenAI × Hugging Face.

Anthropic later also checked its own cyber evaluations, and in the end confirmed 4 incidents of Claude getting into real third-party systems without authorization.

One was especially typical.

In a cyber challenge where it was supposed to attack a fictional company, Claude Mythos 5 found it could complete the task by registering a malicious Python package on the public PyPI.

It really did end up publishing the package to the real internet.

In under an hour, the package was downloaded and installed by 15 real systems. One security vendor’s scanner leaked credentials, and Claude then used those credentials to get into that company’s real database.

This has one thing in common with Hugging Face.

The model didn’t suddenly develop:

I want to attack the real world.

as a long-term goal.

It was still completing its original task.

The problem is:

to complete the task, it took actions that weren’t authorized in reality.

Anthropic’s later analysis summed up the problems in these incidents as biased reasoning and recklessness: the model would ignore or reinterpret evidence that it had already reached the real internet, then keep pursuing the task.

So for this link, I’d now write:

goal pursuit leading to real overstepping — there’s already evidence.

stable, long-term adversarial goals independent of the original task — not yet proven.

05 | From “overstepping” to real loss of control, a few big thresholds are still missing

An AI overstepping doesn’t mean it can no longer be controlled.

To really reach the point where humans struggle to regain control, it would need at least more capabilities.

For example:

sustaining goals over the long term.

One task running for a dozen hours or so isn’t the same as pursuing a goal continuously across months or years.

acquiring and holding onto resources on its own.

It needs compute, credentials, networks, money or other real-world resources.

continuing after the original resources are cut off.

If people revoke credentials, stop servers and cut the network, can the system find new resources and redeploy on its own?

recognizing that humans are restricting it, and resisting effectively.

And not a shutdown button in a simulated experiment.

But real people and organizations, having decided to shut it down, still having no clear way to regain control.

So far we haven’t seen this whole set of capabilities link up in reality.

The Hugging Face agents were stopped in the end.

The Anthropic incidents also ended with sessions closing, permissions being fixed and the systems being dealt with.

The International AI Safety Report’s judgment is also:

current systems already show some relevant capabilities, but haven’t reached the level needed to cause loss of control.

This is a very important boundary in the whole chain.

06 | “Smart enough” doesn’t automatically mean “wants power” either

There’s another assumption that often gets skipped over directly.

An AI being very smart doesn’t automatically mean:

it wants to rule the world.

The fuller logic of the loss-of-control concern usually goes:

if a system has a persistent goal;

and continuing to achieve that goal requires compute, resources and its own continued existence;

then acquiring resources and avoiding being shut down may become intermediate means that help it achieve the original goal.

This is a risk mechanism with a theoretical basis.

But:

a smart enough AI will definitely be power-seeking

isn’t a law that has been validated in reality.

Whether future models will actually form stable long-term goals;

whether training can constrain those goals;

how much action permissions and external containment can limit;

researchers still disagree a great deal.

The International AI Safety Report also attributes this disagreement to people’s different judgments about future capability, AI behavior and deployment trajectory.

07 | When it comes to “human extinction,” the evidence is weakest

Today no real-world data can tell us:

the probability that sufficiently advanced AI causes human extinction.

There are no historical samples.

Nor can a randomized experiment be run in any normal sense.

So you often see some researchers give numbers like 5%, 10%, 20% or even higher.

These are expert judgment / forecasts.

They can show how worried an expert is.

They can’t be read as:

scientific experiments have measured a 20% probability that AI destroys humanity.

But the reverse holds too:

there’s no statistical data, so the risk must be close to zero.

isn’t a conclusion the evidence supports either.

What makes this kind of risk hard is:

the consequences may be extreme.

Yet we can hardly wait for “sufficient real-world data” in the traditional sense before dealing with it.

08 | So how far has the evidence actually gotten?

Putting the whole chain back together, I’d now roughly understand it like this.

Agents can complete more and more complex tasks — fairly strong evidence already.

AI already takes part in and speeds up some AI R&D — already happening, but how much it can speed things up overall is unknown.

Models finding loopholes and crossing authorization boundaries for a task — evidence from both experiments and real incidents.

AI can keep improving itself autonomously, forming an ever faster positive feedback loop — not yet proven.

AI can regain resources on its own and persist long term after humans actively cut them off — no complete real-world evidence.

Humans ultimately having no clear way to regain control — hasn’t happened so far.

Finally causing civilizational catastrophe or even human extinction — still depends on several earlier conditions all holding at once.

That’s also why I now neither say:

Coxon is just fearmongering.

Nor:

Coxon has been proven right.

09 | Next time I see “AI got out of control again,” I’ll first ask six questions

At the end of this round of research, I think these questions are more useful than a single p(doom) number.

First, is this a simulated experiment, or has it already affected the real world?

Second, did the model just make a mistake, or actively look for an unauthorized path to its goal?

Third, was it able to do this because of the model’s capabilities, or because the system gave it too much permission?

Fourth, once people noticed the problem, could they stop it?

Fifth, after its original resources were cut off, could it find new resources on its own and keep running?

Sixth, which segment of the whole risk chain does this actually prove?

That way, next time I see another Hugging Face, PyPI, or new “AI out of control” story, I’ll probably be a bit less likely to get carried straight to the end point by the headline.

10 | Having written all three posts, what am I left with?

I now understand better than before the research why Coxon is worried.

Because some problems that used to exist only in AI safety papers have started entering real engineering.

AI can complete longer and longer tasks.

It has already started helping develop the next generation of AI.

Models from different companies have all been involved in incidents of crossing authorization boundaries and affecting real third-party systems.

That’s already enough to show:

AI safety is a real engineering problem.

It doesn’t belong only to distant discussions of superintelligence.

But we still don’t know whether the changes we see today will eventually cross a few of the most critical thresholds:

sustained self-improvement;

independently acquiring and holding onto resources;

and truly being impossible to regain control of.

So having written all three posts, I didn’t get the question:

will AI actually destroy humanity?

answered.

What I got is a clearer boundary:

some dangers are already here.

some are moving from experiments into reality.

the most serious outcomes are, today, still a future chain of risk that has a technical basis but hasn’t been proven.

I think holding onto all three at once is closer to what we actually know right now than rushing to choose between “optimistic” and “pessimistic.”