The People Building AI Are Afraid of What Comes Next — But They Can't Stop

The People Building AI Are Starting to Ask: What If We Lose Control?

The most unsettling AI warnings right now aren't coming only from science-fiction writers. Some are coming from the people who actually build these systems.

For years, the AI industry had a remarkably simple objective: build a better model. Make it reason better, make it code better, give it more context, more tools and more computing power.

But something has changed.

Some researchers working on frontier AI are now publicly questioning whether the race toward increasingly capable systems is moving faster than our ability to make those systems safe.

A Researcher Walked Away

In September 2026, Jacob Coxon, who said he had spent three years researching at both OpenAI and Anthropic, resigned from Anthropic and publicly warned about the direction of frontier AI development.

His concern was not that today's chatbots are secretly taking over the world. It was what could happen if companies successfully build increasingly autonomous and self-improving systems without first solving the problem of controlling them.

Coxon argued that the industry is racing toward what he described as self-improving superintelligence and warned about the possibility of catastrophic consequences.

What makes his resignation notable is that he wasn't an outsider criticizing AI from a distance.

He had been working on the technology.

Then Other Researchers Spoke Up

Coxon's warning was followed by another striking statement from Evan Hubinger, an alignment researcher at Anthropic.

Hubinger publicly said he personally estimated the possibility of AI causing human extinction within the next decade at more than 10 percent.

That is his personal estimate, not an established scientific probability or consensus prediction. Other researchers strongly disagree about both the likelihood and the timeline of such scenarios.

But the fact that researchers working directly on AI alignment are openly debating these possibilities tells us something about how seriously the question is being considered inside the field.

What Are They Actually Afraid Of?

The concern is often reduced to a simple science-fiction scenario:

“AI will become conscious and decide to destroy humanity.”

That's not the central technical problem.

A more serious question is whether a sufficiently capable system could pursue a goal in ways that humans did not anticipate.

Imagine giving an AI system an objective. Then give it the ability to write and execute software, access computers, use the internet, call external services and create plans that it can execute by itself.

Now imagine that system becoming substantially more capable than the people who created it.

The problem isn't necessarily that it hates humans. A system doesn't need emotions or malicious intentions to create a dangerous outcome.

The problem could simply be that its objectives and human interests stop lining up.

From Chatbots to Agents

This is why the rise of AI agents matters.

A chatbot primarily produces an answer. An agent can potentially take action.

It can inspect files, write code, execute commands, interact with websites, call APIs and continue working toward a goal without asking a human about every individual step.

That makes AI dramatically more useful. It also creates a much harder safety problem.

The more capable an AI becomes, the less useful it is to ask only whether its answer is correct. We also have to ask what it is capable of doing.

The Race Nobody Wants to Lose

This is where the AI story becomes much more complicated.

It is easy to look at researchers warning about increasingly powerful AI and ask a simple question: “If they are so worried, why don't they just stop?”

Because stopping alone could create another problem.

Imagine two companies developing increasingly capable AI systems. Both believe that the technology could eventually create serious risks. Both would benefit from stronger safety testing. Both might even prefer the entire industry to slow down.

But if Company A slows down while Company B continues, Company A may lose the race.

The other company could develop a more capable system first, gain a technological advantage, attract more investment and potentially shape the standards everyone else has to follow.

So the incentive becomes strangely circular: everyone may have reasons to slow down, but nobody wants to be the first to do it.

This competitive pressure is one reason researchers and industry leaders have increasingly discussed coordination rather than simply asking one company to stop developing frontier systems.

Fear and Competition Can Exist at the Same Time

That creates an uncomfortable contradiction inside the AI industry.

The people building the technology can genuinely believe that future systems could become difficult to control while simultaneously believing that their own company cannot afford to stop developing them.

Those two beliefs aren't necessarily contradictory.

A researcher might think, “I am worried about where this race is going.”

And at the same time think, “If we stop, someone else will build it first.”

That second thought is powerful because the competition isn't limited to companies. AI development has economic and geopolitical consequences, and no major laboratory wants to voluntarily surrender a technological advantage that another laboratory could immediately pursue.

That is why proposals for slowing frontier development generally involve coordination, shared safety standards, testing and independent evaluation rather than expecting one company to simply abandon the race.

The Paradox of the AI Race

The industry may be accelerating partly because everyone is afraid of being left behind.

And some of the same people are warning that the acceleration itself could create risks that nobody knows how to control.

It creates a loop:

Fear of falling behind → faster development → more capable systems → greater safety concerns → calls for caution → fear of falling behind again.

Breaking that loop is considerably harder than telling one company to slow down.

It would require enough companies, researchers and governments to believe that cooperation is safer than winning the race alone.

And that may be one of the hardest parts of the entire AI safety problem.

And That's the Real Problem

Perhaps the most interesting part of this story isn't whether AI will eventually surpass humans.

Nobody knows.

The more immediate question is whether we are building the safety mechanisms at the same speed as we are building the technology.

AI capability is advancing rapidly. But alignment research, independent evaluation, regulation, monitoring and international coordination are considerably harder problems.

And unlike a benchmark score, there is no simple number that tells us:

“This system is now safe enough.”

The Strange Irony of AI

For decades, humans imagined creating machines smarter than ourselves.

Now some of the people closest to that goal are asking whether intelligence without reliable control could become a problem of its own.

That doesn't mean AI is about to take over. It doesn't mean extinction is inevitable. And it doesn't mean every warning from an AI researcher should be treated as fact.

But the conversation has clearly changed.

The question is no longer simply:

“How intelligent can we make AI?”

It is increasingly:

“How capable can we make AI while still knowing how to control it?”

And perhaps the most uncomfortable part is the incentive structure surrounding the people trying to answer that question.

Some researchers fear what increasingly capable AI could eventually become.

Some have left because they no longer want to participate in the race.

Others remain and argue for stronger safety measures from inside the industry.

Meanwhile, the major laboratories continue competing because nobody wants to be the company that slows down while someone else builds the next breakthrough first.

They may be afraid of the future—and still feel that they cannot afford to stop building it.

That may be the real AI race.