Why on earth would we rush to build something cleverer than us?

The people building the world’s most powerful AI systems are starting to ask whether we are moving too quickly. I think we should listen. I also think we need to be very careful about which AI we’re talking about.

This morning, just after 7.00, I found myself on BBC Radio Somerset with Charlie Taylor being asked a fairly cheerful breakfast-time question: how concerned should we be that AI could kill us all?

It’s been quite a week for AI headlines. Anthropic co-founder Jack Clark has been talking about mandatory kill switches for highly capable AI systems, and Anthropic CEO Dario Amodei has called for the industry to slow the rate at which the most powerful models are developed. Sam Altman at OpenAI agrees that the frontier needs pacing, Elon Musk responded to Amodei’s essay with the wonderfully concise “Dario is right”, and Google DeepMind’s Demis Hassabis says the proposal points in the right direction.

On the other side, President Trump has called fears about AI taking over humanity a “hoax”, NVIDIA CEO Jensen Huang says some of the more extreme predictions aren’t grounded in science, and China has accused Amodei of fearmongering, arguing that competition between nations risks getting in the way of sensible global governance. So yes, things have got a little spicy.

When I was asked what I thought on the radio, one thought came out more plainly than I’d probably have written it beforehand:

“Why on earth would we rush to create something that was cleverer than us?"

And actually, I think that’s the question at the centre of all this.

First, we need to understand what “frontier AI” means

This entire conversation becomes pretty unhelpful if every use of the word “AI” gets thrown into the same bucket. The systems causing concern here sit at the frontier: the models being built at the very edge of current capability, with growing abilities to reason, use computers, write software, conduct research, take actions over long periods and, increasingly, contribute towards the development of future AI systems.

Dario Amodei’s essay is literally called We Must Pace the Frontier. His concern is that since the summer, the capabilities of these systems have been growing much faster, particularly in their ability to help build the next generation of AI. As he puts it:

“We must slow the pace at which we improve the capabilities of AI models.”

His proposal includes permanent third-party evaluators inside frontier AI companies, with access comparable to employees, so they can independently examine safety practices and investigate serious incidents. Anthropic has already committed to doing that, and OpenAI has said it will do the same.

That’s a very particular world. A handful of companies with huge datacentres, enormous amounts of computing power, specialist chips and billions of pounds, plus teams of some of the brightest researchers on the planet trying to find out just how capable these things can become. Meanwhile, somebody in Somerset asking Copilot to tidy their meeting notes is operating at a completely different end of the spectrum, and the public conversation needs enough room for both of those realities.

We’ve already got some very good AI

During the interview I described today’s AI as a kind of calculator for information, and I quite like that analogy. We stopped thinking that using a calculator for maths was somehow cheating many years ago; it simply became another tool available to us. AI gives us something similar across language and information. I can give it 100 pages of documents and ask it to help me find connections, or use it to summarise a meeting, compare information, explain something unfamiliar, organise my thoughts, create learning materials or take some of the repetitive bureaucracy out of a working day.

I sometimes describe that bureaucracy as digital debt. It’s all the stuff that has accumulated around work because computers allowed us to create more data, more documents, more forms, more emails, more reports, more meetings and more boxes to tick. We’re drowning in information that technology helped us create, and now we’re developing technology that can help us deal with it.

People can make different choices about what they do with that capability. You can use AI as a shortcut and produce something quickly, or you can use it to do more work. Or you can use the time it gives you back to think harder, improve the quality of what you’re doing and spend more time on the parts that genuinely need a human being. That last one is where most of my interest sits.

Then there are examples that make writing emails look rather trivial

The NHS is already using AI-assisted imaging to help identify lung cancer. Figures published by the Department of Health and Social Care in June said more than 4 million patients had received a faster lung cancer diagnosis or all-clear thanks to AI tools. The system gives radiologists another set of eyes, and complex scan analysis that had been taking around 8 days was taking around 4.

Think about those 4 days from the patient’s side. You’re sitting at home wondering whether somebody is about to tell you that you have cancer. 4 days matters.

Then there’s AlphaFold. Google DeepMind used AI to tackle a problem scientists had wrestled with for about 50 years, predicting how proteins fold into their 3D structures, and Demis Hassabis and John Jumper received the 2024 Nobel Prize in Chemistry for that work on AlphaFold2. The technology has since been used across pharmaceutical research, biology and environmental science.

And just last week, DeepMind published AlphaGenome Atlas, which contains predictions for the effects of around 9 billion possible single-letter changes in human DNA. Testing 9 billion mutations individually in a laboratory is practically impossible, but AI can help researchers rank which variations deserve attention, including in work on rare diseases and understanding genetic changes associated with disease.

That’s AI too, which is why I get uncomfortable when the conversation collapses into simply “AI is dangerous”. We need more precision than that.

The engine and the brakes

The analogy I ended up using on the radio was a car. We’re building an increasingly enormous engine, and at some point somebody needs to be thinking seriously about the brakes. That seems pretty sensible to me.

If you told me you’d built a car capable of travelling at 1,000 mph, my first question probably wouldn’t be how quickly we could get it onto the A303. I’d quite like to know how we’re stopping it, and I’d want somebody other than the person who built the thing to check whether those brakes actually work.

This is why Anthropic’s idea of embedded independent evaluators makes sense. The companies building these systems have their own safety teams and their own rules, but they’re still the companies building the systems, and independent scrutiny matters once the consequences become large enough. We do this everywhere else. We inspect aeroplanes, regulate medicines, inspect nuclear facilities and certify electrical equipment, because increasing capability brings increasing responsibility. AI should be no different.

Then we run into the space race problem

Charlie described it this morning as feeling a little like the space race, and that’s exactly where things become difficult.

Imagine OpenAI, Anthropic and Google all agree tomorrow that they’ll spend 2 years moving extremely carefully. Lovely. But what does China do? What does another country, or another well-funded company, do? And what happens when somebody decides the prize for getting there first is simply too big to pass up?

China has already pushed back hard against part of Amodei’s argument. Its Foreign Ministry spokesperson Guo Jiakun said:

“Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance.”

There is an obvious reason China is suspicious. Amodei argues for international cooperation on safety while also arguing for America to maintain restrictions on China’s access to the most advanced AI chips, and from Beijing’s perspective those two ideas are difficult to separate from American attempts to hold onto its lead. China itself has spoken about AI governance and keeping advanced AI under human control, so the disagreement sits partly around who gets to set the rules and whether safety becomes another weapon in the competition between nations. That’s why some form of international agreement eventually seems unavoidable to me.

A Geneva Convention for AI?

I used this idea before going on air, although I’ve since decided there’s probably a better comparison in nuclear regulation.

The interesting thing about frontier AI is that it needs physical infrastructure. Training the very largest models takes enormous amounts of computing, electricity, specialist chips and datacentre capacity. You can hide some lines of computer code fairly easily, but hiding a gigantic datacentre consuming vast amounts of power is rather harder, and that creates potential points of control.

You could imagine an international regime where training beyond an agreed level of computing power has to be registered, and models crossing certain capability thresholds require independent testing. Serious safety incidents could carry mandatory reporting rules, large compute clusters could be auditable, advanced chip movements could form part of the monitoring, and governments could impose consequences on organisations or countries that refuse to take part.

None of this is easy. International agreements depend on verification and trust between countries that frequently have very little trust to spare. Still, we have done versions of this before, because the alternative was allowing increasingly powerful technologies to develop without common rules, and AI may eventually need something similar.

Jensen Huang raises a fair challenge

Jensen Huang sits in an interesting position, because NVIDIA makes much of the hardware underneath this boom, and his response to this week’s warnings has been noticeably more sceptical. He has said AI safety is “paramount” while questioning some of the claims around human extinction, arguing that the more extreme predictions are not grounded in science and that America can keep making progress safely.

I think that challenge deserves hearing too. We shouldn’t turn predictions into facts simply because the consequences sound frightening. Some of the scenarios being discussed are genuinely uncertain and involve capabilities we haven’t built yet; some researchers think they’re plausible, while others think the probability is being dramatically overstated. Science improves through disagreement.

What I find far more persuasive than speculative percentages about human extinction are the concrete behaviours researchers are actually observing and testing. Can these systems conduct serious cyber operations? Can they deceive people during evaluations, or operate autonomously for long periods? Can they materially contribute towards developing their successors, and can human beings reliably intervene when they behave unexpectedly? Those are questions we can actually test, so let’s test the hell out of them.

“I think people might have the potential to cause harm”

Towards the end of the interview I was asked directly whether AI has the potential to cause harm, and my immediate response surprised me slightly:

“I think people might have the potential to cause harm."

The more I think about it, the more I stand by it. There is risk in the technology itself, and there is also risk in the race around it. Companies want to win, countries want to win, investors have placed staggering amounts of money behind the expectation that capability keeps increasing, and researchers want to discover what comes next. Human beings also have a pretty impressive history of asking whether we can do something before spending quite enough time asking whether we should.

That’s where I see the immediate danger. A powerful technology combined with enormous resources, international competition and incentives to get there first deserves proper oversight. Somebody needs to inspect the brakes.

We can use the technology and still ask hard questions about where it goes next

I spend most of my professional life encouraging people to get curious about AI, and I’m going to keep doing that, because I’ve seen what happens when someone who’s been scared of this stuff suddenly realises they can talk to it.

I’ve watched people use AI to understand complex information that previously shut them out. I’ve seen businesses remove hours of pointless administration, educators create better learning materials, and people use these tools as another pair of eyes, another way of thinking something through and another route into knowledge. Beyond the work I do personally, we’re watching AI assist doctors, scientists and researchers with problems that actually matter to human lives. That deserves excitement, and the development of vastly more capable systems deserves care. We can manage both.

I keep coming back to that car. Build the engine, test the brakes properly, let independent people check that they work and agree some rules about where we’re allowed to drive the thing. And please, before somebody puts their foot flat to the floor, make sure we know how to stop.