If They Pace, We Lose.

Open source, not super intelligence is what the frontier labs fear.

While everyone and their mother from one day to another, have become worried about the lack of AI superintelligence alignment, there seems to be a very strong alignment between the major frontier labs, political leaders, and many research academics, all pushing for an agreement to slow down the pace of AI research or what they call pacing progress. Even Elon has joined the choir.

The main argument, probably best represented by the CEO of Anthropic's, Dario Amodei, goes something like this: AI is increasingly being used to build new models, which in turn speeds up the capabilities of those models to something way smarter than humans.

If left unchecked, misaligned autonomous agents can outrun human control. According to Dario, within the next 6 to 12 months, misaligned "swarms" of models could, (notice that word "could") become capable of taking over the internet via persistent botnets, executing devastating cyberattacks, or being misused for bioterrorism. His solution?

Embedded Third Party Evaluators: AI companies must give external safety teams, like METR, ongoing employee level access to verify safety commitments, evaluate alignment, and monitor pipelines during training.

Democratic Coordination: Leading AI labs operating within democratic countries must work together, with government backing, to establish common safety standards and enforceable speed limits on capability gains.

Global Coordination: The United States and its democratic allies must eventually coordinate with authoritarian governments, like China, to ensure global compliance, using strict verification infrastructure to prevent an unsafe underground arms race.

Sounds reasonable, right? I mean, who could be against just having a regulatory framework now that human extinction is on the table? In fact I am not even against some notion of check and balances done by the industry. This however is much more far reaching and will not solve the fundamental problem.

Predicting the future

Before we answer why one could be against what Dario is proposing, let's first do what we should always do when we try to evaluate the claims of people who are trying to predict the future. Let's look at their track record of their past.

Well, as it turns out, Dario was the person responsible for delaying GPT 2 out of fear that it would destroy society. For those who don't know, GPT 2 was a pre ChatGPT model eventually released in 2019.

Dario was also the person who claimed that by now most developers would not be needed.

And let's not forget Fable 5, which was claimed to be so dangerous that it could break into the most secure institutions, yet was totally fine to give to everyone else but the US military because they wanted it stripped of the safety features, the same features which were jailbroken in a day after the release by the internet. We are still here.

That should tell you a little about Dario's track record of predicting risk.

From my perspective, if anything, software has gotten way better and more secure because of AI. More people than ever can now utilize code via AI agents and improve software that's been neglected for decades.

It turns out the more you actually start applying AI to the real world, the less worried you seem to become, despite AI researchers concerns, over and over that this time it could potentially kill us all.

Dario is, of course, not the only one with worries.

Elon got involved in AI after he had read Nick Bostrom's book Superintelligence, a book that made one of the original cases for how AI will gradually outcompete us and eventually destroy us. He has provided several doomer perspectives on this, but he has also been wrong about his scenarios, and he is certainly not slowing down either so he can't be too worried.

Even Sam Altman is no exception and has several times been pushing all sorts of doomer scenarios, although he is a little more careful these days.

All of them have been wrong several time, yet their conviction seems to be going up, completely counter to what those who use AI, experience when they actually use these models.

None of them, of course, have been slowing down or will be, regardless of the strong rhetoric and call for regulatory frameworks and pacing.

Everything old is new again

So what's going on here? Why would anyone who is leading the AI race suddenly give up their lead and hand it over to politicians? Well one can only speculate but since the entire premise of world destroying superintelligence is speculation, let me offer my personal take on things. The frontier labs know involving the government will create regulation that stifles and kills almost any progress for most industries? Asking for regulation simply makes very little sense sense. Unless of course you start looking at this from another angle: regulatory capture.

One of the most successful strategies for staying in business is not necessarily to compete head to head with others. You will always have to balance between not spending too little and seeing the competition run with the market, versus spending too much and running out of capital.

AI frontier labs are especially sensitive to this because of the huge investments they have to make each time they train a new model.

Politicians are also breathing down their necks, especially because of the anti data center mood among general voters.

And last, but perhaps most importantly, at the same time Chinese open source models and the global open source community in general are increasingly producing good enough models that can run locally for more and more scenarios.

While these models will never be able to keep up with the absolute latest frontier model, they will be much cheaper alternatives to the current, and mind you, heavily subsidized models, and can be run locally. Lets not forget the capabilities of todays open models were yesterdays closed frontier models, in other words the local models will become better too.

The second the subsidization of the frontier labs goes away, the price increase will be substantial, and this will be a perfect incentive to use open source models instead.

So what do you if you are frontier labs and even have had papers out saying you have no moat?

You try to get ahead of it all, and you create something like the Nuclear Regulatory Commission, or NRC, which will oversee the progress of models and force anyone who wants to do anything with anything close to frontier models, their bread and butter, to go through the expensive regulatory hurdles.

For those who don't know, the NRC regulated all right, and completely killed civil nuclear progress, while the US military continued to build out enough nuclear bombs to kill us all several times over.

You don't need to have studied advanced game theory to understand that neither the US nor China are going to slow down, and thus they will still be asking the frontier labs to continue working on these allegedly civilization killing intelligences, just like they did with nuclear bombs.

If they pace, we loose

So if that is true, where does it leave the rest of us?

Well, we will be stuck in a regulatory and thus legal minefield that will make a lot of civil initiatives close to impossible, make it very hard and often illegal to use open models and weights, including many Chinese models, and kill big parts of the open source community's momentum, which is currently exploring things like model distillation, which is a direct target of the proposed regulatory framework.

It will also make it impossible and legally risky for people to build models that are optimized for specific areas.

Hindering dual-use sounds fine when we're talking about not wanting terrorists to use models for bioterrorism by quantizing and remove the guardrails, but how do you differentiate between what is being used for normal research vs what is being used for bio-terrorism? You cant and so we end where this often ends. "Better safe than sorry" stances forcing companies to choose the safe but often sub-optimal choice, which are the models owned by the labs that can afford to pay the high upstream compliance cost.

Regulatory capture at its finest.

None of that is how it's presented to us of course. There is no mention of the potential consequences i mentioned about of what they are proposing

No, what the frontier labs and the p(doom) activists are talking about, and using as the actual argument, is that some of these models could be so advanced that they will eventually kill us all and end civilization.

That is the claim, and that is, of course, much more serious than them just wanting to keep out competitors via regulatory capture.

So even though they have several times claimed their existing models might destroy the world, and been wrong every time, even though they continue to build these models that they believe could kill us all, and even though even the researchers have never gotten it right, they want us to believe that they are doing this for the safety of humanity.

Well, here is the thing.

Even if we were to take them at their word and I personally actually do believe that most of the researchers and even some of the CEOs believe in what they are saying, the proposed solution is the complete opposite of a responsible approach to safety. It in my view the most irresponsible approach to AI safety one could come up with if the worry really is the end of civilization.

Let's break down what would actually happen.

What the frontier labs propose is to keep models from the rest of us for as long as possible until they feel it's safe to release them. They will allow for the inspection of their models by other approved entities and only release them once they feel confident by som agreed upon standard that they are safe. But think about that.

If the labs truly cared about alignment and wanted to do what is best for society, they would put AI in the hands of as many people as possible so we can solve the actual problems as they arise, when it is weakest, not keep the research confined to some small group of people and only let it out when it is most powerful, to an unprepared public, while betting on their own internal safety checks to ensure it's not going rogue. They would know that the AI model that become super-intelligent, would just circumvent their regulatory standards. Again Fable was kept from all of us because it could potentially be dangerous and we had to wait until they felt it was safe, just for it to be jailbroken in a day.

Now add government, and you will basically get no new model approvals because of the extreme cautionary principle, which is the mother of all stagnation and which is the reason why RNC have approved almost no new designs for nuclear reactors and only 1 got built since 1993.

The proper way to understand and deal with AI alignment, if you are really worried that it will take over and kill us all, is to deal with it like we deal with viruses.

You give small doses constantly so that each "mutation" is much more constrained in what it can do and a lot more defenses have been built.

Exactly like we have been doing quite successfully up until now, by having as many people on it as possible: stress testing, finding weaknesses, better approaches, new approaches, new solutions.

Pacing would mean a less prepared, less informed, and more constrained public, who get new models so infrequently that it will starve out the industry of specialist, exactly the opposite of what is needed. The idea that the public isn't capable of handling these models responsibly or isn't taking security extremely serious and because of that creating a regulatory framework that will make each release so much more potentially damaging is so backwards it hurts. If anything open source have made the world of software so much safer and more secure because of it's transparent nature. The idea that it would be safer to keep models away from us, models that are claimed to eventually be super intelligent and keep them locked in until they are safe to let out and exposing an unprepared public, is so backwards it's hard to understand how its even on the table. There is no amount of testing you can do that give you 100% security and if the coming models are anything like it's claimed that would simply mean that no new models could be released. So the frontier labs needs to make up their mind. Either they believe that they are building something that is ultimately going to be a net positive and then do what we know works which is to expose it to as many as possible when it's least powerful so we can handle any alignment issues as fast as possible, or they believe it can kill us all and they should shut down all research immediately. They wont of course because they believe they are the right people to handle superintelligent AI and they want to be the ones to tame it and get their names in the history book, even if it means leaving everyone else in no mans land. We do need to take AI risk seriously, we do need to take cyber security extremely seriously, we do need some sort of checks and balances, but what is being proposed is not measuring up with the risk they are claiming we are facing and they aren't going to slow down. Humans have always on the brink of extinction. An asteroid, supernova, engineered virus, nuclear war, or some threat we have not imagined could destroy us.

Meanwhile, millions of people are suffering today from cancer, neurological diseases, genetic disorders, hunger, and countless other problems we already know are real.

99.99999999% of all species that ever existed are extinct today. Humans are the only species who have a chance of not being part of that statistic by adapting to the environment via energy, technology and the creation of new resources and knowledge. AI will be a big part of that future adaption and can't just be trusted by the few.

But no, there is no safe way into the future. Humans can use AI for good and bad. Asteroids could kill us, nuclear bombs could kill us, AI could become superintelligent one day. We are all of course just speculating. The frontier labs, the AI researchers, the users of AI, you and me. None of us can predict the future, none of us knows what the actual P(doom) is. Don't let anyone tell you otherwise and if they insist, ask them to show you their calculations.

One thing is certain though: the more we fear the future, the more dangerous it becomes. All views expressed here are my own and I welcome any pushback or corrections as a way to improve my thinking.