In this essay I explain some of my guesses for where AI development is going in 2025, using a dense set of heuristics and mental models that I hope are particularly useful for those who work in AI policy.

I argue for several framings that are key to predicting future developments:

• AI will eventually diffuse widely. Costs are falling extremely rapidly, and distillation and compression are powerful. It will simply not be very expensive, in 2030 $, to make AGI.

• Reasoning model progress is driven by repeated iteration in domains with verifiable or close to verifiable reward. The number of domains can be expanded over time to achieve wide coverage. This is complemented by periodic scaling up of pretraining.

• “The model does the eval”—models are good at things which we can build evals for, and poor at things for which we struggle. However, building evals becomes easier with more capable models.

• Agents are key to economic effects, not chatbots, but they are expensive to run! On current trends, there will be a “capabilities overhang” for a few years, with significantly more demand for compute than can be met globally.

• An easy way to measure the abilities of AI agents is their time horizon—the length of time they can act autonomously and reliably. The reliable time horizon of frontier AI agents will be significantly ahead of others, but given enough time even open source agents will be able to act as a “drop in remote worker”.

• The “automated researcher” is possible, and we should expect them to significantly accelerate progress over the next few years.

Tick-tock?

In the midst of a chaotic information environment, I think it’s worth stepping back to reflect upon how far we’ve come and consider the current drivers of AI progress. Throughout 2023 and into late 2024, we have seen the steady development of language models into AI chatbots, as they become more and more integrated into our daily lives. We use them as search tools, as writing assistants, but also increasingly for advice, counsel, and as a brain partner.

We are now moving into a new, even more rapid phase of AI development, focused on autonomous AI agents and reasoning models. The term “agent” is overhyped, but the core concept is still valuable: an autonomous system that can take actions over many steps to achieve its goals, without needing direct supervision. In today’s context, this increasingly means actions on the internet or in a coding environment.

The main engine of this new rush in progress is the breakthrough, years in the making, of the ability to get AI models to think reliably for longer periods of time, reasoning their way to a better answer. We first saw true inference-time scaling with OpenAI’s o1 release, although researchers had been trying to make variations work for years prior. DeepSeek’s r1 paper lifts the veil from the core research idea: using reinforcement learning on LLMs, with rewards on the outcome of tasks with verifiably correct answers, will automatically teach the models to error correct, generate hypotheses, check their work, and reason towards a final answer. “Unhobbling” has arrived in earnest.

A real piece of magic here is that the model’s learned reasoning heuristics—the ability to consider an idea, then backtrack and reconsider upon gaining further information—generalize outside of the domains it was trained on. You can ask DeepSeek’s r1 to write you a story, and its chain of thought will follow similar patterns, despite clearly not being trained with RL to write stories.

This generalization shows a tantalizing path forward to further, self-reinforcing, advances in AI’s capabilities. Of course, to some extent these will still depend on the ability to verify the correctness of an answer, and get reward for the RL training process. Many commentators have pointed out that this may significantly limit the potential of reasoning models in the short term, because only a relatively narrow range of tasks have exact and cheap verification.

But very often, verification need not be exact! There are a surprising number of ways to obtain proxy rewards with a combination of specialized models, heuristics, and format specifications, even in relatively open-ended domains like legal writing or navigating web pages. These are domains with pseudo-verifiers. For many web browsing or enterprise software navigation tasks, synthetic environments and rollouts can be generated which are sufficiently realistic (consider that a huge portion of white collar work simply occurs in Google Docs, Office, Gmail/Outlook, and Slack). Researchers must be cautious, of course, to not Goodhart or over-optimize models for these proxy rewards, causing worse performance in the real world. And using a pure LLM reward model as a verifier without any grounding will usually fail to scale. But often, the implicit gap between the difficulty of verification and the difficulty of generation will be sufficient to progress in important domains.

In hindsight, perhaps it’s not surprising that this works so well. Previous attempts to get models to reason their way through complex problems often depended on supervising the correctness of each step of reasoning—process-based supervision. But especially for large and complex tasks, why should we expect models to follow the same reasoning process as humans? AI models have non-smooth skill distributions: in contrast to humans, we cannot reliably predict how capable an AI model will be on closely related tasks. This property is improving as models get better and more robust, but there are still, for example, many types of problems for which GPT-4o is worse than a nominally weaker competitor, or where performance varies drastically based on minor prompt differences. As a result, the model designer should not try and force the models down a reasoning path that is most natural for humans. Instead, let the model figure out for itself the best path to solve the problem: give them a goal and let them run. The models just wanna learn.

In parallel to advances in reasoning (and post-training), we also expect improvements via scaling up pretraining. Grok 3 and particularly GPT-4.5 are the early signs of this—GPT-4.5 is clearly a much bigger model, but it hasn’t received significant RL fine tuning for reasoning, so is most clearly comparable to GPT-4 to see the effects of scale. And what do we see? The public benchmarks are relatively underwhelming (mostly due to saturation), but the model topped the Chatbot arena and qualitatively seems to be an extremely good model, with some users reporting that they prefer it for common coding questions.

Overall, development seems to be trending towards a “tick-tock” model, in which pre-training scale-ups every few years are complemented by increasingly fast progress in continually finetuning the models using RL across a spread of verifiable or pseudo-verifiable domains.

Significant efficiency improvements are also being driven by the power of distillation. As researchers train larger and more capable models with each generation, they can use them to generate data and reward signals to train smaller, more compressed, cheaper models which retain much of the original capability. The abilities of the small Gemma 3 models would have shocked the AI researchers of 2021. The fact that such strong capabilities are possible in so few weights is a powerful hint from the universe that many components of human intelligence are just not that complex, and that this technology is ultimately destined to proliferate cheaply.

These factors drive the widespread, albeit lagged, capability diffusion of AI into smaller and cheaper models over time, including those created by non-frontier AI labs. Because larger models tend to be more capable, and yet more expensive to run, AI labs are increasingly incentivized to follow a scheme of “train large strong model, distill into smaller cheap model”, as seen with Gemini 2.0 and o3-mini. Wide deployment then ends up being limited largely by the effectiveness of distillation and compression.

So far, we have seen distillation (and other algorithmic advances) improve over time, to the point that small models can continually increase in performance despite not changing in size. For economically useful tasks, models at the frontier are still markedly more valuable than their smaller, less capable cousins. But most tasks which are highly valuable have a fixed “difficulty”—at some point, due to distillation and compute improvements, models of a fixed low $/token price will be capable of automating the work of e.g. a junior investment bank analyst, whilst the most capable models will still be struggling to complete much higher complexity tasks. This dynamic will likely continue until the vast majority of economically valuable tasks can be cheaply automated, even if it takes some time to convert the models from expensive to cheap—although many difficulties remain, especially around tasks which are harder to verify or build good environments for.

Finally, in the bigger picture, all the progress we have seen so far is a result of the effort put into building better and better evaluations for capabilities. Anything we can robustly quantify, we can turn into a benchmark—and then the model does the eval. In the long term, this means that we should expect AI models to do well on almost anything we can specify and quantify, and poorly in things we struggle to define rigorously, like top 0.1% fiction writing ability.

The Automated Researcher Dream

A huge driver of progress, still underestimated, is the advent of AI agents capable of AI research: automated researchers. The frontier AI labs are racing fast towards end-to-end AI R&D agents. They know that once this capability is achieved, progress could be made with much greater velocity, potentially in a self-reinforcing loop—newly discovered algorithmic advances themselves helping to bring the next breakthroughs closer. The best AI researchers in the world are rushing headlong into a grand project to automate their own jobs. Indeed, there are rumors that within Anthropic, some researchers have raised concerns to management that their jobs could be at risk.

Of course, AI safety advocates have many concerns over this: it may be harder to supervise in some way—how could we tell whether our automated researcher is capable of robustly evaluating the latest, more capable AI model? This problem, known as scalable oversight, has long been a reason for some to argue against building superhuman AI systems. But I’m more optimistic: the advent of inference-time compute scaling implies that the scalable oversight problem is potentially solvable by simply providing more compute to the supervisor or evaluator model.

However, automated research has more difficulties than meets the eye. First, most AI lab research teams are compute bottlenecked for their experiments, and are limited to some GPU allocation handed down from on high. Researchers are strongly encouraged to use all of their compute, and typically have far more useful experiment ideas than they can carry out, even if they somehow coded them up instantly. To achieve significantly better utilization of a compute budget and thus faster research, automated researchers need to have better ideas than the average frontier AI lab researcher: quite unlikely, at least in the short term.

That leaves the other parts of the research loop, most prominently engineering: coding up new research ideas and developing faster or more efficient infrastructure. Significant parts of frontier AI lab workflows are bottlenecked on simple implementation speed and testable software improvements (e.g., CUDA kernels), and this looks much more tractable for the first automated researchers to tackle. I believe that automated researchers are likely to provide a large boost here, but it may be effectively a one-time boost to overall research speed without improvements in the other parts of the research loop.

Overall then, the incentives are strong for AI labs to race towards capable automated researchers. Attaining this capability means that labs are less talent constrained: it enables a pure conversion of compute into intellectual effort, and implies that even labs which struggle to attract talent are guaranteed at least some baseline of research ability, provided they can obtain a good enough starting model.

However, the biggest unanswered question is still that we don’t know how fast the automated researcher feedback loops will run. Will we get a modest “one and done” bump in the short-term from automating engineering, or will the self-reinforcing feedback loop be so strong that we unlock significantly more capable automated researchers in short order? To some extent this is a fundamental question about the difficulty of AI research itself, and whether the marginal research secret (or “micro-Nobel”) becomes easier or harder to obtain over time, if you are continually improving at research ability. Regardless, we will likely find out the answer soon: I currently believe leading AI labs are on track to have the first fully automated researcher prototypes by the end of 2025.

An AI research ecosystem with a dramatically larger capability differential between labs only ~6 months apart in progress may have interestingly destabilising effects. As new capabilities appear even quicker, new implications for economic growth and national security are unlocked with even less time for companies and governments to react. Currently, the speed of automated research is set to be closely guarded by AI labs—I think that reporting some statistics about this to say, the Office of Science and Technology Policy in the U.S. government could help improve decision making significantly in the future.

Agents and their Effects

I’ve been playing with Deep Research and Operator for several weeks now, and I’m convinced: these systems1 are our first glimpse at the agents of the future. They have an unprecedented degree of coherence and reliability over long time horizons, despite many rough edges. It’s often captivating to see these models generate tens or even hundreds of thousands of tokens of reasoning before giving their final answer.

However, there are still many issues to overcome as this new form-factor develops. Agents operating within a browser have key limitations on a technical side: reliability (which has noticeably increased, driven by dataset improvements), visual understanding (for operator-like agents), planning capability for high level tasks, and familiarity with key common apps and websites. This last one is grounds for optimism about the speed of economically-relevant development: since vast quantities of office work takes place essentially within a handful of apps (Google Suite, Office, GMail, Slack, etc), it should be possible to vastly increase the reliability of agents in these specific apps by creating handmade training environments for each.

There is a framework for describing progress here which I quite like, called t-AGI. If we take AGI to mean the “drop-in remote worker” imagined by many AI lab leaders, the idea of t-AGI is that we should, before full AGI, expect to have systems capable of acting like a drop-in remote worker for time-bounded tasks that would take an expert human, say, 30 minutes. Then, upon further development, the AI system will become capable of 2 hour tasks, and so on. We should also consider the probability of success: the system may be capable of the average 30 minute task with an 80% chance of success, and we can plot the progress over time at a fixed success rate.

But this is too low-resolution, since we know that AI capabilities are spiky! Models which can reliably code features that would take an expert software engineer 2 hours may struggle terribly to do things that a junior consultant or physicist can do in 10 minutes. Of course, there is some amount of general capability transfer, and in many ways AI progress for the past few years has had the effect of making things less spiky. But overall, I currently prefer to refer to t-AI, not t-AGI, and describe things in specific but broad domains.

We already see this t-AI dynamic with agents to some degree, especially in areas easier to evaluate like software engineering and AI research (as shown by METR). I currently think we have AIs averaging 80% reliability at a broad sweep of 10 minute tasks, though the time horizon is probably closer to 30 minutes or longer for specific domains like software engineering and compiling basic research reports from the web—and longer still for 50% reliability. I expect these time horizons to extend to ~8-12h by mid 2026, and a full day by 2027.2

We also see it with Operator and Anthropic’s computer-use agents, which can often reliably complete web browsing tasks of short duration (e.g., 1-10 minutes) but which struggle to continue the more actions are required. I expect this increase in t, for many domains, to be one of the biggest factors underlying the development of economically useful models. Deep Research is currently a leading indicator of this—the model is capable of searching over the web up to ~30 minutes in duration, before compiling a report about its findings.

Operator inherently delegates authority to humans for some actions which are particularly important or sensitive. This functions as an implicit "decision threshold", a quality that is partially social and cultural, which we can expect to rise over time as the system proves its reliability & trustworthiness. Eventually, this threshold may be high enough for contracting or managing workers, making major purchases, sending critical emails, and more. The autonomous corporation beckons?

Compute Bottlenecks

Now, suppose an AI company develops a 60m-AI agent for many tasks common in the professional workplace, including software engineering, producing, analyzing, and reviewing reports & slide decks using publicly available information, drafting or editing articles or papers, and so on—tasks chosen because they appear to be amongst the easiest to automate with current technology. This AI agent would be immediately extremely valuable to many businesses—but how many businesses would actually be able to run it? Do we have enough compute for widespread economic benefits? Indeed, labs have shown some indications that reasoning agents are strongly compute constrained: even paying for OpenAI’s pro $200 subscription only gives you 120 Deep Research queries per month.

Assuming that this 60m-AI agent might be based on a model of similar size to DeepSeek’s r1 (or indeed larger), then it requires at least a full set of 8xGH200 GPUs to run. This is the size of NVIDIA’s latest AI server, costing $300k. Some basic napkin math shows that OpenAI’s announced Stargate project could sustain running at least ~3.1 million of these agents 24/7 by 2029 for a cost of $500b, (albeit not counting batching, so a very loose lower bound).

In 2025, the approximate existing stock of NVIDIA H100s in the US could sustain, on similar assumptions, at least ~125k agents. If we assume distillation and model compression improves on trend, this capability could eventually fit onto a single chip, making the numbers ~25m and ~1m respectively (let me know in the comments if you have better numbers!). The excellent provides some similar numbers for the long-term estimate (on a single chip) via a different route.

These are not huge numbers, especially if models grow to be capable of automating large portions of work in many professional sectors. Similar calculations may be driving some of Sam Altman’s desire to build Stargate. Of course, there are many other factors in inference economics which could both alleviate or increase this potential bottleneck. For example, distillation will continue to be strong (enabling older GPUs to be used at scale), inference runs well on older GPUs or alternative chip providers, other algorithmic factors like batch sizes, parallelism, and token speed will likely improve, and adoption is set to be slow throughout many sectors even after the technology exists. On the other hand,if pretrained model size needs to be scaled significantly to achieve high reliability, the inherent inefficiency of serving large models (see GPT-4.5 token costs) may delay large-scale use of effective agents for some time. We also have yet to explore how useful it will be to run many agents in parallel for a given task—perhaps the best configuration consists of a swarm of agents completing different subtasks? And what new forms of work may be enabled by the existence of these agents?

Conclusion

Overall, I expect AI agents to be very compute bottlenecked in the next couple of years, largely because of the sheer speed of progress creating an “economic overhang” of sorts. By the time we have 24h AI agents for many common tasks—perhaps mid to late 2026—most data center projects currently planned & approved will be nearing completion. Other infrastructure is not so speedy, and things like energy and power transmission construction have significantly longer lead times. At some point, progress (and importantly, economic adoption) may slow down due to fundamental constraints on the availability & cost of chips or energy.

A large compute supply bottleneck will create new (temporary) political and economic dilemmas for both AI labs and governments. Should compute be preferentially allocated to companies, academic researchers, and other groups, to satisfy the interests of both the public and AI developers? As we move towards capable autonomous scientific researchers, can governments use their compute to accelerate key public interest research projects in many domains of science?

Longer term however, there is plenty of light at the end of the tunnel, driven by the combination of increased AI chip production and efficiency improvements for a fixed level of capabilities. Eventually, I expect t-AGI for most common workplace tasks3 to be at a level of capability distillable into fairly cheap models, which will greatly decrease the compute requirements and potentially unlock t-AI on consumer devices.4 Ultimately, I believe the cost of intelligence will tend towards zero.

This is the first of two big picture essays bringing together my views on AI progress and where we are headed. The second part, coming soon, discusses the implications of this AI progress for the economy and geopolitics. I will argue that in the context of the US-China competition, AI should be viewed as a form of raw economic advantage, with compute as the key lever that the US can use to secure this advantage.

Two months ago, the political board in the U.S. was flipped, and many assumptions in the world of policy that felt like constants are no more. We’re in a New Year, and in this, the first piece of AI Pathways, I aim to answer the question: what can we expect from the next government’s AI policy? And what might this imply for the future development and deployment of the world’s most advanced AI systems?

My main message is this: the strategic landscape has dramatically shifted. After spending the last two months chatting to AI researchers and tech policy folk around Washington, D.C., San Francisco, New York, London, and Brussels, I'm convinced that almost everyone outside of D.C. is strongly underrating the implications of the election. Many exciting new opportunities have opened up for both AI research and tech policy, while some Biden-era policy paradigms (e.g., pre-deployment evaluations and risk assessment as a dominant form of governance for frontier AI) are on their way out.

I’ll briefly list the main takeaways here, before going into specifics. The main policy motifs I see from the new administration are:

1. An increased focus on China as a technological adversary and effort to maintain American advantage in AI. This is a core motivation behind many potential policy actions, including boosting domestic energy production, onshoring of more semiconductor manufacturing, export controls on chips, and building out AI applications for national security.

2. One of the most underappreciated effects in AI policy of the U.S.–China dynamic is greater interest in using AI for national security and military applications. There are two sides to this coin:

   1. building defense-specific AI applications, which will likely lead to the government conceiving of AI as increasingly a defense technology;

   2. securing existing frontier AI systems and IP from foreign corporate espionage or other attacks. As long as the capabilities gap with China is maintained (an open question given DeepSeek’s impressive progress), there will be strong incentives for tighter security partnerships between the U.S. government and the AI labs.

3. A potential conflict in the new admin is between those favoring tighter government control over the security of frontier AI model weights & IP, motivated by viewing AI through the U.S.–China competition lens, and those arguing in favor of minimal security restrictions & boosting the open-source AI sector to make adoption throughout the economy easier. Senior figures in the new admin’s tech policy, including David Sacks and Sriram Krishnan, are strong advocates for open-source, and there are good arguments in favor of their views given the likely inevitable diffusion of frontier AI capabilities over time. I strongly believe there is a viable way to unify these two perspectives—the securitization view and the open-source view—and hope to articulate a path forward in a future piece.

A key concept here is that as capabilities grow, we should increasingly expect governments to view AI as a strongly dual-use technology. Up until now, AI has been an almost entirely civilian-first technology, but government interest in it seems likely to grow along with capabilities that provide more possible applications to defense.

This leads to some guesses for Trump’s AI-relevant policy moves:

• Moves to boost AI datacenter growth, by unblocking permitting, or funding and incentivizing the construction of new energy supply and transmission capacity. U.S. energy demand forecasts for the next 5 years have increased rapidly, largely due to AI datacenters but also because of greater than expected growth in manufacturing. The electricity grid is underprepared for this, and only speedy action can prevent economic growth being bottlenecked by power.

• Scaling up of initiatives within the federal government and the defense sector to use frontier AI in national security or military applications. We see steps towards this outside government in the burgeoning Palantir-Anduril defense tech consortium, OpenAI’s partnership with Anduril, and Palantir’s partnership with Anthropic & AWS. I expect defense-specific finetunes of leading LLMs to be ultimately sold as products to the government, which strongly motivates a need to secure these model weights from cyber attacks. We may see export controls on model weights to prevent some models being deployed outside of U.S. datacenters.

• Testing and evaluation work within the federal government is likely to move to more national security focused agencies and teams. For example, USAISI may become a much smaller share of this work, and could be moved out of NIST into a more relevant place like the Department of Energy (DOE).

• Trump’s administration seems likely to maintain and tighten export controls on semiconductors, to slow down Chinese AI efforts. Their effectiveness will depend on how well the multiple different agencies working on export controls, including BIS in Commerce, can actually be coordinated and run efficiently. DOGE may play a part in the latter effort.

• Deeper partnerships between frontier AI labs and the U.S. Intelligence Community to improve the cyber and info-security of lab development & deployment. This may involve a so-called public-private partnership (PPP) between e.g., the DOD/DOE and one or more leading AI companies, including potential government security experts embedded in the labs. Due to Elon’s involvement, x.ai is an obvious lab to watch closely for potential partnerships with the government.

• A greater unknown is how to deal with the patchwork of currently-proposed state-level AI bills. The more insane the patchwork appears (*cough* Texas *cough*), the greater the incentive for the federal government to pre-empt states by passing some federal level AI bill that overrides them. This might be a light-touch AI bill (to avoid slowing down developers, particularly open-source) aimed at reducing uncertainty for businesses deploying AI systems and boosting economic growth. But it is currently unclear how high this will be on the priority list for Congress, especially given the slim Republican majority.

In summary: if you’re working on AI policy or on technical research which you think will be useful for policy (particularly on anything in the bucket sometimes called technical AI governance), then you should seriously consider the vibe shift and its implications. Some people like to make In/Out lists for the New Year, so my guess is that in terms of usefulness for AI policy, we’re looking at In: work on AI security, ensuring robust & reliable AI for defense applications, and forecasting the likely economic impacts of AI agents; Out: AI bias evals, pre-deployment third-party safety evals, safety cases, adversarial robustness and jailbreaking, risk assessment frameworks, and mechanistic interpretability. I will expand on these takes in a separate piece.

Background

There are already several good pieces you can read to get a sense of Trump’s existing statements on AI, as well as hints from those working with the Trump campaign. The main explicit policy action we know of is the existing commitment to repeal Biden’s AI Executive Order. This seems likely to be replaced with a Trump AI Executive Order at some point.

The Project 2025 policy agenda is also a useful an indicator of interest from a group connected to the transition team, and contains a variety of AI-relevant suggestions (findable by searching “AI” or similar). However, Project 2025 should be viewed more like a list of potential directions, considering the document was drawn up many months ago now.

Elon Musk—A Wildcard

Of course, much also depends on Elon Musk—how influential will he be in the new administration, and what will his preferred AI approach be? Will he maintain a good relationship with Trump for the entire term?

This is currently very uncertain, since we both don’t know what work Elon will spend most of his time on (for the short-term it is clearly DOGE), and Elon’s own preferences for AI policy are unclear, although he has taken a strong interest in its development for many years.

Many in Silicon Valley AI circles hope that Elon’s deep familiarity with the AI industry and influence in the Trump administration will help shape the next government’s policy. So far, this seems likely to play out, potentially to the benefit of x.ai.

More broadly there are, as we have seen with the recent immigration discourse, strong tensions between the populist right side of the Republican party, and the “tech right” (encompassing Elon, Silicon Valley Republicans & libertarians, e/accs, etc). Whether Elon maintains influence depends to a large degree on how these political tensions shake out.

The U.S. AI Safety Institute

The fate of the U.S. AI Safety Institute (USAISI)—a small team of technical experts & policy staff within NIST in the Department of Commerce—is a question on the minds of many in AI policy this month. Trump has promised to repeal the Biden AI Executive Order, part of which concerns pre-deployment safety evaluations of frontier AI systems (USAISI’s main activity). The later Biden National Security Memorandum also directs USAISI to act as the central point within the government for frontier AI work & engagement with the labs.

USAISI has attracted attention from influential anti-regulation Republicans in 2024 for its collaborations with UKAISI and other non-governmental AI safety organizations. This particularly includes the November International Network of AISIs event in San Francisco, with a side event on safety frameworks (e.g., regulatory mechanisms) partly run by UKAISI. When I speak to observers in AI policy, many express concern that USAISI simply hedged insufficiently for a Trump, and is now trying to re-work its agenda and focus to be closer to the expected desires of the new administration. A narrowing of focus onto AI for national security seems like the right path for them—they have recruited some valuable technical experts, whose abilities will be a useful resource for other government teams seeking to use frontier AI.

As a result, a full winding down of USAISI seems unlikely. Although they are small (their technical team is a handful of people, the rest are policy staff), they have accumulated a number of well-known experts at the working level with strong track records. However, both hiring new talent and retaining existing experts will likely be even more challenging than previously. USAISI are known in the community to have a very restrictive conflict of interest policy that nixed at least one potential ex-industry-lab hire (which generally does not bode well for government’s ability to hire top AI talent).

Going forward, USAISI seems likely to still serve a useful function as an advisory body for various parts of the federal government with a stake in frontier AI, as well as a convenient point of engagement with the AI labs for some parts of pre-deployment testing.

Another rumored possibility, and perhaps a way to alleviate hiring inefficiencies, is that USAISI may be moved outside of Commerce. In many ways USAISI’s place in DOC is a historical accident born of Gina Raimondo’s interest in AI—given that much of USAISI’s testing work and coordination has been on national security risks, the Department of Energy is a potential natural home. The DOE has a much larger budget, significant computing expertise within the national labs, the existing DOE-DOC testing collaboration, and is separately running tests with several leading AI labs. Either the DOE or DOD would fit nicely as a home for USAISI, given the increasing interest in the testing and use of frontier AI for national security applications & risks.

International Collaboration

What then, for international collaboration on AI? Under Biden, the international AI governance landscape saw a great proliferation of mostly symbolic fora, agreements, dialogues, summits, policy frameworks, and commitments. Examples include the Hiroshima Process (via the G7), the OECD AI Principles, the UN AI Advisory Body & report, the AI Safety Summits in the UK and Seoul, and the International Network of AISIs.

Some of the most significant for frontier AI were the AI Safety Summits (although I am biased, having had a small hand in organizing the first), which were able to secure commitments from the leading companies & many countries to collaborate on testing for frontier AI systems. Despite appearing vague and high-level, these kind of major international commitments can serve as a form of social pressure, since they can be cited in any engagement between countries and companies as a motivator for why the frontier AI labs should do, for example, joint safety testing.

This works fine if your goal is purely imposing political cost on companies that do not engage with government interest in AI deployments. However, there is nothing binding here—these commitments do not either force companies to take action or offer additional options (e.g., additional types of deployment mode or form factor), they simply raise the political cost for a company to follow a path that a government may disagree with, a cost that may be worth the price in a more high-stakes situation.

From an America First perspective, the motivation to engage significantly in internationalizing frontier AI governance looks quite weak given that all the main companies involved are U.S.-based, and so I do not expect as much engagement on AI evaluation & testing as under Biden (whose administration was more ideologically motivated to pursue that project).

The International Network of AISIs

The International Network of AISIs is an informal coordination mechanism between the governments of several countries to collaborate on the testing and evaluation of frontier AI systems. I say informal, because it is fundamentally bound by Memoranda of Understanding (MoUs) and other forms of non-hard law agreement.

The Network was preceded by the MoU between the U.S. and U.K. AISIs, who collaborated on joint safety for Anthropic’s Claude 3.5 Sonnet, and OpenAI’s o1 model. These two AISIs were the first and are, as far as I know, the largest. As a demo of testing amongst a larger group, this pair recently ran a joint safety testing project on Meta’s Llama-3 405B together with Singapore AISI.

The Network’s activities are likely to be significantly affected by USAISI’s fate, the Network’s Chair. This is because structurally, AI labs based in the U.S. have a strong incentive to work primarily or only with the U.S. federal government on safety testing of their models—the U.S. government is the most natural partner here, especially when it comes to testing on national security risks like CBRN. The Network acts as a mechanism to transfer this incentive, via U.S. AISI, to other countries who have the capability to do safety testing, setting a precedent towards internationalization of this testing.

However, this mechanism only happens via the scope of testing USAISI conducts; if USAISI loses their mandate from the Biden AI EO to act as the central point for safety testing of frontier AI within the U.S. government, we should expect more testing to be done in other parts of the federal government, outside the scope of the International Network. This is doubly true since the Biden White House acted as a forcing function for this centralization to happen, often proactively nudging parts of the government interested in frontier AI to collaborate with U.S. AISI.

Therefore, it is likely we’ll see less international collaboration for safety testing on frontier AI models. This has pros and cons: in some sense the Network can be viewed as an attempt to reduce centralization of governance over frontier AI, but on the other hand it may also bring increased security risks or cause confusion by adding too many additional actors with different definitions of safety.

But will national AI Safety Institutes other than the U.S. or U.K. be relevant to the global trajectory of AI? This move to internationalize evaluation-based AI governance faces a fundamental problem: the United States is the base for of all of the world’s leading frontier AI labs, except for (arguably) DeepSeek in China. In regulation and governance, other jurisdictions rely purely on the desire of foreign tech companies to make profit from deploying there (e.g., the so-called Brussels effect, significantly overhyped by the EU Commission).

Almost none of the other countries in the network have so far built sizeable safety testing teams staffed with technical experts, or shown intention to do so. Singapore AISI, for instance, is primarily focused on encouraging more safety research in neglected directions via academia and building evaluation tools for the ecosystem. Other countries (Canada, Australia, France, Japan) have announced or begun setting up their own AISI, yet these look more like a gesture towards AI’s importance from their respective governments rather than efforts likely to lead to highly productive research organizations. And does Kenya really need an AI Safety Institute? To what extent is pushing an international consensus on things like best practices for misuse risk evaluations actually useful, given the rapidly shifting state of evaluations research?

Finally, it’s worth noting the inclusion of the EU AI Office in the International Network, which is in many ways the “odd one out”. Unlike the other AISIs, the Office is a regulatory body within the EU Commission, tasked with implementing the EU AI Act. This has given NIST a delicate line to navigate, since the Network thus ties NIST to a foreign regulator, even if it is for informal collaboration purposes.

Securitization

The securitization of AI is increasingly a hot topic in the tech policy world, which I define as work to improve the cyber and information security of frontier AI training and deployment—particularly model weights and IP. The last year has seen numerous public calls to improve this security, including Leopold’s Situational Awareness essay, and the RAND securing model weights report. The primary motivation for this goal is to maintain the U.S. lead in AI capabilities by ensuring that other actors, including nation-state adversaries, cannot hack or otherwise obtain frontier AI capabilities, under the assumption that these capabilities will become increasingly important for national security.

As many observers have noted, the effectiveness of this strategy is much reduced when both China and open-source are close behind the frontier AI capabilities of “closed” industry labs like DeepMind, and where that capabilities gap may credibly narrow. Nonetheless, my personal intuition is that securitization is still very valuable to pursue, because it provides much greater optionality for long-term U.S. AI strategy. Many China hawks with connections to the Trump administration are also advocates for this view.

Relatedly, we see a strong desire to apply frontier AI systems for national security and military applications, including moves like OpenAI’s partnership with Anduril, Palantir’s partnership with Anthropic & AWS, and OpenAI’s board appointment of an ex-NSA general. Indeed, fine-tunes of leading frontier LLMs for defense applications are a likely way that frontier AI labs end up increasing their security in the short-term. The U.S. government’s interest in securitization of frontier AI is likely strongly correlated with the utility of these defense applications—and given the rapid progress in AI development towards autonomous agents, it is plausible that some applications could soon become a significant national security capability.

This overall direction also coincides with Silicon Valley’s increasing recognition, in the making since the Ukraine invasion, of the necessity of tech’s involvement in maintaining American security and military leadership, and the stakes of the geopolitical tensions with China. In stark contrast to the anti-military vibe of several years ago, defense tech is finally cool in SF—I attended a recent Palantir party where guests cheered at the host’s description of the “Stanford to crypto to AI to defense tech” pipeline.

Early moves by the government to encourage this security may include creating stronger partnerships between the national security world and the frontier labs. For example, this could include embedding government cyber and info security experts into labs to beef up their measures.

The DOE has the potential to play a role here, given that it contains the National Labs, which posesses several AI compute clusters (both classified and unclassified) as well as significant high-performance computing expertise. If high-security AI training or deployment clusters are needed to support national security applications, the DOE is a natural home for their construction, in partnership with industry labs.

Energy, Semiconductors, and Compute

For context, there are already several excellent policy proposals arguing the case for serious investment in energy for U.S. datacenters, and in the compute build-out itself. In this, the new administration is likely to support similar high level industrial strategy themes to the Biden administration: building more energy supply, onshoring more chip manufacturing, and building more compute capacity. President Trump’s tariff agenda may become relevant here, since he floated the idea of putting tariffs on chips from Taiwan on the Joe Rogan podcast.

Even if AI capabilities stagnate, we still expect energy demand to increase drastically over the next 5-10 years—driven by demand from batteries, semiconductors, and other manufacturing being onshored. Adding AI on top, the potential of increasingly capable AI agents to create vast deployments throughout the economy, using many times more compute than now, means that the U.S. is in danger of severely underbuilding its power supply and energy transmission infrastructure. Much recent industrial policy from both Democrats and Republicans recognizes this and is advocating for huge investment in energy.

The Trump administration seems likely to continue this trend. One indicator is the proposed appointment of Jacob Helburg to undersecretary of State for economic growth, energy, and the environment. Helburg is a prominent Silicon Valley Republican with deep connections to the AI industry, who has recently advocated for facilitating energy investment via permitting reform, including oil, gas, and nuclear, as well as reshoring manufacturing of all elements in the AI supply chain.

The CHIPS Act, which subsidizes domestic chip manufacturing, has received criticism from some Republicans, primarily due to provisions around union labor and climate research. However, its core idea of boosting U.S. chip production still receives bipartisan support. Given the urgency and looming supply bottlenecks, further legislation to incentivize private-sector investment in energy, semiconductors, or AI datacenters could be on the table for the new administration.

A Note on DOGE

DOGE’s focus is likely not particularly relevant to AI policy, though it may have some impact depending on which teams and offices it is directed to focus on by Elon and Vivek. To briefly summarize, the latest rumor about DOGE is that it:

• is likely to operate analogously to Palantir: a talented team of software engineers, many of whom will be “forward-deployed” and embedded inside government agencies. These embedded SWEs will be able to see how things work, solve problems, and build tools within the bureaucracy, but also have rapid lines of communication to the White House, in case executive action can help fulfil DOGE’s mission;

• will primarily focus on efficency, both by slimming down staffing at key government offices, but also by building efficient technology to let government employees be more productive.

Most of the parts of the government DOGE could focus on to achieve the greatest financial savings are not that relevant to AI, but one suggestion might be this: dedicate some DOGE staff to the Bureau of Industry and Security (BIS) in the Department of Commerce. The BIS handles (a part of) the implementation and enforcement of export controls on semiconductors, and its function can plausibly become significantly more effective with DOGE assistance.

If you have any thoughts or disagreements with this piece, do post in the comments section or on Twitter, and let me know what you think! You can also reach out at mail [at] herbiebradley.com for a chat.