Nationalization, AI Security, and Open Source: The Most Influential AI Governance Articles in 2024

We’re thrilled to announce the upcoming launch of AI Policy Bulletin, a new digital magazine dedicated to shaping the conversation around AI policy.

For our first newsletter, we wanted to spotlight the most compelling ideas, research, and debates in the AI policy space in 2024.

‍

9 Pieces on AI Policy that Set the Tone in 2024

1. Situational Awareness

One of the most discussed pieces of 2024 was Leopold Aschenbrenner’s Situational Awareness, which rose to fame even before being shared by Ivanka Trump.

In this essay, Aschenbrenner argues that the AGI race is on and that the US will need to nationalize its AI industry to succeed. There are many critiques of this (very long) essay, but there’s no question that it captures a mindset fuelling Trump’s first few moves in office.

‍

2. How robust are compute thresholds for defining frontier AI?

With Biden’s now-repealed executive order on AI coming into effect in October 2023, 2024 saw the rise of a debate on the robustness of compute thresholds as a proxy measure of whether models are high-risk.

The argument in favour was set out in a long report and blog post on the relationship between compute and AI capabilities. Later in the year, Sara Hooker from Cohere wrote a critique of this approach, calling for dynamic risk thresholds, including proxies for risk other than compute. Exactly how to capture the right AI systems in policy remains a difficult question, and we invite further writing on the topic.

‍

3. Securing AI Model Weights

2024 saw a shift towards the importance of AI security – making AI more governable by ensuring model weights and algorithmic secrets can’t be trivially stolen or copied. RAND’s paper Securing AI Model Weights outlined why this is so important, setting the tone for AI security.

The paper analyzes the existing attack vectors and proposes security systems that will help achieve a certain security level. The topic of information security (that also appeared in ‘Situational Awareness’) certainly deserves more attention.

In fact, further research on espionage has been encouraged in a recent collection of Research Ideas in AI Governance. If you’d like to write about any of these ideas, we’d invite you to submit an article pitch!

‍

4. Governments Legislate on AI (or, try to)

The UK’s AI Safety Institute (now AI Security Institute) celebrated one year of existence in 2024, and the new UK government has promised an AI Bill.

Alongside the EU AI Act, California’s now-vetoed Senate Bill 1047 (“SB 1047”) faced heavy criticism from various companies. But was the discussion grounded in fact or fear? Efforts to reassure the AI industry that the Bill wouldn’t touch most companies were perhaps too little, too late.

‍

5. Consolidating concerns across AI policy communities

As 2024 progressed, risks that seemed speculative have become more and more realistic, and 2025 will be no different. While there have been tensions between groups focusing on different AI risks, we expect that finding common ground across a range of risk motivations will be essential for influencing policy in 2025.

One paper addressing this discourse head-on was Two Types of AI Existential Risk: Decisive and Accumulative by Atoosa Kasirzadeh. It points out that risks can elapse decisively, or accumulatively, and advocates for a consolidated approach to AI governance. The paper points out that these ideas help harmonise a range of concerns about AI’s impacts.

‍

6. Standardising discussion: MIT Risk Repository

Last year, the MIT Futures Group published one of the most comprehensive AI risk taxonomies to date: The AI Risk Repository. A comprehensive language for AI risks helps standardise the field, and improves our ability to compare work from different sources.

‍

7. Grounding the debate: the marginal risk of Open Source AI

In an impactful publication this year, Stanford’s HAI (Human-Centered AI) group and others called for more sanity in the AI risk debate. It challenges researchers to always consider the marginal risk of open-source AI systems - where “marginal” means comparing the harmful action enabled by AI to the current methods of performing the same action. For example, AI systems can conduct cyber attacks - but what’s the marginal risk introduced compared to humans doing the same?

This is indicative of where AI policy discussions have reached in 2024: We now have highly capable foundation models in our hands, and can increasingly discuss threat models empirically, rather than hypothetically.

This also paves the way for a crucial conversation in 2025: How do we deal with open source technologies? The technical research institute Epoch reported in 2024 that open source was closing the gap to closed-source capabilities. With the benefit of being 3 weeks into 2025, we’ve already seen DeepSeek r1 almost match the evaluation results of OpenAI’s flagship reasoning model, o1.

‍

8. OpenAI’s Head of Policy on the merits of research inside and outside an AI company

Miles Brundage’s Substack post Why I’m leaving OpenAI and What I’m Doing Next kicked off a series of posts on societal readiness for highly advanced AI systems (or lack thereof). Coming after a series of significant departures from OpenAI in 2024, Brundage explains why he thinks he can work more effectively outside of the industry.

In addition to explaining the end of his journey at OpenAI, Brundage argues that policymakers should take the prospect of advanced AI much more seriously and act with urgency, and shares his detailed analysis of topics that seem particularly valuable to explore and dedicate more attention to right now. Topics include the acceleration of beneficial AI applications and the economic impacts of AI.

‍

Looking ahead: o3 and the rise of reasoning models’ test time compute

Finally, at the end of 2024, OpenAI released its most capable AI system yet. Experts were shocked over the Christmas break by o3’s reasoning capabilities - particularly demonstrated by achieving over 75% on ARC-AGI - where the previous best had been around 30%.

The result involved OpenAI allegedly spending up to $1 million in compute on running the benchmark alone. This rings in-the era of increasing relevance of ‘test-time compute’ over pre-training compute, potentially having important implications for training compute thresholds, and compute governance more broadly.

All in all, this model’s release gave us all a big question to ponder over the holiday break. Will 2025 be the year we see AI begin to have notable economic impact across domains? We have new governments in Europe, the USA and the UK - what will they do to govern this era of AI?

‍

---

Subscribe to stay in the loop as AI policy develops in 2025!

‍