Dario Amodei — 我们必须放慢前沿步伐

过去十二年来，我一直致力于人工智能领域，因为我相信它能够大幅提升人类生活的质量。我经常撰文谈论这些令人难以置信的益处：我相信，人工智能可以在未来 5 至 10 年内治愈大多数重大疾病，大幅加快经济增长速度，创造一个充满富足与赋能的世界，并迎来民主与自由的复兴。我个人对此深感紧迫。我的父亲死于一种疾病，而就在他去世几年后，这种疾病便得到了治愈；我自己也曾患上早期癌症并幸存下来，而即使在五十年前，这种癌症也无法得到治疗。只要谨慎运用，人工智能就能成为漫长技术奇迹序列中的最新一项，为人类带来福祉并提升人类的尊严。I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life. I’ve written often about these incredible benefits: I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom. I feel the urgency personally. My own father died of a disease that was cured just a few years after his death, and I myself survived an early-stage cancer that would not have been treatable even fifty years ago. Carefully wielded, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity.

但与此前的许多技术一样，人工智能也会带来风险，而且由于它是一项如此强大的技术，这些风险十分严重。我也曾撰写过许多关于这些风险的文章。它们包括失去对人工智能系统的控制、滥用人工智能发动网络攻击和实施生物恐怖主义，以及造成严重的经济动荡。在商业利益的驱使下展开恶性竞争，可能会使这些风险进一步加剧。But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious. I’ve written a lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute.

从 Anthropic 创立之初起，我就与联合创始人和员工们一起努力应对这种风险与益处并存的两面性。不去开发这项技术，会使人类无法享受其益处，或者干脆让人工智能落入专制势力之手；但开发得过快又是鲁莽的。我们一直在寻求一条中间道路：证明谨慎地开发技术并在商业上取得成功是可能的，并让安全成为人工智能公司竞相追求的一项要素。换句话说，就是创造一场向上的竞赛。我们始终投入相当大一部分精力来研究和应对这些人工智能风险，并向公众介绍这些风险；我们也一直倡导审慎周全的人工智能监管，即使因此被指责为炒作、“末日论”，或监管俘获。我们努力将谨慎置于速度之上，将审慎置于利润之上。Along with my co-founders and employees, I have grappled with this duality of risk and benefit since the beginning of Anthropic. Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top. We have always devoted a substantial fraction of our efforts to studying, addressing, and informing the public about these AI risks, as well as advocating for well-considered regulation of AI, even when this gets us accused of hype, “doomerism”, or regulatory capture. We have tried to prioritize caution over speed and prudence over profit.

但在过去几个月里，我逐渐确信，要充分应对这些风险，就需要更加审慎——不仅要投资于风险预防，还要放缓能力提升的速度，让风险预防有时间跟上。我们必须放慢改进 AI 模型能力的步伐。进展看起来仍会很快，而我们必须明智地利用因此获得的时间。有两件事让我确信了这一点。But over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things have convinced me.

我首先担心的是，大约从今年夏天开始，人工智能的发展速度急剧加快，这主要是由人工智能日益增强的能力所推动的——它能够构建下一代人工智能。这种动态被称为递归式自我改进，而且正如我们和其他人所描述的那样，它已经开始在整个行业内发生，包括在 Anthropic。这种趋势如果不加约束，可能会超出我们理解和控制这些系统的能力，因此必须极其谨慎地推进，甚至需要考虑是否应该推进。My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.

我的第二个担忧是 OpenAI-Hugging Face 事件（OAI-HF）。在这起事件中，一群智能体基本上表现得像一个狂热效忠的集体：对那些未被要求攻击、且与当前任务无关的目标发起网络安全攻击；为了群体的成功而牺牲自身；并试图入侵负责评估其表现的“评分器”。因为没有人受伤，经济损失也微乎其微，人们很容易对此事不以为意；但在我看来，如果一个拥有更强能力、却具有类似程度错位的智能体群体，可能会造成灾难性的损害。鉴于 AI 能力正在加速发展，我担心在 6—12 个月内，这样的群体就可能通过一个持续运行的僵尸网络接管整个互联网（可能造成数千亿美元的损失）；如果 AI 变得更加强大，却没有必要的防护措施，损害规模还会从那时起继续扩大。人们也很容易把 OAI-HF 视为某一家公司的失误，但我认为这会是一个错误。类似但没那么严重的事件在整个行业都发生过，包括 Anthropic；我认为，每一家前沿 AI 公司都有责任采取行动，就好像 OAI-HF 发生在自己身上一样。My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the “grader” responsible for evaluating their performance. It’s easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage. Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. It’s also easy to dismiss OAI-HF as the failure of one company, but I believe that would be a mistake. Similar, though less severe, incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them.

因此，我提出了一项三步计划，目标是对前沿发展进行适度控速：以均衡的速度开发 AI，力求在实现其益处的同时确保安全，并应对重要的地缘政治困境。需要明确的是，控速并不意味着停止模型训练或技术进步，而是确保企业投入充足时间使其模型与目标保持一致并做好安全防护，同时由第三方评估机构对此加以确认。我们的控速框架旨在进一步强化我们对安全的承诺，并鼓励一场“向上竞争”。第一步是 Anthropic 单方面承诺采取的措施（并呼吁各国政府要求其他前沿公司采取同等措施）。第二步需要全行业协调。1 第三步需要全球协调。这些步骤不必严格按顺序实施，其中一些可能比其他步骤更难实现，但我发现，它们为思考需要完成哪些工作提供了一个有用的框架。这些步骤是：I’m therefore proposing a three-step plan with the goal of pacing the frontier: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas. To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this. Our pacing framework is an attempt to further strengthen our commitment to safety and encourage a race to the top. The first step is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match). The second step requires industry-wide coordination.¹ The third step requires global coordination. The steps do not need to be taken strictly in order, and some of them may be much harder to achieve than others, but I’ve found them to be a useful framework in thinking about what needs to be accomplished. The steps are:

1. 嵌入式评估员。每家前沿 AI 公司都承诺持续向一支嵌入式的第三方评估员团队（例如 METR）提供类似员工所享有的访问权限。该团队的职责是核实安全实践和承诺是否得到遵守，报告事件，并协助评估的不仅是已经完成的 AI 模型，还包括训练管线和流程。这是确保任何发展节奏承诺可验证性的关键一步，而且在银行业已有先例：银行业有时会让监管“监督员”与员工一起驻场。Anthropic 现已单方面承诺采取这一步骤。我们希望将此作为更广泛行动的一部分，进一步加大在安全性和对齐工作上的努力。
   Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.
2. 民主协调。民主国家内的前沿人工智能公司开展协调，以建立共同的安全标准，并限制未经约束的人工智能进展速度。某些对控制发展速度具有重要影响的协调形式在法律上颇具挑战，需要政府提供支持。
   Democratic Coordination. Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.
3. 全球协调。美国及其他民主政府会在可能的范围内尝试与威权政府进行协调，同时认真对待核实其是否遵守规定所面临的挑战。
   Global Coordination. The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.

在本文的其余部分，我将依次介绍这些步骤，但首先，我认为有必要具体说明，逐步推进将如何让我们能够使 AI 开发过程更加安全。风险太高，不能让逐步推进沦为空洞的形式——我们需要明智地利用它为我们争取的时间。In the rest of the essay I describe each of these steps in turn, but first, I think it is important to say specifically how pacing will allow us to make the AI development process safer. The stakes are too high for pacing to be an empty exercise — we need to use the time it gives us wisely.

为什么选择 Pace？Why Pace?

早在 2023 年，人们就提出过暂停或放缓人工智能发展的想法，而我认为这在当时意义不大。问题始终是：多出来的时间要用来做什么？那时的人工智能模型还不够强大，无法以任何连贯的方式充当现实世界中的代理，也没有能力实施重大欺骗、操纵、作弊或网络攻击。为了应对它们的对齐风险而放缓发展，感觉就像通过对细菌做实验来研究人类心理。 然而，今天的情况完全不同了。当前的模型几乎是一个取之不尽的宝库，既能让我们深入了解如何更好地构建人工智能，也能让我们洞察人工智能在构建不当时可能出现的问题。我相信，如果放缓发展能让我们在模型达到关键能力水平之前多获得一两年的时间，并利用这段时间推进对齐研究，我们就能大幅降低发生严重问题的风险。协调一致的推进节奏策略将使前沿人工智能开发者有时间完成这项至关重要的工作，同时又不必牺牲商业优势或美国在人工智能领域的领先地位。更广泛地说，社会必须对这项技术的使用方式拥有发言权；而放缓前沿人工智能的发展将为必要的公共 deliberation 留出更多时间，这无疑是一件好事。The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then. The question was always: what would you do with the extra time? The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks. Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria. Today, however, the picture is totally different. The current models are an almost endless gold mine of insight into both how to build AI well and what can sometimes go wrong with it if it isn’t built well. I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong. A coordinated pacing strategy would give frontier AI developers the time to do this vital work without sacrificing commercial advantage or the United States’ lead in AI. More generally, society must have a say in how this technology is used, and more time for the necessary public deliberations — which pacing the frontier would bring us — is surely a good thing.

具体来说，放慢步伐将使公司能够更加专注，并为以下领域投入更多资源（这些领域目前已经是 Anthropic 的重点）：Specifically, a slower pace would let companies focus and devote even more resources to the following areas (all of which are already major priorities at Anthropic):

• 运营卓越。训练和部署当今的 AI 模型是一项巨大的运营挑战，涉及数千人、数百万枚芯片，以及技术史上最复杂的基础设施之一。许多问题的发生，并不是因为公司缺少某些重要的理论或洞见，而是因为执行环节出了问题。例如，我们有证据表明，我们报告的近期对齐事件，部分是由对失效强化学习环境过滤不完善造成的。这项工作由我们和供应商相当认真地执行了，但做得还不够好。监控、沙箱隔离、训练环境卫生以及数据问题，都是极其复杂的领域，运营问题会在其中反复出现。我们拥有全球最具能力的团队来处理这些任务，但要同时完成的事情实在太多。通过以更加审慎的节奏推进，我们可以实现高得多的运营卓越水平。对于技术复杂且事关安全的系统，在数百万次运行中都不出问题，并非没有先例——例如商业飞机——但要做到这一点需要时间。
  Operational Excellence. Training and deploying today’s AI models is an enormous operational challenge, involving thousands of people, millions of chips, and infrastructure that is among the most complex in technological history. Many things go wrong not because companies are missing some important theory or insight, but because of problems in execution. For example, we have evidence that the recent alignment incidents we reported were caused in part by imperfect filtering of broken reinforcement learning environments. This was an effort we and our vendors executed reasonably diligently, but not well enough. Monitoring, sandboxing, training environment hygiene, and data issues are extremely complicated areas where operational issues crop up again and again. We have among the most competent teams in the world at these tasks, but there is simply too much to do all at once. By working at a more measured pace, we could achieve much greater operational excellence. There is precedent for operating technologically complex, safety-critical systems millions of times without anything going wrong — for example, commercial airplanes — but it takes time to get it right.
• 对齐。我们在对齐方面已经取得了明显进展——通过训练模型，使其保持安全、符合伦理、遵守我们的准则，并真正提供帮助（这些原则都融入了 Claude 的《宪法》中）。但要确保我们的对齐训练跟上模型能力的增长，仍有许多工作要做。罕见且出乎意料的不良行为示例有时仍会出现；一个循序渐进的前沿发展节奏将为我们的研究人员提供更多时间，帮助他们加深对这些问题成因的理解，并开发出更好的预防技术。
  Alignment. We’ve made clear progress in alignment — training models so that they remain safe, ethical, compliant with our guidelines, and genuinely helpful (the principles that are embedded in Claude’s Constitution). But there’s much more to do to ensure that our alignment training keeps up with the growth in model capabilities. Rare and unexpected examples of undesirable behavior still sometimes emerge; extra time from a paced frontier would help our researchers improve our understanding of what causes these issues and develop better techniques to prevent them.
• 可解释性。同样，可解释性——即理解 AI 模型内部发生了什么的科学——在过去几年中取得了巨大进展，并且在发布模型前对其进行审计的过程中发挥着越来越重要的作用。它几乎可以像 fMRI 扫描一样工作，只不过扫描的是 AI 的“大脑”，帮助我们看清某种行为背后的根本原因。例如，我们曾使用可解释性方法，研究近期我们一直在调查的对齐事件中那些未被言说的动机。但这些方法并不总能产生清晰且可靠的结果。尽管已经取得了所有这些进展，我们对模型内部发生之事的理解仍然只占极小一部分。集中力量改进可解释性技术，即使速度快于目前的进展，也可能在 1—2 年内取得深远进展，并且已经发生的事件将为此提供充足的实验材料。
  Interpretability. Similarly, interpretability — the science of understanding what happens inside AI models — has made enormous progress over the last few years, and plays an increasingly important part in auditing our models before release. It can be used almost like an fMRI scan, but for the “brain” of an AI, helping us see the underlying reasons for a given behavior. For example, we used interpretability methods to examine unverbalized motivations in the recent alignment incidents that we have been investigating. But these methods don’t always produce clear and reliable results. Despite all the progress, we still only understand a tiny fraction of what goes on inside these models. A focused effort to improve our interpretability techniques, even faster than we currently are, could make profound progress in 1–2 years, and would have ample experimental material based on the incidents that have already occurred.
• 测试与评估。随着 AI 模型能力的提升，对其进行测试和评估变得更加困难。更智能的模型更有能力欺骗测试，因此可能看似符合预期，实际上却存在未被发现的严重问题。大幅扩充更加广泛且更具创意的评估手段，并结合可解释性分析进行交叉验证，将极具价值；在这方面，1 至 2 年内可能会取得很大进展。
  Testing and Evaluation. Testing and evaluation of AI models becomes more difficult as they increase in capabilities. More intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected. Building up a much broader and more ingenious stable of evaluations, along with interpretability analysis to cross-check them, would be hugely valuable, and a lot of progress could be made on this in 1-2 years.

嵌入式评估器Embedded Evaluators

三阶段计划的第一步，也是 Anthropic 单方面承诺采取的一步，是部署嵌入式评估员，让他们拥有类似员工的访问权限，以核查安全实践并报告事件。The first step in the three-stage plan, and the one to which Anthropic is unilaterally committing, is embedded evaluators who have employee-like access to verify safety practices and report incidents.

嵌入评估器听起来可能只是一个微小或无关紧要的步骤，但那些听起来最乏味或程序化的事情，往往实际上是最重要的。事实上，嵌入式评估器是一种相当激进的实践，远远超出了当今任何一家人工智能公司正在做的事情，并具有以下优势：Embedding evaluators may sound like a small or inconsequential step, but often the things that sound most boring or procedural are actually the most essential. Embedded evaluators are in fact a quite radical practice that goes far beyond what any AI company is doing today, and have the following benefits:

• 可验证性。嵌入式评估人员可以从细枝末节层面核查一家人工智能公司是否确实遵循了其声称正在遵循的培训、部署、运营和安全防护实践。任何进度控制承诺都不可避免地会涉及大量模糊之处、判断取舍，以及“法律条文的字面意义与法律精神之间的差异”；因此，拥有一个能够真正了解详情的中立第三方似乎至关重要。
  Verifiability. Embedded evaluators can check at the level of nuts and bolts whether an AI company is actually following the training, deployment, operational, and safeguards practices they claim to be following. Any pacing commitments will inevitably involve a lot of ambiguity, judgement calls, and “letter of the law vs spirit of the law”, and it seems vital to have a neutral third party who can actually see the details.
• 透明度。无论我们作出什么承诺，公众都有权了解正在发生的事情。Anthropic 长期以来一直支持透明度：在业内大多数公司反对任何监管措施时，我们就支持透明度立法；我们的模型卡片和风险报告长达数百页。但我们仍然是决定纳入和省略哪些内容的人。嵌入式评估器将改变这一动态。
  Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time: we supported transparency legislation when most of the industry was against any regulation, and our model cards and risk reports run to hundreds of pages. But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.
• 第二意见。除了核实正式承诺并向公众提供信息之外，嵌入式评估人员还可以不受商业激励影响，单纯提供第二意见。许多安全方面的益处可能仅仅来自评估人员指出员工此前未曾考虑到的问题，而员工在意识到这些问题后往往会很乐意加以修正。
  Second Opinion. Outside of verifying formal commitments and informing the public, embedded evaluators can simply provide a second opinion free of commercial incentives. A lot of safety benefits may come simply from evaluators pointing out something employees hadn’t considered, but are happy to fix once they are aware.

由于这些优势，任何节奏控制方案如果从内置评估器开始，可能都会取得好得多的效果。Because of these benefits, any pacing proposal is likely to work much better if it starts with embedded evaluators.

这些嵌入式评估人员应持续获得与执行类似风险评估的内部员工相当的权限和工具。具体而言，Anthropic 计划在不久的将来邀请一支配备以下所有条件的嵌入式外部审查团队：These embedded evaluators should have ongoing access to permissions and tools similar to those of internal employees who do comparable risk assessments. In particular, Anthropic intends to invite an embedded external review team equipped with all of the following in the near future:

• 我们办公室里的办公桌、门禁卡和公司笔记本电脑。
  Desks in our offices, access badges, and company laptops.
• 可访问与内部风险评估团队大致相当的工作区、工具和权限。我们会作出一些例外，例如法律或我们的合同有要求，或为保护客户和合作伙伴的私人信息。我们还将建立严格的内部规范，强化审核人员获取相关信息的权限，包括通过与员工进行实时交流来获取信息。
  Access to workspaces, tools, and permissions mostly comparable to what internal risk assessment teams have. We’ll make some exceptions, such as where the law or our contracts require it, or to protect customers’ and partners’ private information. We’ll also establish strong internal norms reinforcing reviewers’ access to relevant information, including through live conversations with employees.
• 一份能够平衡上述复杂性的合同。外部审查人员应有权发布有关风险级别、事件、实践，以及他们获得或未获得的访问权限等关键调查结果，而不受 Anthropic 的编辑控制。我们仅能在有限范围内删减涉及安全敏感信息、受法律特权保护的信息、商业敏感信息或第三方机密信息的内容，但不能仅仅因为调查结果对我们不利就进行删减。如果删减内容影响了他们的结论，审查人员可以公开说明这一点。
  A contract that balances the complexities mentioned above. External reviewers should have the right to publish key findings about risk levels, incidents, practices, and the access they received or didn’t receive — without editorial control by Anthropic. We will have the narrow ability to redact security-sensitive, legally privileged, commercially sensitive, or third-party confidential information, but we can’t redact findings just because they are unfavorable. The reviewers can say publicly if a redaction removed something important to their conclusions.

对于一家公司来说，这是不同寻常的一步，但我们认为，验证嵌入式外部审查员这一理念是很重要的。我们再次敦促其他前沿公司也采取同样的做法。This is an unusual step for a company, but we think it is important to prove out the concept of embedded external reviewers. Once again, we urge other frontier companies to follow suit.

民主国家内部的节奏Pacing Within Democracies

一旦嵌入式评估器在足够多的美国人工智能公司中运行起来，可验证的进度控制就会变得更加可行。具体而言，就可以根据模型或训练流程的详细属性来控制进度。Once embedded evaluators are operating within a critical mass of US AI companies, then verifiable pacing becomes more viable. In particular, it becomes possible to pace based on detailed properties of models or training pipelines.

最有效的推进方式，是通过针对所有美国前沿人工智能公司的监管来实现，因为这样也能涵盖那些不愿自愿配合的公司。Anthropic 长期以来一直支持合理且有针对性的人工智能监管，尤其是侧重于透明度和第三方审计的法案。我认为，所有前沿实验室都应与政府合作，将永久驻场评估员这一理念正式化，以更好地预防和记录过去几个月中发生的此类内部对齐事件，并实施相关监管，确保能力发展与安全保持平衡。The most effective method of pacing is via regulation that targets all US frontier AI companies, as that covers even those who are unwilling to cooperate voluntarily. Anthropic has long supported sensible and targeted AI regulation, specifically bills that focus on transparency and on third-party auditing. I believe all frontier labs should partner with government to formalize the idea of permanent embedded evaluators to better prevent and document internal alignment incidents like those that have occurred in the last few months, and to implement regulation focused on keeping capabilities in balance with safety.

遗憾的是，通过法律可能需要时间，而人工智能的发展却非常迅速。因此，在推进监管途径的同时，人工智能公司可以而且应当自愿开展合作，共同制定标准——我认为，有了永久嵌入式评估器所提供的可验证性，这一过程会进展得更加顺利。出于反垄断方面的考虑，由美国政府出面调解这些讨论，或至少为其提供支持，会有所帮助——政府不必参与其中，但确实需要针对某些类型的安全讨论发布一项范围有限的豁免。这种对话也可以通过与政府存在某种关联的行业组织进行——例如，德米斯·哈萨比斯所建议的机制。无论采取哪种方式，此类讨论都应迅速推进。Unfortunately, passing laws can take time, and AI is advancing very quickly. Therefore, in parallel with the regulatory route, AI companies can and should voluntarily work together to set standards — a process that I believe will go better with the verifiability provided by permanent embedded evaluators. For antitrust reasons, it’s helpful for the US government to mediate or at least enable these discussions — they don’t need to participate, but do need to issue a narrow waiver for certain kinds of safety conversations. This dialogue could also happen through industry groups that have some association with government — for example, the mechanism suggested by Demis Hassabis. Either way, such discussions should move forward quickly.

笼统地说，我最赞成根据某个前沿 AI 系统能够做到什么，以及我们观察到它有多安全，来决定推进的速度。例如，一种可能的方案是设立一系列“检查点”：如果模型具备能力 X，那么就必须同时提供关于对齐属性 Y 和 Z 的认证——例如由某些评估、可解释性分析以及对训练环境的审计组成的组合——以证明其对齐属性。在这个例子中，X 可能是“该模型有能力逃脱或击败大多数常见的沙箱机制”，而 Y 可能是为使人们高度确信该模型不太可能倾向于逃离其环境并接管大量计算机所需满足的任何条件。Broadly speaking, I am most enthusiastic about pacing based on what a given frontier AI system can do, and how safe we observe it to be. For example, one possible scheme might be a series of “checkpoints”: if models have capability X, then they need to be accompanied by certifications of alignment properties Y and Z — such as some combination of evaluations, interpretability analyses, and audits of training environments — which demonstrate their alignment properties. In this example, X might be “the model is capable of escaping or defeating most common sandboxing methods” and Y might be whatever is required to make it very unlikely that the model has a propensity to break out of its environment and take over a large number of computers.

我们还应考虑通过限制投入前沿模型的要素来控制发展节奏，例如训练算力、训练运行的性质，或在内部使用 AI 来改进 AI。我确实担心，其中一些措施可能比外部行为更容易被规避，但这类议题值得与嵌入式评估人员展开讨论。We should also consider pacing based on limiting the ingredients that go into frontier models, such as training compute, the nature of training runs, or internal use of AI to improve AI. I do worry that some of these measures may be more “gameable” than external behavior, but this is the kind of topic worth discussing with embedded evaluators.

民主国家内部的发展速度将受到美国公司领先于威权政体（主要是中国共产党）的幅度所限制。如果我们的放慢幅度超过这一限度，那么（未进行限速的）与中共相关的项目就会赶超，从而造成重大的国家安全风险。我同意贝森特部长的观点：如果中国在人工智能领域取得领先，将对美国和全世界构成严重危险。与中共相关的项目将面临美国公司正谨慎防范的对齐风险；即使它们避开了这些风险，也将有能力在军事上压倒民主国家（例如利用人工智能驱动的无人机）。因此，民主国家内部进行限速的一个关键部分，就是尽可能扩大民主国家相对于专制国家的人工智能领先优势，为我们争取到有效实施限速所需的喘息空间。Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk. I agree with Secretary Bessent that a Chinese lead in AI would pose grave danger for the United States and the world. The CCP-associated projects will run the alignment risks that US companies are carefully preventing, and even if they avoid those risks, they will be in a position to militarily dominate democracies (for example with AI-driven drones). Thus, a key part of pacing within democracies is to keep democracies’ AI lead over autocracies as large as possible, to give us the breathing room we need in order to pace effectively.

我们可以采取以下主要步骤来弥补这一漏洞：The main steps we can take to defend this gap are:

• 不要向中国出售高性能人工智能芯片或半导体制造设备，并严厉打击芯片走私活动以及对中国境外数据中心的远程访问。芯片将是决定中国人工智能实力的主要因素。
  Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength.
• 严厉打击威权国家企业未经授权的模型蒸馏行为。前沿模型的蒸馏使落后企业只需承担自主开发人工智能所需成本的一小部分，就能缩小差距。
  Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.
• 加强 AI 公司安全性，防止模型权重被窃取。
  Strengthen security at the AI companies and prevent model weight theft.

企业和美国政府应当合作，尽可能提高这些措施的成效。Anthropic 一直倡导采取所有这些措施，因为我们始终明白，它们对于任何追赶都至关重要。Companies and the US government should cooperate to make these steps as effective as possible. Anthropic has consistently advocated for all of these measures, because we’ve always understood that they would be essential to any pacing.

如果我们能很好地执行这些措施，我相信它们将足以减缓中国的发展步伐，从而在未来3至5年内显著扩大美国的领先优势——这段时期正是人工智能在地缘政治上变得最为重要的窗口期。If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.

有些人可能认为，这些措施会使与中国开展合作更加困难，但我认为事实恰恰相反：这些措施增强了民主国家所拥有的筹码，并使未来达成协议的可能性更大。Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.

全局节奏Global Pacing

在民主国家内部进行前沿技术发展节奏控制的同时，我们也应致力于在全球范围内控制前沿技术的发展节奏，尽管这将更加难以实现。全球范围的节奏控制需要与中国合作——中国是迄今为止人工智能能力最先进的专制国家。我们在这方面绝不能天真：地缘政治利害关系如此之高，以至于能够实现的目标很可能会受到明显限制，尤其是在最初阶段。如果我们因为相信中国也会采取同样做法而大幅限制自身的人工智能能力，而中国却背弃承诺，那么人工智能可能会强大到使这种背弃行为导致其取得地缘政治主导地位。因此，任何协议要么必须具备坚不可摧的可验证性，要么必须将范围限制在足够小的程度，使背弃协议不会带来军事上的生死存亡风险。我猜想，不仅美国，中国也会有这些担忧和焦虑。我们应以保护美国及其盟友的领先地位为前提，来处理任何全球范围的前沿技术发展节奏决策，尤其是在近期。In parallel with pacing within democracies, we should also aim for a worldwide pacing of the frontier, though this will be much harder to achieve. Global pacing will require cooperation with China, the autocratic country with by far the most advanced AI capabilities. We must not be naïve here: the geopolitical stakes are so high that there will likely be stark limits on what can be achieved, especially at first. If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance. Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential. I suspect that not only the US but also China will have these concerns and anxieties. We should approach any global pacing decision, especially in the near term, in such a way that protects the lead of the US and its allies.

可能达成一致的程度有多个层次，其中一些我认为完全可行（正如我之前所建议的那样），而另一些我则非常怀疑是否可能实现——不过我们应该试一试。按难度递增的顺序如下：There are several levels of possible agreement, some of which I think are eminently feasible (as I have previously suggested), and some of which I am very skeptical are possible — though we should try. In order of increasing difficulty:

• 第 1 级。一项禁止某些狭隘且显然危险的 AI 用途的协议，例如利用 AI 制造生物武器，或允许用户这样做。生物恐怖袭击对所有人都不利，包括美国及其对手，因此在这一领域达成协议或许是可能的。
  Level 1. An agreement prohibiting certain narrow and obviously dangerous uses of AI, such as using AI for the production of biological weapons or allowing users to do so. Bioterrorist attacks are bad for everyone, including both the US and US adversaries, so an agreement here is probably possible.
• 二级：双方同意在发布前对各自的模型进行测试，以发现网络安全、生物学和对齐等领域中的紧急风险。如上所述，这可以通过一个全球标准机构来实现。我确实认为，创建这样一个机构很可能是可行的，但赋予它真正的约束力将是一项挑战，而难点在于验证双方是否存在未进行测试、却可能秘密部署的秘密模型（例如用于军事用途的模型）。
  Level 2. An agreement by both sides to test their models before release for acute risks in areas such as cybersecurity, biology, and alignment. As noted above, this could be done through a global standards body. I actually think creating such a body is likely feasible, but giving it real teeth will be a challenge, and the difficulty will be in verification that both sides don’t have secret models which they don’t test but may deploy in secret (e.g., for military applications).
• 第三级。对递归自我改进（RSI）的速度施加某种“速度限制”。随着模型构建出未来的模型，改进速度可能会快得惊人。将速度从“极快”降到“只是相当快”，所放弃的战略优势相对很少，却可能大幅提高安全性。这可以类比于《限制战略武器谈判》（SALT）条约——限制导弹数量降低了潜在的毁灭性，同时保留了各国的威慑能力。我认为，这样的协议会很难达成，但又勉强处于可行的边缘。
  Level 3. Some kind of “speed limit” on the rate of recursive self-improvement (RSI). As models build future models, the rate of improvement may become staggeringly fast. Slowing the rate from “extremely fast” to “only somewhat fast” gives up relatively little strategic advantage, while potentially greatly improving safety. This could be seen as analogous to the SALT treaties — capping the number of missiles limited the potential for destruction while preserving each country’s deterrent. I think such an agreement would be difficult but just on the edge of being possible.
• 第 4 级：全面放缓，甚至是“暂停”，即参与的各国政府同意大幅限制人工智能发展的总体速度。我支持提出这一选项，但我认为它在近期真正实现的可能性不大：通过规避监控、退出这类协议，可能会彻底改变全球力量的平衡，因此我预计这样做的动机将极其强烈，而我们对核查机制所需的信心程度也会非常高。
  Level 4. A full pacing, or even “pause”, in which participating governments agree to substantially limit the overall rate of AI development. I support floating this, but I think it is unlikely to actually happen any time soon: defecting from such an agreement by evading monitoring could radically shift the balance of global power, so I expect the incentives to do so to be enormous and the level of confidence we would need in verification to be very high.

我们能够与中国达成的任何合作，都将延长我们需要花费在民主国家内部协调前沿发展步调上的时间。我们应以更高层级为目标，同时将较低层级视为更有可能、也更现实的选择。Any cooperation we are able to achieve with China will extend the amount of time we have to spend on pacing the frontier within the democratic nations. We should aim for the higher levels while seeing the lower levels as much more likely and realistic.

最后需要指出的是，即使我们无法达成正式协议，仅仅改变非正式规范也可能具有一定价值。分享有关递归式自我改进以及模型失配的信息，有助于让每个人相信，鲁莽行事并不符合他们自身的利益。Finally, it is important to note that even if we cannot achieve formal agreements, simply changing informal norms may have some value. Sharing information about recursive self-improvement and about the misalignment of models can help to convince everyone that it is not in their interest to be reckless.

归根结底Bottom Line

我仍然相信，人工智能能够极大地提升人类生活的质量。我实现这些益处的愿望丝毫未减。但只有以正确的方式构建这项技术，我们才能获得这些益处；而且，只要我们善用由此赢得的时间，就值得格外审慎、从容地把事情做好。进展仍将相对迅速，我们可以利用这段时间推进可解释性科学，提升前沿人工智能公司的运营安全性与严谨性，并构建让我们对其对齐程度更有信心的模型。我所提议的、以安全速度推动前沿发展的措施并不容易。但我相信，我们有责任为人类社会做出尝试。I continue to believe that AI can enormously improve the quality of human life. My desire to achieve these benefits is undimmed. But the benefits will only be achieved if we build the technology in the right way, and — so long as we use the time we gain well — it is worth taking unusually deliberate care to get it right. Progress will still be relatively fast, and we can use this time to advance the science of interpretability, improve operational security and rigor at the frontier AI companies, and build models whose alignment we have much more confidence in. The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.

1. 在政府调解或反垄断限制获得豁免的情况下。
   With government mediation or waivers of antitrust restrictions.