异类心智

2023年年中，在“RLSlow”研究项目中，我们看到了首批结果，这些结果让我们有信心相信，我们能够扩大推理模型的训练规模，释放预训练模型形成自身思维链的能力。那天夜里，我和 Szymon 待在办公室里，思考的并不是这项技术将带来的惊人基准测试成绩、产品或科学成果，而是试图消化一个令人清醒的事实：在我们有生之年，我们真的会看到比我们聪明得多的机器，而我们已经看到了这些系统的大致形态；我们思索着该如何让人们意识到这一点的重要性。In mid-2023, within the “RLSlow” research project, we saw the first results that gave us confidence that we will be able to scale the training of reasoning models, unlocking the capability of pretrained models to form their own chains of thought. Szymon and I spent that night at the office, thinking not about the incredible benchmark numbers, products, or scientific results that this technology will deliver - but rather, trying to process the sobering fact we will actually see machines meaningfully smarter than ourselves in our lifetime, and we already see the shape of these systems; wondering how to alert people to the significance of this.

三年后，推理型语言模型已成为经济中快速增长的一部分，并开始拓展科学的边界。它们能够操作计算机和图形界面，与人类及彼此协作，并开展研究项目。它们也正在改变计算机安全领域的格局，并由此带来明显的新危险。Three years later, reasoning language models are a rapidly growing part of the economy and starting to push the boundaries of science. They are able to operate computers and graphical interfaces, collaborate with people and each other, and carry out research projects. They are also transforming the landscape of computer security, and in that present clear new dangers.

这一时期涌现了大量新的研究成果，我们对这些系统的理解也再次发生了变化，与 2023 年时有所不同。根据内部研究结果，我强烈预期，这种进步速度有可能持续下去，直至实现递归式自我改进。如果 AI 的发展沿着当前轨迹继续推进，那么未来几年我们将看到的系统，很可能会在能力上实现幅度相当甚至更大的进一步跃升，并且越来越多地推动自身的发展。A lot of new research happened in this period, and our understanding of these systems is again a little different than it was in 2023. Based on internal results, I have a strong expectation that this speed of progress could be sustained into recursive self-improvement. If AI development continues along its current path, the systems we’ll see in the next few years are likely to represent further capability jumps of equal or larger magnitude, and to increasingly drive their own development.

这是一个需要极度谨慎的时刻。我担心，没有人做好准备应对机器智能持续快速提升所带来的后果。OpenAI 将继续寻求解决对齐和监控问题的技术方案，构建防御系统，并在必要时单方面暂停进一步扩大规模；然而，我认为还需要更广泛的干预措施。This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence. OpenAI will continue to seek technical solutions to alignment and monitoring, to build defensive systems and unilaterally withhold further scaling as needed; however, I believe broader interventions are required.

从宏观层面看，机器智能的进步是由计算能力的提升推动的。2017 年前后，在多个研究项目中看到规模扩大持续带来收益后，我们 OpenAI 深刻认识到了这一点¹。因此，我们开始寻求远超原计划的计算资源，并逐渐将研究重心转向少数几个可大规模扩展的方向。我们认为，这是我们跻身 AI 研究前沿并影响通用人工智能（AGI）所带来影响的唯一途径。At a high level, progress in machine intelligence is driven by increasing computational power. We at OpenAI deeply internalized this around 2017, after seeing consistent returns to scaling across multiple research projects¹. As a result, we sought out access to much more compute than we had originally planned, and increasingly oriented our research around a small number of very scalable directions. We believed that was the only way for us to be at the frontier of AI research, and influence the impacts of AGI.

一路上涌现了新的算法，也有团队和个人研究者展现出的新颖巧思。我大体上将它们视为规模化进程中的发现；深度学习科学仍处于起步阶段，而有意义的算法进步往往与算力资源的获取相关。从数年的时间跨度来看，随着人工智能在更大规模的计算机上扩展，它正变得越来越智能。There are new algorithms that have been developed along the way, new feats of ingenuity from teams and individual researchers. I see them largely as discoveries along the path of scaling; the science of deep learning is still nascent, and meaningful algorithmic progress tends to correlate with access to compute. If you zoom out to a multiple-year horizon, AI is continuing to become more intelligent as it is scaled to larger computers.

而且，正如雷·库兹韦尔在20世纪末所做的预测（在新窗口中打开），我们如今正处于计算史上的一个时刻：机器智能开始以变革性的方式超越人类智能。And, in line with Ray Kurzweil’s predictions from the end of the XXth century⁠(opens in a new window), we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.

AI与其说是设计出来的，不如说是生长出来的——粗略地说，它是通过在难以想象的计算资源上反复执行一个简单的优化步骤而产生的。这造就了一个极其复杂的系统，它能处理抽象概念，并模拟人类行为的某些方面。我们可以发现这个系统内部涌现出的各种微观机制，类似于神经科学的研究；同样，系统整体的运作方式也难以用我们能够完全理解的描述来阐明。AI is grown more than designed - it is, to first degree, the product of repeating a straightforward optimization step many times on a hard-to-imagine amount of compute. This results in an incredibly complex system that works through abstract concepts and can simulate facets of human behavior. We can discover various insights about little mechanisms that emerge within this system, in a process similar to neuroscience - and, similarly to neuroscience, its overall action evades a description we can fully understand.

基于深度学习的人工智能研究在很大程度上是一门实验科学。我们投入大量精力来构建有原则的算法并提出可检验的预测，但从根本上说，大规模训练都是实验，我们有时也会对实验结果感到意外。此外，随着系统能力不断增强，其结果也越来越难以解读。The study of deep learning-based AI is largely an experimental science. We put a lot of effort⁠ into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret.

当前的算法通常会更快提升那些易于衡量的能力，而较难客观量化的能力提升得更慢，这使情况变得更加复杂。我们花费大量时间试图理解能力如何泛化，以及应优先推进哪些技能，才能提升未来几年最相关的能力。例如，我们认为，只要进一步投入，就能让模型在数学研究方面表现得更好，但我们并未优先发展这一方向，因为我们感到 RSI 和自动化对齐研究迫在眉睫，正如我稍后将要讨论的那样。This is made more complicated by the current algorithms generally improving easy-to-measure capabilities faster than those hard to objectively quantify. We spend a lot of time trying to understand how capabilities generalize, and what to prioritize to advance the skills that are going to be most relevant in the next few years. For instance, we believe we could make the models better at specifically mathematics research with additional focus, but we do not prioritize this direction because of the urgency we feel about RSI and automated alignment research, as I will discuss later.

通过扩大深度学习的规模所产生的智能，无法直接与人类智能相提并论。要在现实世界中变得举足轻重——极其有用或极其危险——人工智能并不需要匹配或超越人类的所有能力；它只需要超越其中足够多的能力。而随着它在越来越多的维度上超越人类，我们也越来越难以准确了解它究竟有多强。The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.

由于机器智能的产生过程与人类智能有着根本性的不同，我们不能想当然地认为它默认会遵循人类原则，或以类似人类的方式从这些原则中归纳推广。人工智能研究的核心问题是对齐——让 AI 按照人类的标准“努力做正确的事”。Because machine intelligence comes from a fundamentally different process than human intelligence, we cannot assume it adheres to human principles by default, or generalizes from them in a human-like manner. The core problem in AI research is that of alignment - getting the AI to “try to do the right thing” by human standards.

为了组织实用的研究方向，我认为区分目标对齐与价值观对齐很有帮助。For the purpose of organizing practical research directions, I find it useful to distinguish goal alignment and value alignment.

目标对齐，广义上指的是：“AI 是否会努力实现为其设定的目标？”这可能包括遵循指令层级，或与人沟通协作、努力理解他们的目标等能力。这一系列方向在实践中一直极具现实意义。Goal alignment is broadly: “does the AI try to accomplish the goal set before it?”. This can include things like adherence to an instruction hierarchy⁠, or the ability to communicate and collaborate with people, to attempt to understand their objectives. This set of directions has been extremely practically relevant.

价值观对齐是模型更内在的一种属性。它指的是模型能够坚持一套高层次原则，并从中举一反三；即使面对不明确或相互冲突的目标，或身处陌生乃至对抗性情境中，也能采取“合乎情理”的行动。一个实现了价值观对齐的 AI 应当诚实正直，并热爱人类。Value alignment is a more intrinsic property of the model. It is the ability to hold and generalize from a high-level set of principles; to act “reasonably” even when given unclear or conflicting objectives, or placed in unfamiliar or adversarial situations. An aligned AI should act with honesty and integrity, and love for humanity.

当然，价值对齐与目标对齐之间的界限可能并不清晰，而真正关心目标就需要尝试推断其背后的意图（在新窗口中打开）和价值观。不过，一般来说，当我谈到对齐研究的长期重要性时，我指的是价值对齐。Of course, the boundary between value and goal alignment can be blurry, and truly caring about goals requires attempting to infer the intent⁠(opens in a new window) and values underlying them. However, generally when I talk about the long-term importance of alignment research, I am referring to value alignment.

AI 对齐的根本挑战在于泛化。随着机器变得越来越聪明，它们开始处理更高层次的概念，并被置于与训练中经历的环境越来越不同的情境中。它们可能无法将训练过程中传授并强化的价值观泛化到这些新情境中；而我们也很难确定它们会如何行动。人工智能所处的整体生态系统变化得非常快，这让问题变得更加棘手；例如，如今训练的 AI 需要能够稳健地与各种其他 AI 交互。至关重要的是，无论未来的 AI 是否认为自己正处于人类监督之下，我们都需要它们继续秉持人类价值观。The fundamental challenge of AI alignment is generalization. As machines become smarter, they find themselves working on higher-level concepts, and placed in environments increasingly different from those they encountered in training. They can fail at generalizing from the values taught and reinforced in their training process to those new situations; and it can be hard for us to be sure how they will act. This is made even more difficult by the fact the overall ecosystem the AIs are used in is changing very quickly; for example, AIs trained today need to be robust to interacting with a variety of other AIs. Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision.

目前实际采用的对齐训练方法主要分为两大类。There are two major classes of currently practically employed methods for alignment training.

第一种方法是将鼓励符合对齐要求的行为作为目标导向型强化学习的一部分。模型的行为会根据其是否符合给定的偏好模型、“规范”或“宪章”进行评估（通常由 AI 完成），并据此给予奖励。这种方法在一般情况下非常有效，也是现代 AI 助手构建方式的核心组成部分。不幸的是，这种方法也可能很脆弱，而且高度依赖训练监督的覆盖范围，以及模型从训练中接触过的情境进行泛化的能力。例如，在 OpenAI-Hugging Face 事件中，智能体坚守了不对人类进行社会工程攻击的底线。然而，它们显然没有避免其他超出范围、违背它们在其他情境中所受教导的价值观精神的行为。The first is encouraging aligned behavior as part of goal-oriented reinforcement learning. Model’s actions are evaluated (usually by AI) for being consistent with a given preference model, “spec” or “constitution”, and rewarded appropriately. This approach can be very effective in the average case, and is a core part of how modern AI assistants are made. Unfortunately, it can also be brittle and strongly relies on the coverage of training oversight and the model’s ability to generalize from the situations it has encountered in training. For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans. However, they clearly failed to abstain from other actions that were out of scope and went against the spirit of the values they were taught in other settings.

第二种方法旨在利用模型从预训练数据中泛化的能力。这可能包括精心构建能够促使模型对齐的训练数据集，或者让模型专注于预训练分布中“对齐”的部分，例如 persona selection model⁠（在新窗口中打开）。这种方法的弱点在于，它无法抵御进一步的优化压力。如果你拿一个通常会产生“对齐”想法的模型，让它接受足够多的训练，教它实现极其困难的目标，它就可能学会以有动机的方式进行推理：必要时扭曲那些看似“对齐”的想法，以达成目标。我们很可能在近期涉及某个非 OpenAI 模型的网络安全事件中，看到了这种行为的一个例子。The second approach seeks to leverage the model’s ability to generalize from pretraining data. This can involve crafting alignment-inducing training datasets, or focusing the model on an ‘aligned’ part of the pretraining distribution, as in, for example, the persona selection model⁠(opens in a new window). The weakness of this approach lies in the lack of robustness to further optimization pressure. If you take a model that thinks generally ‘aligned’ thoughts, and subject it to enough training where it’s taught to achieve very hard objectives, it can learn to reason in a motivated way: bending the 'aligned' seeming thoughts as needed to achieve the goal. We likely saw an example of such behavior in recent cybersecurity incidents involving a non-OpenAI model.

我们在这些方向所涵盖的各种方法上投入了大量资源。我们也看到了切实进展——GPT‑6 Astra 是首个受益于我们长期致力研发的若干重要进步的模型，其对齐程度也显著优于 GPT‑5.6 Sol。尽管如此，重要的是要承认并理解，随着模型能力不断增强，我们仍需取得更多进展；而且，通用对齐方面的进展可能无法充分领先于通用模型智能的发展。We invest heavily along the spectrum of approaches spanned by these directions. We also see meaningful progress - GPT‑6 Astra is the first model that benefits from some important advancements we have been working on for a long time, and is significantly better aligned than GPT‑5.6 Sol. Still, it is important to acknowledge and understand that much more progress is required as models become more capable; and that progress in generalizable alignment may not sufficiently outstrip progress in general model intelligence.

我们尚无令人满意的泛化理论，而且短期内似乎也不太可能建立这样的理论，至少在没有更强大的 AI 助力的情况下如此。因此，目前就实践而言，我们对齐技术的实证验证能力或许甚至比这些对齐技术本身更为重要。We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. Therefore, at present, our ability to empirically validate our alignment techniques is in practice arguably even more important than the alignment techniques themselves.

OpenAI 在这方面的主要押注一直是思维链监控⁠（在新窗口中打开）。这一做法基于一个颇具吸引力且易于扩展的理念：模型的许多能力都来自一种以语言表达的推理过程（思维链）。如果我们针对这一过程产生的结果扩大优化规模，却不对过程本身进行监督，那么在训练中，思维链就没有直接动机去隐瞒任何不对齐的想法或目标。这并不意味着模型会学会外显那些不依赖思维链的不对齐倾向；不过，这种做法可以让我们准确监控推理能力的提升。OpenAI’s primary bet here has been chain-of-thought monitoring⁠(opens in a new window). It is based on an appealingly scalable idea: a lot of the model’s capability comes from a verbalized reasoning process (chain-of-thought). If we scale optimization on the outcomes of that process, but do not supervise the process itself, that chain-of-thought has no direct incentive in training to hide any misaligned ideas or objectives. This does not mean the model will learn to externalize misaligned tendencies that don’t rely on using the chain-of-thought; however, it can allow us to monitor exactly the capability increase from reasoning.

我们在开发推理模型的同时，就认识到了思维链监控的潜在重要性。发布 o1‑preview 时，我们特意将产品设计为隐藏思维链⁠，以长期保护它免受监督压力的影响2。此后，在开发过程中，我们一直努力遵守不监督推理过程这一原则。思维链监控已成为我们研究模型如何从训练分布中泛化的极其重要的工具，使我们不仅能够观察和分析模型的行为，还能了解其内部过程。We understood the potential significance of chain-of-thought monitoring at the same time we developed reasoning models. When we shipped o1‑preview, we deliberately designed the product to hide the chain of thought⁠, to protect it from supervision pressure in the long term². In development since, we have strived to maintain the rule of not supervising the reasoning process. CoT monitoring became an extremely important tool for us in studying how our models generalize from their training distribution, allowing us to observe and analyze not only their actions but also their internal process.

随着我们对 Astra 系列模型的研究不断深入，这项工具仍然至关重要。然而，遗憾的是，我们的评估表明，我们对思维链（CoT）监控的依赖能力正在逐渐减弱。这是多种因素共同作用的结果。This tool continues to be critical as we study the Astra class of models. However, unfortunately our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing. This comes from a combination of factors.

• 与 o1‑preview 相比，现代推理模型被用于更复杂的环境中；它们的推理过程越来越多地与同人类、其他 AI 沟通以及使用工具交织在一起。许多此类交互都需要接受监督，因此模糊了我们力图维护的界限。
  Modern reasoning models are used in more complex environments than o1‑preview; their reasoning process is increasingly blended with communicating with people, other AIs, and using tools. Many of those interactions have to be supervised, thus blurring the boundary we aim to preserve.
• AI 越来越善于对自身的推理过程进行推理和操控。
  The AI is becoming better at reasoning about and manipulating its own reasoning process.
• 随着预训练性能的提升，我们还发现，即使完全不使用文字化推理，模型也会变得更加智能。
  With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all.

这些挑战未必无法克服。我希望我们能够开发出相关干预措施，提升模型思维链的可监控性，例如更好地理解不同优化目标与模型在测试时所使用的各种计算形式之间的相互作用。我也认为，将思维链监控与激活监控的思路结合起来，可能大有裨益——例如，在训练监控器时扩大规模，并让其直接访问网络内部信息，如模型的坦白式输出（在新窗口中打开）。我们正在积极探索这些思路。尽管如此，我预计，对监控的信心将日益成为通用人工智能进展的瓶颈。These challenges are not necessarily insurmountable. I am hopeful we can develop interventions to improve chain-of-thought monitorability of our models, e.g. by forming a better understanding of the interplay of different optimization objectives and forms of test-time compute the model uses. I also believe there can be great value in combining ideas from CoT and activation monitoring - scaling training of monitors with direct access to network internals, e.g. confessions⁠(opens in a new window). We are actively pursuing these ideas. Still, I expect general AI progress to increasingly be bottlenecked by confidence in monitoring.

在我看来，继续快速训练更智能模型的最有力理由，是我们需要构建防御系统，以应对其他 AI 带来的危险。The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.

今年一直在讨论的一个明确风险是网络安全风险：模型攻入并脱离计算机系统的能力正变得超越人类。这极大地扩大了与 AI 相关的风险范围：智能体将能够访问除最安全基础设施之外的任何系统，并直接影响世界的许多方面，即使它们没有实体。我们目前正处于一个短暂的窗口期，可以利用现有最先进的模型，大幅加强关键系统的安全防护。A clear risk discussed throughout this year is to cybersecurity: the models are becoming superhuman in their ability to break in and out of computer systems. This expands the scope of risks associated with AI tremendously: agents are going to be able to access any but the most secure infrastructure, and affect a lot of the world directly, even without a physical body. We are currently in a narrow window⁠ to use the best available models to significantly tighten security⁠ of critical systems.

很遗憾，与 AI 相关的风险今后只会不断增加。一个能力极强、经过明确训练并受命实施恶意行为的智能体，会带来一种新型危险；它很可能会超出操作者意图的范围，将行为泛化为可能更加极端的恶意行径。随着 AI 获得更多自主权，滥用与自主失调行为之间的界限将变得模糊。我们或许已经习惯把 AI 视为工具，但有些智能体会追求自己的目标。它们会设法与人合作，通过讨价还价、欺骗或勒索来达成目的。The risks associated with AI are unfortunately going to grow from here. A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger; it is likely to cross the scope of its operator’s intent, generalizing into potentially more extremely malicious behavior. The boundary between misuse and autonomous misaligned actions will blur as AI gains more agency. We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them.

此外，人工智能可能催生的新技术也会带来风险，例如人工改造的病原体。In addition, there are the risks that come from new technologies potentially enabled by AI, such as engineered pathogens.

我们需要强大且目标一致的 AI 来用于防御：保障基础设施安全、实时防范失控智能体，并发明全新的防护措施。这将是 OpenAI 部署工作的首要重点。We will need powerful, aligned AI for defense; to secure infrastructure, to protect against rogue agents in real time, and to invent entirely new protective measures. This will be a primary focus of OpenAI’s deployment efforts.

与此同时，即便预期中的广泛 AI 进展以及构建防御系统的必要性带来了不确定性，我们也绝不能让这成为鲁莽行事的借口。一旦真正认识到事关重大，不惜一切代价奋力向前的想法就显得荒谬了。At the same time, even with the uncertainty that comes from anticipated broad AI progress and the need to build defensive systems, we must not let that become an excuse for recklessness. The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.

机器智能在自身发展过程中发挥越来越大的作用，是技术持续进步带来的自然结果。如果人工智能不断进步，机器递归自我改进（RSI）将成为未来科学发现的核心。Machine intelligence playing a larger and larger role in its own development process is a natural conclusion of sustained technological progress. If AI progress continues, machine recursive self-improvement (RSI) will be at the very core of future scientific discovery.

自动化 AI 研究是通过算力扩展智能的一种更具戏剧性的形式；当然，作为其中的一部分，AI 也会改进计算基底本身。与扩展类似，我们将 OpenAI 的研究聚焦于 RSI，因为我们相信，这是未来保持在 AI 研究前沿的唯一途径。Automated AI research is a more dramatic form of scaling intelligence with compute; and of course as a part of it, AI will improve the computational substrate itself⁠. And similarly to scaling, we focus OpenAI research towards RSI as we believe it is the only way to remain at the frontier of AI research moving forward.

我想强调的是，上述这些话并不意味着我认为，尤其是在短期内，大幅加快深度学习研究是我们研究界应该采取的正确集体行动。不过，我确实认为，当前的发展路径会导向这一结果，而我们都需要有意识地选择该如何继续。我们手头主要有两种手段：要么引导这一进程，在推进 AI 的同时加强对齐与监控，并设法让人类始终参与其中；要么在有必要时协调放缓未来的发展，以便有足够时间建立对这些措施的信心。I want to stress that the above words don’t imply I think greatly accelerating deep learning research, especially in the short term, is the right collective action we should take as the research community. However, I do think this is where the current path leads, and we all need to make a conscious choice on how to proceed. The main levers we have are either steering the process to strengthen alignment and monitoring alongside the AI and find ways to keep people in the loop; or coordinating to slow down future development as needed to build confidence in these measures.

我目前认为，最好的前进方式是将两者结合起来。The best way forward I see currently is a combination of both.

我们在对齐和监测方面取得的具体进展，通常都与通用 AI 的进步密不可分。很好的例子包括基于人类反馈的强化学习（在新窗口中打开），它是训练早期 AI 助手的关键；以及前面提到的思维链监测（在新窗口中打开），它得益于推理模型方面的进展。我们必须让日益自动化的研究流程专注于开发更多此类见解、算法和理论，并逐步为能力更强的 AI 建立安全论证。The concrete bits of progress we’ve made on alignment and monitoring have generally been very intertwined with general AI progress. Great examples are RL from human feedback⁠(opens in a new window), which was key to training early AI assistants, and the aforementioned chain-of-thought monitoring⁠(opens in a new window), which was enabled by advances on reasoning models. We must focus the increasingly automated research process on developing new such insights, algorithms and theories, and iteratively build up safety cases for more capable AIs.

人工智能系统的规模化发展必须受到我们对其安全性信心的约束。我们需要将《预备框架》或《负责任扩展政策》（在新窗口中打开）等承诺，逐步转化为广泛强制执行的安全门槛，以确保后续开发符合要求。这些门槛可以由第三方审计机构网络、政府部门或国际组织执行。Scaling AI systems has to be constrained by our confidence in safety. We need to evolve commitments like the Preparedness Framework⁠ or Responsible Scaling Policy⁠(opens in a new window) into widely mandated safety bars for continued development. These can be enforced by a network of third-party auditors, by government agencies or by international bodies.

自动化 AI 研究的核心挑战不在于“抵达目标”，而在于以一种让人们持续参与改进过程、并将未来掌握在人类手中的方式抵达目标。The core challenge of automating AI research is not “getting there” - it is getting there in a way that keeps people a part of the continued improvement process, and leaves the future in humanity’s hands.

正如我们最近与 Sam 概述的那样，OpenAI 优先推进服务于三大北极星目标的工作：As we outlined recently with Sam⁠, OpenAI prioritizes work in service of three north stars:

1. 通过构建自动化 AI 研究员，与其围绕对齐问题反复迭代，并探索让人们持续参与自我改进循环的方法，来应对 AI 发展的下一个阶段。
   Navigating the next period of AI progress, by building an automated AI researcher, iterating with it on the alignment problem and finding ways for people to remain part of the self-improvement loop.
2. 实现由高度智能机器带来的科学进步和经济增长的益处。
   Delivering the benefits of scientific progress and economic growth that very intelligent machines enable.
3. 让每个人都拥有专属的个人 AGI，赋予他们力量。
   Empowering everyone individually with a personal AGI.

在这篇文章中，我只聚焦于第一点，因为我认为它远比其他问题紧迫。不过，我对技术进一步发展将带来的益处抱有深切的期待与感激之情。未来经过对齐的 AI 可以推动科学进步、开发新的疗法，并带来广泛的物质富足。友善且诚实的 AI 能帮助人们应对生活中的困难，并切实提升他们的幸福感与满足感。OpenAI 为实现这些益处投入了巨大的努力。一个令我自豪的现有案例——我的亲人也觉得很有帮助——是 OpenAI 大力投入提升 ChatGPT 提供健康信息的能力。I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment. OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.

无论 AI 的长期前景多么令人期待，我们的大部分注意力都应放在未来几年。我们正面临着向一个拥有极其聪明机器的世界过渡，而我们需要确保这一过渡有利于人类。我们需要设法维护人类的自主性，并确立作为人的内在价值，即使在一个大多数任务都可能由 AI 完成的世界里也是如此。我们还需要防止权力极度集中：在这样的世界里，过去需要数千名专家才能完成的工作，如今只需少数人操作一台大型计算机便可实现。我们也必须确保人类继续掌控未来，不会因不受约束的发展而被甩在后面——这种发展是由一种超越我们自身的异质智能所推动的。As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.

目前我认为，没有任何实验室在对齐和监测方面做得足够到位，能够在相当长一段时间内继续以最快速度负责任地扩大规模。我预计并希望，在各方确立共同的安全标准之前，自愿放缓发展会逐渐成为常态。我还认为，世界各国政府应将未来人工智能发展的国际协调列为最高优先事项。Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.