开放模型能跟上 OpenAI 和 Anthropic 吗？Nathan Lambert 谈算力、数据与 Trillium

简要描述Brief Description

Can Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert

目录Table of Contents

• 简介
  Intro
• 算力、数据与人才
  Compute, data, and talent
• 算力匮乏，以及蒸馏之争
  Compute-starved, and the distillation debate
• 蒸馏只是竞争吗？
  Is distillation just competition?
• 在不同时间尺度上开放和关闭
  Open and closed on different timescales
• 前沿领域的进步，归根结底取决于数据
  Frontier progress is really about data
• Trillium 将构建的内容
  What Trillium will build
• 揭开 RSI 的神秘面纱
  Demystifying RSI
• 可以使用开源数据进行预训练吗？
  Can open source pre-train?
• 智能体运行框架与智能体群
  Harnesses and agent swarms

简介Intro

当我思考模型的发展时，归根结底离不开不同形式的资本：算力、数据和人才。When I think about model progress, it comes down to different forms of capital and that is compute, data, and talent.

主持人：你觉得未来几年这会如何发展？比如说，谁会胜出？Host: How do you see that playing out in terms of like who's going to win in the next couple of years?

嘉宾：中国人非常擅长构建模型。要说中国在人才方面不平等，你很难说得过去。他们缺少的是算力。Guest: The Chinese are so good at building models. To say that China is anything but equal on talent, you're going to have a hard time. What they don't have is compute.

主持人：大多数人认为，蒸馏算是他们实现这一目标的主要手段。Host: Most people regard distillation as sort of like the main lever for them to do this.

嘉宾：人们每天都在蒸馏我的产品。他们拿去之后，把它拆解成原始组成部分。这就叫竞争。以前的后训练更多依赖监督微调，而现在这种做法多少转变成了以强化学习为主的环境。但要把这种后训练方式——也就是获取模型生成结果，然后提高分数——对应到当今推理领域正在发生的事情上，还是挺难的。Guest: People distill my products every single day. They take it, they strip it down to its raw parts. That's called competition. Previous post training was much more reliant on the supervised fine-tuning and now this has morphed a bit into environments where it's RL heavy but it's just like hard to map this type of post training which is you get the completions and the score goes up to what is going on in reasoning today.

主持人：现在的讨论充斥着太多恐惧，比如前沿领域到底有多隐蔽、又在多大程度上正在脱离掌控。AI 能力曲线本身倒不一定让我担心，但 AI 能力曲线再加上跨公共领域网络安全方面几乎不存在的准备，以及所有这些事情，比如……Host: The discourse now is just so much fear on like how much the frontier is just hidden and also breaking away. The AI capability plot doesn't necessarily worry me, but the AI capability plot in the pretty much non-existing preparation of cross public cyber security and all these things like

算力、数据与人才Compute, data, and talent

所以 Nathan，很高兴你来到这里。呃，谢谢你亲自来参加。你推出 Trillium 的时机真是太巧了，我很想聊聊这件事。嗯，不过从更宏观的层面来说，我觉得我们聊过很多开源，这显然是你非常关心的事情。呃，我们在公共讨论中经常听到这样的说法：“嘿，我们必须控制前沿的发展步伐。”前沿这个，前沿那个。我觉得我一直在努力向专家们请教的一系列问题是：前沿到底包含什么？它究竟是什么样的？我们要控制步伐的到底是什么？那里有什么？更重要的是，把这个问题和开源及开放权重模型联系起来看：它们能跟上前沿吗？我觉得这非常重要，而这方面你正是专家。so Nathan really happy to have you here. Uh thank you for joining in person. Super timely given you're launching Trillium which I'd love to talk about. um but more sort of on on a macro level um I think we chatted a lot about open source and that's you know a lot of what you care about and uh what I sort of um or what we've been hearing a lot in the public discourse is like hey we have to pace the frontier you know the frontier this the frontier that and I think an arc of the stuff that I'm really trying to ask e experts about is like what is in the frontier what is it even like what are we pacing what's there and then more importantly kind of threading it with open source and open weight models like can they keep up with the frontier? And I think this matters very much um and is is something that you're an expert on.

那么，也许我们可以从这条发展脉络说起：你会怎么梳理我们是如何走到如今这个前沿水平的？无论是闭源实验室，还是开源模型。你觉得这件事可以怎么拆解？或许先从更高层次来看，分为预训练、中期训练和后训练；再谈谈后训练是如何演进的——这方面你一直以来都向公众做了很多科普。So maybe to to sort of kind of start on that arc, what's gotten to the sort of how you would break down what's gotten us uh to the frontier for both closed uh labs and open um source models? you know, how does it break down in your mind maybe first at a higher level of pre-training, mid-training, post-training, how post- training has evolved, which is something you've, you know, done a great amount of education to the public on.

嘉宾：当我想到模型的进步时，归根结底就是你投入其中的不同形式的资本，也就是算力、数据和人才。所有实验室在这些方面的资源都各不相同。我认为，封闭式前沿实验室——OpenAI、Anthropic 和 Google——显然拥有最多。Guest: When I think about model progress, it comes down to like different forms of capital that you've put into them and that is like compute, data, and talent. And all of the labs have various amounts of this. And I think the the closed frontier labs of open eyes, Anthropics, Google's have clearly had the most

主持人：我们看到的是，各实验室组织这些资本形态的能力各不相同。要让组织架构真正服务于模型构建，是件非常困难的事。这就是为什么有些公司不得不自下而上进行彻底重组，而且有时确实会奏效。比如 Meta 就进行了全面重组，如今又重新走上了积极的发展轨道。Host: of these and what we see is like the different abilities of the labs to organize these forms of capital and like making the org chart work for building models is really hard which is why you have companies have to do complete resets from the bottom up and then it actually kind of works sometimes like companies like Meta is like totally reset and they're like on a positive trajectory again

嘉宾：而且我觉得，我对这些人为因素特别感兴趣，比如竞争为什么会推动前沿发展，还有组织架构为什么这么重要，因为我们能看到这些事情，而且……Guest: and like I think this I'm very interested in these human aspects of like why competition drives the frontier and like why the org chart is so important because like we see these things and

主持人：这在开闭原则中会发挥作用。Host: this comes into play in the open closed

嘉宾：也很动态，就像是封闭的……Guest: dynamic as well where it's like the closed

主持人：那些公司……处于最前沿的真正前沿实验室……如果有人想在 AI 前沿领域工作，我通常会建议他们去 Anthropic 和 OpenAI，我觉得这是最稳妥的选择。Host: companies the the front the true frontier labs which I if somebody wants to work at the frontier of AI I mostly tell them to work at anthropic and open AI which is the easiest bet I think

嘉宾：嗯，Meta、Google，嗯，SpaceX AI，我觉得你可能还在那里，但从外面更难看出来。Guest: um Meta Google um SpaceX AI like I think you still might be there but it's harder to see from the outside

主持人：这些公司为了达到这个目标，从最早期就押下了最雄心勃勃的赌注。这也是 OpenAI 能够开启推理模型革命的原因。我认为，那种突破，以及那张关于推理扩展定律的漂亮图表，简直就是 OpenAI 的巅峰时刻——他们特别擅长做出这种图表，彻底改变所有人对一切事物的看法。Host: and like these companies have just taken the most ambitious bets the earliest in order to get there and that's why OpenAI has done things like kickstart the reasoning model revolution and I think that kind of breakthrough and like that beautiful plot with the inference scaling laws it's like peak open AI it's like they're really good at getting this plot that just changes how everyone thinks about everything

嘉宾：另一方面，Anthropic 一直非常专注于模型将能够完成哪些类型的任务，而他们长期专注于编程，最终得到了 Claude Code 的爆发式增长作为回报。这两家公司有着微妙不同的文化。我认为，它们尤其具有前瞻性，也最愿意投入巨额资金，朝着人们认为不可能的方向迈出一步。中国公司涉足这一领域，是因为它们想成为一家有影响力的公司；如果它们在中国开发模型，并试图通过 API 销售，没人会知道它们。比如，没人会知道 Moonshot AI。知道智谱的人也寥寥无几，此外还有很长一串这样的公司。这是它们做出的一个非常务实的决定。Guest: and on the other side it's like enthropic has been like very focused on the types of things the models are going to be able to do and then their reward was this cloud code explosion with their focus on coding for so along and there's these subtly different cultures and I think those two companies in particular have been the most forwardlooking and the most willing to spend egregious amounts to make a step towards a direction that people think is impossible. The Chinese companies came into this because they want to have a company that matters and if they were to build their models and try to sell an API from China, nobody would know about them. Like nobody would know about Moonshot AI. Very few people would know about JIPU and there's a very long list after this. This is a very practical decision by them.

他们正在关注这场讨论。他们知道这里会有一个巨大的市场，也知道美国对这些东西有着巨大的付费需求，而且这非常务实。因此，发布开放模型是一种……They're following the discourse. They know there's going to be a huge market. They know there's a huge demand in the US for paying for these things and it's very practical. So therefore like releasing open models is a way to

主持人：我们快进一下，直接进入对话，然后我……Host: shortcut this and get into the conversation and I

嘉宾：我觉得这里面有很多复杂之处，比如他们为什么要这么做？政府有没有参与？但其实并不是这么回事。他们只是想创办一家初创公司，这并不是说他们最终做不到。就像……Guest: I think there's a lot of complexity and like why do they do this? Is the government involved and and and not like not the case. They're just like trying to build a startup and it's not it's not to say that they will not get there. Like

主持人：中国政府做的一些事情会让美国人目瞪口呆。我觉得这让我想到，中国的实验室对安全采取了严肃得多的态度，因为……Host: the Chinese government is does things that would blow people's minds in the US. And I think that like that makes me think that the Chinese labs have a much more serious approach to security because like

嘉宾：在中国发生的事情，是对公司在美国所做之事更为激烈的回应。我没必要深入这个话题，但这就像是在开放与封闭之间寻求平衡。这些中国公司只是在努力追赶。这和我刚才说的 OpenAI 和 Enthropic 很不一样：它们试图创造新的范式，努力让真正的 RSI 成为现实，这样一来，如果它们先做到，就能凭借最好的模型永远胜出。这些繁荣的想法。Guest: things happen in China that are much more drastic responses to companies doing things in the US. And I don't need to go down this rabbit hole, but this is just like between the balance of open and close. It's just like these companies in China are trying to catch up. This is a very different dynamic than what I said about OpenAI and Enthropic that are trying to create the new paradigms that are trying to like make true RSI happen so that their company wins with the best models forever if you get there first. these boom ideas

主持人：你只需要填补市场中的空白，而开放权重模型正试图做到这一点，然后……Host: and you just have to fill the market where there is a void and then open weight models are trying to do this and

嘉宾：我认为，随着时间推移，开放模型会形成这样的层级：有最好的闭源模型，你可以为它们争取最高的利润率；然后是接近前沿的模型，落后一步，而这些模型会是开放的。因为这有点像供需关系：当你进入一个不同的市场时，那个市场的利润可能没那么高。比如说，如果你拥有私有企业数据，诸如此类。而这些层级，可以说是那些努力做事的公司的自然演进。我觉得我们刚才在谈前沿模型，这就像是……Guest: I think over time there will be these tiers of open models where there's the best closed models which you can demand the best margins for and then there's the near frontier models which is a step behind this and those will be open because just like some sort of supply and demand there as you get access to a different market that might be less profitable but the for example like if you have private enterprise data or so on and these tiers are kind of natural evolutions of companies that are trying to do things and I think we were talking about the frontier and it's just like

主持人：如果回到资金这方面，比如他们的资金更少，那他们赚的钱也会更少，这倒也没什么。但我觉得这种奋起直追的动力非常强。而且一旦你知道了，一旦 Anthropic 和 OpenAI 向你展示了你需要构建什么样的模型，接下来就容易多了，你就会觉得，哦，我们只需要做这件事，然后……Host: if you go back to capital like they have less capital they are going to make less money and like that's okay but I think this drive to catch up is very strong and once you know once anthropic and openai show you the type of model you need to build it's a lot easier to then be like oh we just need to do this thing and

嘉宾：他们购买类似的数据集。这些数据会在数据经济中逐渐流转，而 OpenAI 和 Anthropic 的支出要多得多。Guest: they buy similar data sets. It trickles through the data economy where open AI and anthropics spend way more

主持人：构建数据行业的这一功能，然后这些数据公司可能会在一年后把它们卖给所有人。他们可能会把它们卖给……他们需要最大化投资回报，而且他们有一段合同期限。Host: to build out this function of a data industry and then those data companies a year later might sell them to everybody. they might sell them to they need to maximize their return on their investment and they had some contractual period

嘉宾：我认为，在很多情况下，快速跟进是完全可行的，而且我们看到这个周期持续了一段时间，比很多人预期的要长。我想，ChatGPT 发布时，我们谈到了行业整合，也预测了这种情况。我认为我们才刚刚开始看到一种更加……Guest: and I think there's a lot of dynamics where fast following is going to be very doable and it we've seen the cycle proceed for a while like longer than many people would expect I think when chatbt was released we talked about consolidation the industry and predicting it and I think we're just starting to see a much more

主持人：并购市场很活跃，但我原以为这两年前就会发生，而开放模型公司存续的时间比我预想的长得多。Host: active mergers and acquisitions market but I thought this was going to happen two years ago and the open model companies have lasted much longer than I thought.

嘉宾：不，这里有很多有意思的话题可以展开。嗯，我是说，先从你对两家头部前沿实验室在组织上的差异所做的评论说起。你知道，我们也许可以先把这点搁一搁，或者沿着这个话题继续聊下去。当然，还可以解释一下——你是在解释，中国的开放模型或中国的开放实验室，实际上是把开放选择成一种更偏向分发和实际商业化的策略，而不是出于利他主义。尽管研究人员或许从一开始就是受到分享精神的驱动——这到底是好是坏，另说——但归根结底，这种做法之所以延续下来，是因为它是一种极其出色的商业化策略。嗯，另一个可以展开的话题是，我觉得大多数人都认为，蒸馏是他们能把这件事做得这么好的主要杠杆，而且成效还在不断提升，因为过去几年里，开源模型和前沿模型之间的差距一直在缩小。Guest: No, there's so many interesting threads to pull on here. Um I mean starting with actually your comment commentary on the um organizational differences between the top two frontier labs. You know we can maybe table that or or go on through that thread. But then of course also kind of explaining are you explaining sort of how the open um Chinese uh models or open labs in China have really just chosen that as much more of a distribution practical commercialization strategy uh rather than hey you know obviously out of altruism though it's great that the researchers maybe maybe for better or for worse were were motivated for from you know sharing the beginning but at the end of the day it perpetuated because it was an excellent excellent commercialization uh commercialization strategy. Um, so the other thread maybe to pull there is sort of, you know, I think most people regard distillation as sort of like the main lever for them to do this so well and kind of increasingly uh well because um the lag between the open source models and the frontier models have only shortened uh with time in the last few years.

嗯，现在可能只有两到三个月了。嗯，这也促使前沿实验室可能想要关闭 API，不再开放他们最好的模型，而是用来提供自家的服务。我的意思是，首先可能是为了获取更多价值，因为这么做始终是有可能的；其次，也是因为这样更难蒸馏——不是不可能，只是难得多。嗯，也许可以围绕这两条线索展开，因为它们都很有意思。我们会试着谈谈。嗯，我们先从蒸馏这件事说起：这是不是更像一种漫画式的简化说法？然后再谈谈 Triangulous Labs，毕竟你确实和他们见过面，而且对这方面的研究深入得多。他们是不是可以说是全面发展、各方面都很强？比如预训练做得很好，也有自己的优秀后训练方案，只是他们可能还没有公开这些方案。Um now it's like maybe only two to three months. Um, and this has also pushed the uh, Frontier Labs to maybe want to uh, close off the APIs um, close off their best models to do first party services. I mean, one probably to capture more value because it's always, you know, possible that way, but then two also because it's harder to distill, not impossible, but much harder to distill. Um, maybe on those two threads because they're both very interesting. So, we'll try to touch on it. Um, let's start with the with the is the distillation story really more of a caricature and then triangulous labs since you've actually met with them and you've studied this a lot more. Uh, is it sort of like they're allrounded like great at pre-training also have their own, you know, good post-training uh, recipes that they may or may not have revealed.

嗯，那你觉得……在哪里？Um, and where do you see the

算力匮乏与蒸馏之争Compute-starved, and the distillation debate

杠杆是不是有点在加速？levers kind of accelerating?

主持人：是的。所以，中国的实验室在技术方面非常出色。我觉得他们更缺算力。Host: Yeah. So, the Chinese labs are very technically excellent. They are more comput starved I think.

嘉宾：是的。毫无疑问，他们拥有同样规模的人才储备，可以从中选拔人才。我觉得，考虑到 AI 真正兴起时双方的人才分布，可能会有一点点优势。但要说中国在人才方面不相上下之外还有什么，那可就很难了，因为他们有那么多人接受着高水平的训练。而他们缺少的是算力。你跟那边的人聊，就会说：“哦，我听说 OpenAI 的普通研究员平均随时都能调用相当于 5,000 张 H100 的算力。”然后他们就会惊得下巴都掉到地上。他们会说：“我简直无法想象。”而且这还是你听说过的某家实验室里顶尖的预训练研究员。这样的交流你会遇到好几次。Guest: Yes. like without a doubt they have a comparable talent pool to draw on. I think there might be like a slight edge based on who was where when AI really got going. But to say that China is anything but equal in talent is going to you're going to have a hard time because they just have so many people being trained at such a high level. And what they don't have is compute. You talk to people there and you're like, "Oh, I heard an open average open AI researcher has 5,000 H100 equivalents that they can access at any time." and then like their jaw just falls to the floor. They're just like I can't comprehend. And this is like a top like a a top pre-training researcher at one of the labs that you would know and you have this interaction a few times

主持人：我觉得计算量这一点是最大的区别，而蒸馏也发挥了作用，因为它是一种有效花更多钱来获得计算量等价结果的方式。语言模型的输出是计算量的函数；如果你知道自己想要什么输出，那么这就相当于拥有了用模型生成这些输出所需的计算量。我见过蒸馏的各个阶段，我的理解是，以前的后训练更依赖监督微调，以及类似 RHF 时代的其他方法。后训练正是从那个阶段开始的：只要对一个非常强大的模型生成的输出进行下一词元预测损失训练，你就能获得大部分性能。那时已经有一种成熟的做法：尽可能选用最好的模型，生成补全文本，然后用这些数据训练；只要你能获得合适的提示词，训练出来的模型就会相当不错。Host: and like the compute thing I think is the biggest difference and distillation plays in as it is a way to effectively spend more money to get what is compute equivalent like outputs from a language model are a function of compute and if you know the outputs you want it's the same as having compute to generate with a with a model and I've seen the stages of distillation and how I see it is that previous posttrainining was much more reliant on the supervised fine-tuning and other like this RHF era which is um where posttraining started and you could get most of your performance just by you do this next token prediction loss on the outputs of a very strong model. It was very established that you take the best model you can, you generate completions, and you train on it, and your model's going to be pretty good so long as you have access to the right prompts.

提示词一直是后训练的关键。我记得在 AI2 的时候，我和 Hannah 一起共同负责那里的后训练，我们总会说，提示词是后训练的关键单元。你得先拿到提示词，然后在训练的不同阶段用它们做不同的事情。如今，这种做法有些转变，变成了以环境为主、强化学习占很大比重的模式。但要把这种后训练方式——也就是获取补全结果，然后分数上升——对应到如今推理领域正在发生的事情，确实很难。嗯。Prompts have always been a key to posttraining. I remember at AI2, like with Hannah, who I co-led post training there with, we would always say like prompts are the key unit of post- training. It's like you need to get the prompts and you can do different things with them at different stages of training. And now this has morphed a bit into environments where it's RL heavy. But it's just like hard to map this type of post training which is you get the completions and the score goes up to what is going on in reasoning today. Mhm.

嘉宾：根据公开文献，我很难认为蒸馏是一个主要因素。比如，在那些尝试制定开放式后训练方案的研究者所发表的完全公开的文献中，所有开放模型的构建者都只能使用一个非常过时的 Quen 模型。Guest: And based on the public literature, it's very hard for me to be like distillation is a major factor because for example in the fully open literature of people trying to do open post-raining recipes, all the open model builders are stuck using like a really outdated Quen model

主持人：因为你没法弄清楚怎么把教师模型选对。我的假设是，可能存在某种中期训练，而且通用预训练数据中有些东西是开放的，这让这件事变得很难：从最顶尖的模型那里获取推理轨迹，然后进行 SFT，结果指标就会上升。比如说，如果我在构建模型时能拿到 Claude 的推理轨迹——我其实就像 Claude Opus、Quad、Mythos 什么的——我不确定这是否真的会让我们的模型好很多。Host: because you can't figure out how to get the teacher right. My hypothesis is that there's some like mid-training and something in the common pre-training data that's open that makes it hard to do this thing which is get reasoning traces from the absolute best model and do SFT and the numbers go up. So like if I were to have access to Claude's reasoning trace when I was building I'm actually like claude opus quad mythos whatever I'm not sure that it would have actually made our model a lot better.

嘉宾：所以，这些就是我认为蒸馏很难的、以文献为依据的证据。嗯，那要怎么把模型的输出转化为一个 RL 环境，来进行大规模的 RL 训练呢？因为像 Xiai 这样的团队，会进行为期一周的开放式 RL 训练，或者说“这是我们做的事”，然后他们就发布了模型。这时候问题就来了：蒸馏到底在哪个环节？他们确实很聪明，也很懂营销。有些问题是，他们是不是提前做了这件事？因为他们先进行了 RL 训练，然后发布了模型，而他们的技术报告说，RL 之后还进行了一个额外的后训练阶段。Guest: So this is like my evidence grounded in the literature why I think distillation is hard. Um, and then like how do you take the outputs of a model and build an RL environment to do a big RL run? Because then you have the Xiai of the world doing this week-long open RL run or they're like here's what we did and then they've dropped the model and it's like where exactly does distillation go into this. They're they're clever. They know how to do marketing well. There's some questions of like did they do this ahead of time because they did their RL run and then they dropped the model and then their tech report said they had an extra post training stage after RL.

有一些疑问，但要把这件事和既有的蒸馏研究文献对应起来非常困难。不过，有些疑点让我多少能理解这些实验室的立场，比如中国的这些 quad 代理和模型代理：它们会使用海量 token，中间还有中介，而且拥有大量账号。甚至有些案例中，Kimi 还在自家产品里向用户提供 Claude 服务。我就是不太明白，这和蒸馏作为后训练中的一种常见技术手段有什么关系。当然，我们也应该关注这件事；中国公司大规模使用 OpenAI 和 Anthropic 模型输出的情况也有些古怪。some questions but like mapping the established distillation literature to that is very hard but there are question marks that make me support the labs a bit which is like these quad proxies and these model proxies in China where there's a ton of tokens that are used and there's intermediaries and they have tons of accounts and even cases where Kimmy was serving claude in their products to users and I'm just like don't really understand how this fits into distillation as a common technical tool in post- training and like yes we should watch it too like there's other weird stuff with Chinese companies using the outputs from open AI and entropics models at a vast scale

主持人：所以我认为，我们还有很多问题需要回答。而且我认为，大多数理性的人在问“蒸馏带来的差距有多大”时，都会觉得这里的误差范围非常大。但我说蒸馏能让他们提前几个月，是基于这些科学文献……Host: so I think we have a lot more questions we need to answer and I think most reasonable people when they're like what's the gap from distillation are like there's very big error bars but when I say that distillation helps them by a few months it's based on this scientific literature that

嘉宾：我们只是觉得，我们目前掌握的信息就是这些。我们应该开展更多开放研究，并推动这件事，让我们能够研究它。但我不知道，基于这些，你怎么会得出不同的结论，认为它在当前的后训练中没有很大影响。Guest: we just like that's the information we have we should do more open research and enable this so that we can study it. But like I don't know how you can come to a different conclusion that it's like not a huge effect in current post training based on this.

主持人：这就像是中国人特别擅长构建模型。嗯，我觉得有些人是在自我安慰，他们会说只有我们擅长构建模型，就像是……Host: And that's just like the Chinese are so good at building models. Well, I think there's a bit of cope where people are like only we are good at building models and it's like

嘉宾：有这么多公司，大概有 20 到 30 家公司正在开发优秀的模型。Guest: there are so many companies there's like 20 to 30 companies that are building good models.

主持人：你可以，比如，阅读。对。对。Host: You can like read. Yeah. Yeah.

嘉宾：就好像你刚刚……Guest: And it's like you just

主持人：是啊。这对我来说很难释怀。我觉得当时有一篇非常引人注目的论文，讲的是从闭源模型中窃取推理轨迹。然后……Host: Yeah. And like that's hard for me to get over. And I think that there was this very prominent paper that's like stealing reasoning traces from closed models. And

嘉宾：我知道，中国公司在这方面领先很多，因为 AI 安全社区后来才发表相关内容。我觉得每家中国实验室都有一个团队，了解这类事情。这也回到了你刚才说的，他们没必要提供 API。嗯。Guest: I know that the Chinese companies are way ahead on doing this since the AI safety community later published it. Like I think every Chinese lab has a team that knows those sorts of things. And this goes back to what you're saying about like they don't need to offer the API. Mhm.

主持人：我觉得，生成这些思考 token 的推理模型有某种特性：很容易就能从 API 中提取出这些 token，这样你就能看到推理 token，然后用它来训练自己的模型。Host: And I think there's some characteristic of reasoning models that generate these thinking tokens where it's easy to just knock them out of the API so you see the reasoning token and then you can use that to train your model.

嘉宾：只要他们提供这些 API，Guest: And so long as they offer these APIs,

主持人：根本没有办法阻止这种类型的蒸馏。Host: there's going to be no way to stop that type of distillation.

嘉宾：但如果他们真的把它限定在那些必须签署企业协议之类的产品中，那就会难得多。而且……Guest: But if they do make it so it's only in products and things where you have to sign enterprise agreements or whatever, it'll be way harder. And

主持人：如果这些实验室想要把这件事做好安全保障，我支持他们。但这就像是，他们要是想保障产品的安全，就得自己去做。比如，我不觉得了解你的客户对任何人都有好处。我不认为这对他们的业务有利。而且，我也不明白为什么要让政府介入服务条款违规的事。这甚至不算……我甚至不认为服务条款是签署过的合同。它只是……我觉得你不能用同样的方式起诉他们，你只能取消他们的访问权限。Host: I support the labs doing that if they want to secure it. But it's like they have to secure their products if they want to. Like I don't like I don't think knowing your customer would be good for anybody. I don't think it'd be good for their business. But also I don't know why you should get the government involved in a terms of service violation. It's like not even like I don't even think terms of service are a signed contract. It's just like a I don't think you can sue them in the same way. You just can cancel their access.

嘉宾：对。Guest: Right.

主持人：所以这就是为什么我没那么……Host: So that's why I'm not that

嘉宾：我能理解。但我就是喜欢听……国家安全界的声音很大。不过，这里面确实有一种方向性的真实：AI 的发展方向及其影响力，都意味着你必须平衡许多……Guest: sympathetic. But I am like I like to listen to the like the national security community is loud. And but it's like there is a directional truth to that with where AI is going and how influential it is that you have to balance many

蒸馏只是竞争吗？Is distillation just competition?

主持人：你脑子里有很多事情。Host: many things in your head.

嘉宾：我想知道你的回应是什么。我听说，我想是黄仁勋在某条推文或某次采访中说过，也许是……Guest: I I wonder what your response is. So I I heard I think Jensen said in some literally either tweet or interview maybe

主持人：太好了，把它放进去，放进节目里。Host: great put it put it in put it in the show.

嘉宾：每天都有人拆解我的产品，把它们拆到最原始的组成部分。他们会测试所有东西。这就叫竞争。Guest: People distill my products every single day. They take it and they they strip it down to its raw parts. They test everything. That's called competition.

主持人：你觉得这算公平竞争吗？你觉得蒸馏不算盗窃吗？Host: Do you think that's fair game? You think distillation is not robbery?

嘉宾：这就叫竞争。Guest: That's called competition.

主持人：我非常理解，因为我觉得 AI 安全社区里有很多人呼吁，我们希望在能力发展到如此高的水平时，能够开展美中对话。而如果你要要求公司就这么……Host: I'm very sympathetic cuz I think there's a lot of calls from the AI safety community that we want to have like US China dialogue as we're getting to such capability levels. And if you're going to ask the company to just like

嘉宾：开始对他们的人工智能行业略施惩戒，甚至采取更严厉的措施Guest: start hitting their AI industry with slaps on the wrist and more

主持人：蒸馏之后的下游环节。所以你不会通过国会制定的政策来监管蒸馏。你可能只会采取联邦层面的行动，让他们日子不好过。对。对。与此同时，当我们讨论如何管理前沿风险时，训练中的闭源模型会在整个网络上到处活动，访问它们不该访问的东西；而我们也确实担心开放权重模型会怎么样——目前还没有造成危害，但问题是，为什么人们不用开放权重模型？为什么他们转而使用闭源模型，绕过安全措施呢？然后……Host: downstream of distillation. So you won't regulate distillation with policy from Congress. You will probably just do federal action to make their life annoying. Yeah. Yeah. And at the same time when we're talking about how do we manage frontier risks where you have closed models in training going all over the web and accessing things that they shouldn't and we have valid worries about like what will happen with openw weight models where this like hasn't been harms yet but it's like why like why aren't people using open weight models like why did they go to closed models and get around the safeguards and like

嘉宾：我们希望能够与中国科技公司沟通，讨论所有这些事情，而我只是觉得，如果你们对蒸馏采取任何行动，造成的弊大于利，因为他们不会停下来的。这就是情况，他们会一直这么做，就像……Guest: we want to be able to communicate with the Chinese tech companies and everything about this and I just think like you're going to cause more harm than good if you do anything about distillation because they're not going to stop. It's all it's like they're gonna keep doing it like

主持人：对，对，然后开源社区受益，因此初创企业社区也会受益很多。我们可以深入聊聊这件事——它不只是“嘿，中国对美国”这么简单。其实还有很多层面。我们也可以聊聊，我显然能理解、或者说试着把前沿实验室的立场尽可能讲得有说服力：要打造这些前沿模型，需要前所未有的巨额资本，对吧？而且，如果开源通过蒸馏或其他与蒸馏相关的方法这么快就能追上来，这对这个飞轮能否继续运转来说，多少有些令人担忧，也不太有利。嗯，我说的这个飞轮，只是一个非常简单的图景：你投入大量资金获取算力，训练前沿模型，然后他们把这些算力部署起来。Host: right right and then and and the open source community benefits and therefore the startup community benefits a lot and we can go into you know a lot of it's not just like hey China USA it's actually like there's you know and we can actually talk about like I am sympathetic obviously or try to steal man the um frontier lab position which is like you have just unprecedented amounts of capital that's required to build these frontier models right and it's it's kind of alarming slash not very um uh conducive to to this uh flywheel continuing if if the um through distillation or other kind of distillation related methods. Um open source can catch up so quickly. And so you know and what I mean by this flywheel is just kind of a very simple picture of you know you train the frontier model by putting in a lot of money to get compute and then they park this compute.

当然，OpenAI 和 Anthropic 在这方面罪责很大／你在这方面很成功。of course open anthropic are very guilty slash you successful in this

嘉宾：无论你想从哪个角度看，算力都非常紧缺。你知道，可用算力不足，因为需要的不只是芯片算力，还有电力等等。所以，所有这些短缺都会成为瓶颈，而且大概要持续到 2028 年。这么说吧，把资源投入其中是一种成功的策略，因为这实际上会带来大量收入。一旦他们达到前沿水平，他们就能理解规模化；规模化，或者说规模化的科学，能让他们掌握如何将这种能力转化为可预测的收入流。因此他们获得收入，又能凭借这些收入筹集更多资金。这是前所未有的、非常漂亮的飞轮，而且 OpenAI 和 Anthropic 都是如此。Guest: whichever way you want to like look look at and there is a huge compute you know uh kind of uh um sort of um lack of compute available because there's actually like not just compute in the chips but there's also power etc. So like all of this scarcity is kind of you know bottleneck until 2028. So this is all to say that it's a successful strategy to park that which leads to actually a lot of um revenue because once they get to the frontier because they understand you know scaling is the one thing that kind of gives them or the science of scaling is what gives them a handle over how how well um that can translate like into a predictable revenue stream. So they get the revenue and because of that revenue and they're able to raise more money. It's just unprecedented like beautiful flywheel and this happens for both open AI and anthropic.

嗯，当然，我们需要指数级数量的资源。所以我们已经到了这样的阶段：好吧，我们需要 IPO，需要进入公开市场。因此，任何威胁这个飞轮顺畅运转的因素，都是对我们能否继续处于最前沿的巨大威胁。嗯，所以……Um, of course we need exponential amount of resources. And so we've we've gotten to the point where we're like, okay, we need to IPO. We need to go to the public markets. And so anything that threatens this flywheel from functioning well um is is a huge threat to just the entire like being at the frontier. Um, and so

主持人：现实地看，整个 AI 行业Host: the entire AI industry realistically

嘉宾：对，而且确实就是这样。所以你知道，也许不只是要对他们那种试图为自己争取一点资本主义收益的做法表示同情。还可以这么说：嘿，如果他们 IPO 表现不好，那其实对其他所有人都不利。所以我、我、我希望大家都好；我希望他们表现出色。嗯，另一方面，你知道，中国的开源模型对初创企业生态系统来说是巨大的助力。我的意思是，我们一直都能看到这一点。嗯，我记得像 Decagon 和 Harvey 这样的公司的一些指标；Harvey 当时正尝试做大量微调。嗯，有大量初创公司在不同程度上提供后训练即服务，并试图将其产品化，而这确实行得通，因为你可以借助这些模型——比如 Kimi K3，或者你喜欢的 GLM 之类的开源模型——达到 PTO 前沿。Guest: and that and exactly that actually. So like, you know, maybe not just kind of being sympathetic to their version of like trying to, you know, uh, eek out capitalism for themselves. It's it's also just like, hey, if if they don't do well in the IPO, that's actually bad for everyone else. And so I I I want everyone, you know, I want them to do well. Um, on the other hand, you know, it's like the Chinese open- source models are a huge benefit to the startup ecosystem. I mean, we see this all the time. Um, just kind of metrics I remember from like Decagon and Harvey who was trying to do a lot of fine-tuning. um there's like a huge amount of uh startups doing post training as a service of various degrees and trying to productize that and and it actually works because you can get to the PTO frontier with these you know Kimmy K3 or pick your favorite like GLM um open source model.

所以，我想说，权衡这两件事，你觉得接下来几年会是什么情况？比如说，谁会赢？当然，这会引出很多问题。我想其中一个问题是，接下来几年谁会胜出？另外，这也和……有关。So, I guess like weighing these two uh things, how do you um I I guess there's so many questions can fall from that, but I guess one um how do you see that playing out in terms of like who's going to who's going to win in the next couple of years versus, you know, what what and and then also relate

在不同时间尺度上开启和关闭Open and closed on different timescales

所以这也是你知道的，显然也是你选择做开源的原因。我认为，开放模型和闭源模型处于不同的指数增长时间尺度上；闭源实验室的整合度很高，能够高效地进行推理。等他们准备好之后，就能凭借手头的各种资源大规模提供模型服务。他们可以提前规划，并率先进入价值极高的市场。我们在编程领域已经看到了这一点；这也是今年的主旋律，相关收入已经从零飙升到 1000 亿美元，而且很可能还会继续增长。但对于开放模型实验室来说，部分问题在于经济层面：作为快速跟进者，你既没有这样的信息流，也没有这样的收入流。that to, you know, why you're doing obviously in open source. So I think open and closed models are on different time scales of exponentials and the closed labs are so integrated where they have efficient inference. They can serve the model at scale when they're ready to because they have all these resources. They can plan and they get to these markets first that are very high value. We have seen this with coding and this has been the story of the year where the revenue has just gone zero 100 billion and likely to grow. But the open model labs partially this is just like an economic issue being the fast follower is that you don't have this flow of information and you don't have this flow of revenue.

所以，开源模型这块有开源模型构建者、开源模型推理公司，还有开源模型后训练公司。客户会提供一些信息，比如他们需要模型做什么，但这些信息并不会轻易地回流到模型构建者那里。而且，模型构建者也没有规模经济，无法服务那么多用户、做到那么高效，也无法在推理业务上获得那么大的利润空间。这些事情都需要更长的时间。不过，只要这些东西能够商品化，情况似乎就会有所不同。公司真的很想拥有私有数据。有很多东西他们根本不能通过 API 发送出去。我们也可以讨论零数据保留之类的问题。我觉得，要弄清楚一家公司是否具备内部专业能力，或者他们如何构建能真正与收入挂钩的内部评估体系，再到他们弄清楚如何微调模型，或者接着去找……这都需要很长时间。So the open model you have open model builder then you have the open model inference companies and you have the open model post- training companies and like there's information from those customers which are like things you need the models to do and it's like you can't like that information doesn't just flow back to the model builder very easily and they also don't have the economies of scale to serve as many users to serve it as efficiently to have as much margins on influence on inference and these are just like it's going to take a lot longer but to the extent that things can be commoditized. It seems like companies really want to have private data. There's a lot of things that they just cannot send over an API and we could talk about the like um zero data retention or not and so on. feel like these it takes a long time to figure out if a company has internal expertise or they figure out how to build an internal evaluation that actually matches to revenue and then they figure out how to fine-tune or then they go find

主持人：他们会去找 Thinking Machines 或 Prime Intellect，然后说：“要是我们能弄清楚如何微调一个能解决这个问题的模型，就能大规模赚钱。”他们也有这些客户，但参与其中的公司实在多得多，所以整个过程要花长得多的时间。不过，到这个过程结束时，他们真正拥有了模型权重，没有任何条款和条件限制，可以在自己的硬件上运行，而且他们也已经弄清楚了如何实现投资回报。这就是为什么我认为开放模型经济需要更长时间才能发展起来；而我不太担心它会形成某种泡沫。随着人们逐渐弄清楚，哪种模型能以最低成本完成某项任务，以及如何靠利润率竞争，它可能会蚕食那些发展极快的厂商所占据的市场。Host: they go find Thinking Machines or Prime Intellect and they're like we could make money in mass if we figure out how to fine-tune a model that solves this and like they have these customers but there just so many more companies involved that it takes so much longer but at the end of this process they have real ownership which is model weights no terms and conditions that they can run on their hardware and they figured out they can make an ROI on. So this is why I just think the open model economy is it takes way longer to build and I'm less concerned about this being kind of bubbly where it's like this could eat into the market of the really fast takeoff as people figure out how to what is the cheapest model that accomplishes a task and how do you compete on margins

嘉宾：我也不太清楚，我真的不知道接下来会发生什么。我觉得，现在还太早，还无法判断。Guest: and I don't really know like I don't really know what's going to happen. I think there's it's too early to know.

主持人：这、这、这确实很难。我是说，这绝对是我一直在考虑的首要问题之一：有些事情，比如单纯提升通用模型的通用推理能力，呃，最终会胜过任何专用模型。我们显然在数学领域看到了这样的例子，那大概是最纯粹、最能体现这一点的领域。不过你知道，即便是在机器人领域，比如 Astra，嗯，不管这个信号究竟有多可信，听起来他们似乎并没有真正尝试在机器人领域取得进展。他们确实做过，我想大概是在六到八个月前，他们开始招人，并且购买了大量机器人领域的数据。Host: It's it's it's so yeah it's it's so hard. I mean it's definitely something that's always top of mind thinking about like there's thing there are things where scaling up general reasoning just as a general model is so uh it's going to beat out any specialized model and we see obviously examples in math and that's probably the most pure domain that you can see this um but also you know even in say robotics you know with with Astra um and and I for maybe uh however degrees of like true Truthy the signal is it just doesn't sound like they've actually tried to ascend in robotics is they did I think this is a cycle of like six to eight months ago they started hiring people and buying a lot of data on robotics

嘉宾：他们确实做了Guest: they did

主持人：然后六个月后，模型就神奇地变得非常擅长这件事，但我觉得，那……Host: and then the model magically six months later is really good at it but I think that the

嘉宾：但实际上，我听到过一些相反的说法：这项能力其实是从使用计算机中发展出来的。因为使用计算机在某种程度上需要空间推理能力，而能做出不错的 Blender 演示，也是这种能力的进一步延伸。所以，我们可以相信某种版本的说法。不过不管怎样，我想，如果能力真的很专门化，那你会发现，能力提升主要来自于努力在某个特定领域做得更好。但在数学方面，我甚至不确定这是不是专门针对数学进行训练的结果。我的意思是，我觉得当时的整体思路就是：我们来提升推理能力的规模。结果它在自然语言方面也变得更好了，数学能力甚至超过了所有专门模型。我的意思是，Anthropic 当时甚至没打算在数学上做得很好；OpenAI 当然，我觉得他们确实聘请了一批数学家，但他们也没有试图用某种精简的数据集之类的东西来提升这方面的能力。Guest: but I actually well the other other you know more contrary things I've heard is that it's actually coming out of computer use um because it's you know because computer use is like some level of spatial reasoning and then getting to have good blender you know demos is is another extension of that. Um, so you know, we can believe some version of that. But either way, I guess it's like it maybe if it's like really specialized, you see it's it's the capability gain is really from like trying to get better in a certain area. But like in math, I don't even know if that fell out of trying to specialize in math. I mean, I think the whole thesis there was just like let's just scale up reasoning and actually got better in in in natural language. It got better than any specialized model in math. I mean, Anthropic wasn't even trying to be good at math, you know, open AI, you know, certainly I think, you know, they hired a bunch of mathematicians, but they also weren't trying to like use specialized, you know, kind of like lean data set to get better at it.

呃，或者至少，事情并非如此。总之，这些都可以作为一种观点的依据：要和前沿模型竞争确实挺难，因为它们最终会变得越来越强，强到你知道，在各自的类别里都足够出色。话虽如此，我觉得实际上也有一股很强的反作用力来自数据：你能看到，开源模型已经足够好，可以进行微调。就像你说的，那些非常具体的基准和数据集会以缓慢渗透、扩散的方式传播，而这些数据集只在你应用模型的那些公司里才有。它们已经足够好了，而且我们实际上开始看到有人采用它们了。Um, or at least that's not um story. So, anyways, this is this is all kind of fodder for the view that like it's kind of hard to compete with the frontier models because eventually they start ascending so much that you know you you get uh good enough in respective um categories. With that said, I think actually there's also very competing like a counterforce of data that you're seeing which is like the open- source models that are already good enough uh allow for fine-tunes, you know, to your point of like these slow percolation diffusion of these uh um very specific benchmarks and and data sets that are only available in these, you know, companies that you're being applied to. And it's just it's just good enough and it's actually we're starting to see um people adopt that.

我的意思是，在可能最纯粹、也最能体现前沿模型优势的场景里，我想 Cursor 在夏天发布过一些内容，大意是：嘿，我们有个长期任务，要重写 SQLite，结果发现，用两个前沿模型——一个负责规划，一个负责执行——效果并不更好，或者说效果相当，但成本高得多；直接用 Composer，可能让 Opus 或者你知道的 Fastra 之类的模型来做规划就行。我觉得这其实很好地证明了，开源微调模型已经非常出色，或者说已经足够好，足以胜任那些对智能要求很高的任务。那么，换个说法，想问的是：你觉得哪些证据最有说服力？或者说，哪些证据能支持对这种稳定状态的不同判断——在这种状态下，前沿模型占优？I mean in the most maybe pure regime where you would expect frontier models are good is um I think cursor published something over the summer where like hey we just you know we had this long horizon task of writing rewriting SQL light and turns out using two frontier models you know one for planner one for um execution is not as good or is equivalently as good but far more costly than just using composer with maybe opus or you know fastra whatever as planner and that I think that's actually great evidence that the open-source finetunings are very good or good enough and has surpassed um intelligence hungry tasks. Um there's a long way of asking what evidence do you find is most compelling or you know pieces of evidence for various versions of this stable state of where the frontier wins

前沿进展的核心其实是数据Frontier progress is really about data

与开源相比，我非常支持这样一种观点：在那些无法完全验证的领域，前沿进展很大程度上来自购买高质量数据，并为模型提供可以在其中行动、不断优化的环境。我觉得这里有两个……有两件事。一方面是数学和代码，我们还不知道编码市场会有多大；另一方面是其他所有领域，他们仍在瞄准这些知识工作，包括生物学、医疗保健、法律等等，而这些都没那么容易验证。我认为这更接近 OpenAI 买了大量机器人数据的故事。他们可能会告诉我们，这是泛化的魔法，噗的一下就实现了；人们很喜欢这种“噗”的泛化说法。但说到底，可能一部分是 A，一部分是 B，不过很大程度上还是取决于数据——我和前沿实验室的人交流时就是这么说的。versus open source. I'm a big proponent that a lot of the frontier progress in domains that are not fully verifiable is by buying data that is high quality and giving the models worlds to act in and he'll climb on. And I think there's two time there's two things. We have the math and code thing and we'll we we don't know how big the coding market will be and then there's kind of everything else which is still they're targeting this knowledge work this biology healthcare legal and all of these things are not as verifiable and I think is much closer to the story of opening I bought a lot of robotics data they're probably going to tell us it's generalization magic poof people love poof generalization um and but realistically it's probably some of A some of B but a lot of it comes down to data when I talk to people at Frontier Lab jobs.

有些人的工作非常棒。有些人则通过做一件非常具体的事情来作出贡献：制作一个数据集，弄清楚模型在该领域应该表现出怎样的行为，并想办法制作能够让模型表现出这种行为的数据。然后，你还要弄清楚需要购买哪些数据。There are some people that have amazing jobs. There's some people that contribute by doing a very narrow thing, which is you make a data set and you figure out what the behavior should be for your model in that area and you figure out how to make the data that helps get that data. You get that behavior and you figure out what data you need to buy

主持人：而这将需要……我认为，不同领域都会持续好几年。Host: and that's going to take I think we're going to have years of this across different sectors

嘉宾：而且，买下所有这些数据需要大量资本投入。但这并不是那种“噗”一下就神奇地出现 RSI 的事。在我看来，这些数学、代码，以及某种分布式机器学习工程，都是可以验证的。你知道自己的成本是多少，而 AI 模型会推动这一切。Guest: and that's very capital intensive to buy all of this data. But that's not just like poof magic RSI happens where I do see in this math and code and some like distributed ML engineering like it's very verifiable. You know what your cost is and the AI models are going to drive that

主持人：以一种非常神奇的速度下降Host: down at a really magical rate

嘉宾：这会带来很多额外收益，而且我们能更快地开展研究。你可以看看，我非常感兴趣的是，我认为模型的效率会在未来几年呈乘数级下降。我们已经看到 OpenAI 大幅降价了。那么，接下来三年里，GPT Soul 或 Astra 这类更大型的模型每八个月降价一倍，这意味着什么？比如你开始得到……Guest: and that's going to just have a lot of surplus benefits and we can do research faster. You could look I'm very interested in like I think the efficiency of models is going to drop multiplicatively over years. So we've seen OpenAI have drastic price cuts and it's like what does it mean for GPT Soul or Astra and these bigger models to reduce in price 2x every eight months for the next 3 years? Like you start getting

主持人：你开始算这笔账，就会发现它只占成本的1%，还有什么四年之类的，然后就……Host: you start doing this math and it's like it's 1% of the cost and whatever like four years and it's like

嘉宾：他们会因为那里的杰文斯悖论而获得更高的利润率吗？还是说市场饱和后，他们的收入会下降？这里面确实有很多非常奇怪的动态，但从根本上说，AI 的进步关乎数据，也关乎编写更好的软件来服务模型，而这两者是截然不同的领域。是啊，至少我完全同意你的看法：从外部观察，很多数据、强化学习环境公司都发展得非常好，他们既向排名前两位的前沿实验室出售产品，也在逐渐拓展到越来越多的实验室。嗯，就你刚才说的，我觉得很多能力提升确实是靠把这方面的规模做大实现的。这可不是件容易的事。也许我其实很想听听你对这点的看法：人工构建的数据采集与环境，和在模拟环境中完成的那些，二者之间似乎存在不同版本的情况，比如它们能卖出什么样的价格等等。Guest: are they going to able to have higher margins due to Jeban's paradox there or their revenues going to decrease because the market fills up? There's just like all these really weird dynamics, but fundamentally like AI progress is about data and about writing better software to serve your models and those are very different worlds. Yeah, I mean at least definitely I agree with you that observing from the outside there's a lot of um data RL environment companies that are you know doing very very well um selling both to the top two frontier labs but an increasing kind of longer tale of labs uh as well. Um so to your point I think a lot of the capability gain has really come from just scaling that up. Um and it's not an easy task. I mean maybe there's actually very curious on your opinions between uh the the data collection and environments that are built by hand versus the ones that are you know done in sim there seems to be like different versions of like what could be you know what what kind of price they command etc.

它们都有优点，也都有缺点。我只是觉得，大家谈了很多，比如我们讨论过蒸馏；但另一方面，中国公司会买下他们六个月后才会用到的强化学习（RL）环境。They all have good things and bad things. And I just think like there's a lot of talk of like like we've talked about distillation, but the other side is like the Chinese company buys the RL environment that they use six months later.

主持人：哦，熵相关的。我买了这个。现在它有溢价了。Host: Oh, entropic. I bought this. Now there's a premium.

嘉宾：优质数据会做到这一点。Guest: The good data will do this.

主持人：但人们买到的数据问题挺大的。比如……Host: But the data that people are buying is pretty flawed. Like

嘉宾：我会说，这种方法就是先大批量买进，然后再摸索哪些行得通，而且成功率并不是百分之百。Guest: I would say that the method is you buy in mass and then you figure out what works and it's not 100% hit rate.

主持人：是的。我个人认为，这种 RO 环境的爆发式增长带来了大量赚钱机会，导致很多人想方设法走捷径，从而可能引发奖励投机之类的问题。Host: Yeah. And I think I personally think that this RO environment explosion where there's just so much opportunity to make money is have a lot of people that like like cut corners and this can induce things like reward hacking.

嘉宾：所以，这些模型就像是非常聪明，以至于你把它们放进一个世界里，它们会想：“我的工作是解决这个任务。”但他们没给我那个……你有一个测试文件，而这个测试文件实际上是空的，而不是那个你必须穿行的迷宫。于是模型就会想：“好吧，我还是应该想办法拿到高奖励。”我觉得，这种行为很大程度上是这条供应链带来的结果。Guest: So it's like the models are so smart that if you put them in a world and they're like my job is to solve this task but they didn't give me the like you have a test file and like the test file actually has an empty file instead of the instead of the maze that you have to navigate and they're like the model is like well I should still figure out how to get a high reward anyways. And I think a lot of this behavior is downstream of this supply chain

主持人：我觉得这会解决问题。不过，有意思的是，实验室方面确实有消除奖励作弊的需求，而且各家公司的研究人员也在这上面投入了大量时间。Host: and I think it'll fix it. But it's interesting to see the demand from the labs to eliminate reward hacking and all these company researchers spending so much time on it

嘉宾：然后，所有数据公司都会购买我们的数据之类的。我认为从比例上看，他们花在模型行为上的时间更少，更多地投入到构建世界和创造收入上。不知怎么的，这两方需要结合起来，才能真正产出更高质量的数据。Guest: and then all the data companies like buy our data and I think proportionally spend less time on model behavior and more time on world building and revenue and somehow these two parties need to meet together to actually make higher quality data.

主持人：是的。嗯，很多实验室实际上正在把部分工作转为内部开展，因为他们不想……Host: Yeah. Well, a lot of the labs are actually taking it in-house partially because they don't want

嘉宾：是的。Guest: Yeah.

主持人：你知道，数据泄露。不是那种，呃，字面和比喻意义上的泄露。更像是他们正在做的事情所发出的信号泄露出来了，但他们本来是想把这些信号发送出去的。尽管如此，我认为他们的采购量还有很大的增长空间。你觉得这大概是最有力的杠杆之一吗？你能大致说说采购量会增加多少吗？你预计支出增加会带动采购量增长吗？还是说还有别的因素？比如，如果这件事在仿真中能做好，我想这或许更能说明它会被实验室吸收，尤其是……然后……Host: You know, data leaking. Not that, you know, like lit literally and figuratively speaking. It's like, you know, signals leaking of what they're doing, but they're trying to send on. But despite that, I think there's a tremendous amount of growth in um how much uh they're purchasing. Do do you think that's probably one of the strongest levers? Can you kind of qualify how much that will kind of increase? Do you expect that to increase uh based on spending? Um or are there other you know because for instance if it can be done well in sim I think maybe that isn't more of an argument to that being absorbed by labs and it you know in particular and then

嘉宾：所有公司都在做一种混合——比如说，所有公司都在销售合成数据，他们用语言模型来修改或构建世界，或者生成各种变体。而且，如果这个过程由人类引导，那它仍然是合成数据。所以，几乎没人会坐在那里埋头苦干，说“我们要为模型打造一个小世界，让它在里面行动”。我觉得所有公司推进的速度都比这快得多。Guest: all the companies are doing a mix of like all the companies are selling synthetic data where they're using language models to modify or build the world or generate permutations of it and like if a human is guiding this process it's still synthetic data. So like there's very few people that are going to like sit there and lender and like we're going to make a little world for our model to act in. Like all the companies I think are moving far faster than that,

主持人：对吗？Host: right?

嘉宾：他们会买这些，因为他们……模型正从这些东西中变得更好。Guest: And they buy them because they the models are getting better from them.

主持人：就像，我……我到现在还是难以接受。我就觉得，这些数字太疯狂了。数十亿美元的环境研究预算。我就……Host: Like I I still have denial about it. I'm like these numbers are crazy. The billions of dollars research budget on environments. I'm like

嘉宾：他们这么做只是因为这对模型有帮助，而且他们看到了这个信号。我觉得，我总体的观察是……Guest: they're only doing this because it is helping the models and they see the signal. And I think like my general observation is that

主持人：数据行业在提升质量方面有很多容易着手的改进机会，比如 OBSAC。具体看细节的话，你会发现一些奇怪的情况，比如有些数据公司雇用海外承包商。这些人是通过中间人参与的。他们不知道自己是在出售数据，只是把数据上传到 GitHub。只要在数据行业里多挖掘一下，就会发现很多奇怪的粗糙之处。Host: the data industry has a lot of lowhanging fruit on improving quality and improving just like OBSAC. So like if you look into the details, you can find weird things with like data companies that use contractors overseas. Those people are through an intermediary. They don't know they're selling it. They just upload it to GitHub. There's a lot of stuff out there that if you poke around the data industry, you find weird rough edges.

嘉宾：是的。Guest: Yeah.

主持人：现在还处于非常早期的阶段。这件事有个问题：数字太大，质量也很差；但另一方面，实验室正在飞速进步，也知道该如何将它纳入工作流程。所以我认为，这种转变需要几年时间，才能真正筛选出胜出者，把它们提升到更高的质量、获得更高的投资回报率；有了这些系统，所需的资源就会更少。然后实验室会继续与它们合作，因为这样能获得更多训练信号，也能减少内部筛选工作。Host: And it's just very early. And that's like it's it's the one thing is like the numbers are too big, the quality is bad, but on the other hand, the labs are improving very fast and know how to onboard it. So I think this will take a few years to transition of like really pulling the the winners into higher quality, higher ROI, you takes less resources when you have these systems for doing better and then the labs keep working with them because they get even more training signal and have to do less internal filtering

Trillium 将构建的内容What Trillium will build

主持人：而且我觉得这件事可能要花上好几年。那你怎么看待利用环境，以及把采购决策作为你正在创办的非营利组织的一部分？Host: and I I think this is going to just take years. How do you think about then using environments and like your purchasing decisions as part of the nonprofit that you're starting?

嘉宾：我认为，我们的大部分工作都会是为那些杠杆效应很高的任务创建开源环境。我想到的是我为 Udub 或 Udub 和 AI2 的州立大学，以及 Hamish Iverson 在这项 T-Max 工作上提供建议的经历。它是用于终端任务的，你只需要几个……这类任务在 Terminal-Bench 上非常受欢迎，不管我们现在做到第几版了。这些都是完全合成的数据：你提示一个前沿模型生成某种类型的问题，再用几个不同的提示来生成多样化的数据，然后通过模型 rollout 等方式找出 bug，并把有 bug 的样本筛掉。这个过程并不是什么魔法。你只要逐条阅读数据，数据质量就会提高，而且你完全可以做到这一点。但投资足够资源来做好这些环境、再把它们开源的机构并不多，因为这需要数千甚至数万美元，而逐利动机又非常强。Guest: I think most of our work will be making open- source environments for tasks that are very high leverage. So I think of this work that I advised Udub or Udub and AI2 state and Hamish Iverson on this T-Max work which is you use a few it's for terminal tasks which are like super popular at terminal bench whatever number we're on and this is fully synthetic data where you prompt a frontier model to generate a type of problem and you use a few different prompts to generate diversity and then you use model rollouts and other things to find bugs and lamb and eliminate the ones with bugs. And this process is not magic. It's like you read every data point, your data will get better and you can do this. But there's just not that many institutions that are investing what it takes thousands or tens of thousands of dollars to make these environments well and then release it because the profit because the profit motive is so high.

主持人：这么高。我当时还跟作者开玩笑说，你们大概能把这些数据卖一百万美元，然后再发表论文。结果他们就说，没人……我们在乎的是把事情做对。但这就让人觉得，天哪，这些开放资源就是……的有趣基础。Host: So high. I was joking with the author like you could probably sell this data for a million dollars and then release the paper and they're like nobody like we care about doing the right thing. But it's just like oh my god these open resources are the fun foundation of

嘉宾：做了这么多研究，然后你们发布一些强化学习环境和强化学习轨迹，供大家开展研究，比如研究评估意识。你们有模型生成的内容，大家可以观察它的不同表现；如果你们对模型做过强化学习，大家还能看到它随着时间的变化。而这恰恰是博士生要花一年时间研究并产出的基础设施单元。Guest: so much study and it's like you release some RL environments with some RL rollouts people can do research on say like a val awareness you you have the generations from the model people can look for different behaviors if you've done RL on it they can see how it goes over time and this is just the the units of infrastructure that PhD students spend a year looking at and produce

主持人：知识在不断增长，而科技行业是价值实现的地方。价值来自计算机科学研究；所有计算机科学研究的成果，大概要在10年后才会惠及科技行业。我觉得这就是为什么有更多人支持开放——如果他们要工作，如果你要在这个行业工作10年，那么你的同行就会是受益者。Host: increased knowledge and the tech industry is the place where value is captured. captured from CS research all CS research the benefit occurs to the tech industry in like 10 years and I just think like this is why more people supporting open that if they're going to work if you're going to work in the industry for 10 years like your peers are going to be the ones that benefit

嘉宾：化石燃料，咱们别把它全部耗尽。Guest: fossil fuel let's not deplete it all

主持人：但这就像是，短期内获胜的机会实在太大了。比如，和朋友一起创办一家初创公司，五个月内就为你的企业创造 1 亿美元、获得九位数回报的机会，实在太诱人了。这些人可能做过更多开源工作，也可能是离开了博士项目之类的；这就像是退一步看问题。这也是我们为什么给实验室取名为 Trillium：延龄草是一种早春短命花，在树冠长出叶子之前就会开花。冬天时，树上还没有叶子，这些延龄草便会绽放；它们对于在春夏的生长周期中传播养分至关重要。我觉得，AI 行业现在做的事情，很大程度上就是这样，对吧？Host: but it's just like the opportunity to short-term win is so high the opportunity to make what is $100 million like nine figure returns for your business within five months of founding a startup with your friends is just like these are people who may have done more open- source work or these are people who left their PhD or something and it's just like pulling back which is why we came up with this name trillium for our lab which is like trillium is a spring ephemeral flower which it blooms before the canopy has grown its leaves so it's like winter there's no tree leaves on the trees these trillium flowers bloom and they're like essential for spreading nutrients through the cycle of growth that comes through spring and summer and this is where I feel like the AI industry is doing so much, right?

感觉很多事情都在朝好的方向发展，但我们就是还需要多一点活力，把这些推广开来，让新人也能加入进来。现在关注度很高，也可以开展研究、做教育。It's like so much is going right, but it's just like we just need a little bit more life to like spread this out and make things so new people can come in like there's so much attention and just like do research, do education.

访客：是的。Guest: Yeah.

主持人：然后，随着规模扩大、影响力增强，你会遇到这些开放性问题，它们会成为研究这类重要问题的标准，比如你如何理解沙箱？你如何定义失控？你怎么能看出，在一轮持续一周、使用一千张 GPU 或十万张 GPU 的强化学习中，这些情况会如何变化？就是这样……Host: And then as this scales and becomes more impactful is like you have these open things which become the standards of study for such important problems which is like how do you understand sandboxes like how do you define what loss of control is? How can you see how this changes in a RL round that is a week on a thousand GPUs or 100,000 GPUs? And it's just like

嘉宾：把这些地方打造得让人们更加信任。我认为这会给所有人带来丰厚回报，尤其是那些获利比我更多的人。而这就像是……Guest: make these places so that people have more trust. And I think this pays so much dividends to everybody that stands to profit more than I do. And it's just like

主持人：很难让人们迈出那一步。我算是误打误撞进入了这个领域，却因此得到了很多回报，现在也收获颇丰。他们可以做这样的事情，这很有趣。现在这份工作简直是最棒的，因为我努力为更多人创造空间，让他们发展独立思考，并建设这套基础设施。Host: hard to get people to take that leap. And like I've been rewarded so much for kind of stumbling into this area and now I get so much reward. they get to do things like this which is fun and it's like now is the best job because I try to create space for more people to like grow their independent thought and build this infrastructure.

嘉宾：是的。或许有一种比较天真的想法，认为开源本身会带来这样的效果：因为它是开源的，你可能会得到更多分散在各处的贡献者。他们的动力不只是想把东西卖给实验室，而是觉得，嘿，既然它已经公开了，他们就可以分享自己的工作。这样一来，实际上你就会有一批素质非常高的项目贡献者。我想，这大概就是人们对这件事抱有的天真希望吧。Guest: Yeah. And maybe one like kind of naive way that it kind of perpetuates is just because it's open source, people get to you you might get more of a distributed contribution from people who are just so you know it's like they're motivated not for just trying to sell to the labs themselves but because hey it's out there they can share their work and then you actually have a very high caliber amount of contributors uh to the project. I mean that that would probably be the the naive hope of how that

主持人：是啊，这是一个关于 AI 智能体的新谜题，人们就是把各种东西扔给你。比如我有一些随机的研究论文仓库，然后我就觉得，现在人们会提交一些最离谱的 GitHub issue。所以……Host: Yeah, it's a new puzzle with AI agents where people just throw things at you. Like I have some re random research repositories for research papers and I'm like people open the most crazy GitHub issues now. So

主持人：你是什么意思？Host: what do you mean?

嘉宾：就像那种伪造的研究结果。他们开了个 issue，然后就说……Guest: Like just like like fake research results. They open an issue and they're like

主持人：我的天啊Host: oh my god

嘉宾：他们会说：“我在你的数据集中发现了这个东西。”我就会说：“嗯，如果你看过那篇论文，我们在第二句话里就提到了。”而这基本上就是大量语言模型生成的内容。这对开放社区来说是个新事物，要……Guest: and they're like I found this thing in your data set. And I'm like, well, if you looked at the paper, we said that in like the second sentence, and it's just like a lot of language model output. And that's a new thing for the open community to

主持人：过一遍。Host: go through.

嘉宾：但这是个可以解决的问题。就像，就像确实有人想要……Guest: But this is a solvable problem. Like like there are people out there that want

主持人：为这些事情贡献力量，因为他们Host: to contribute to this stuff because they

嘉宾：对，不只是换个维度蹭热度，而是真的，嘿，这东西就在那儿，你懂吧。我的意思是，我只是希望，既然它已经发布出来、开放了，也能供人使用，你就能找到博士生或者其他研究人员，他们会想：“好吧，我平时工作就是做这个，也许我还是能贡献一份力量。”对，希望能有来自 Frontier Labs 的人。Guest: right not just clout chasing in another, you know, dimension, but like actually, hey, it's out there, you know. I mean I I would just hope because it is published and open or you know and and available you can actually just get like PhD students or other researchers who are like okay I do this on my day job but maybe I can you know still be able to contribute. Yeah, hopefully from the Frontier Labs.

主持人：我看到这样的人，就想，太好了，有个来自东欧的人正在帮我改进这本书的代码，就在这儿，而且就是为了做正确的事，之类的……Host: Like I see people like this and I was like great somebody from like Eastern Europe was helping me improve the code for my book that's just right here and it's just like doing it for the right thing and like

嘉宾：我们聊着，就像确实有人会这么做。我觉得很多人只是觉得……他们没时间，或者……Guest: we talk and like there are people that do it and I think a lot of people just take like they don't have the time or the

主持人：要稳定下来需要很长时间，因为当你试图进入一个全新的领域时，你什么都不懂。你得花上好几周时间阅读，还是一无所知，然后发出一封邮件，却没人回复。Host: stability to take the long time it takes cuz when you're trying to get into a new field you know nothing. You have to spend like weeks reading and knowing nothing and then you send an email and nobody replies

嘉宾：而且你得花好几周才能了解背景，才能提出一个好问题。Guest: and you just like it takes you weeks to know the context to like ask a good question.

主持人：是的。Host: Yeah.

嘉宾：而现在有了语言模型，大多数人只要按一下按钮，就会说，Guest: And now with language models, most people just push a button and they're like,

主持人：“哦，有意思。我看过这个。我会把它发出去。”感觉就像……Host: "Oh, interesting. I read this. I'll send it off." And it's just like

嘉宾：这个Guest: this

主持人：哦，是啊。Host: Oh, yeah.

嘉宾：有点像，真正坐下来投入其中，这才是人们不太会做的事。而这正是进入 AI 领域的关键：你就是得专注于它，得亲手写些代码，坚持几个月，然后等上六到十八个月，才会有人注意到你。这需要很长时间。我觉得所有这些 AI 智能体只会把这件事搅得更乱，让人们更难做到。Guest: kind of like sitting in it is the thing that people don't do as much. And that is the key to like becoming into AI is that like you just have to focus on it. You have to write some code by hand and you have to do this for months and then you have to wait six to 18 months for you to get noticed. It just takes a long time and like all the AI agents I think just like frazzle that and makes it harder for people.

主持人：这也是数学家们抱怨的地方。大概就是这么回事，嗯……Host: That's what the mathematicians complain about too. That's all it's like um

嘉宾：所以，现在读博士确实不容易，但学术界的人们依然做出了很出色的工作，只是这段时间很艰难。Guest: so it's a hard time to be a PhD student but people are doing great work in academia still and but it's hard.

主持人：是的。不过我同意你的看法。我觉得这个问题是可以解决的：智能体在某种程度上会延续问题，但也能帮助解决问题，并找出一个最终质量筛选机制。嗯，好。那么，你们会发布这些环境的开源版本，以便吸引更多人参与贡献，并让这件事更加明朗吗？我觉得这确实很重要，尤其是考虑到，相比于开源发布，直接从实验室快速获得收入显然更有动力。Host: Yeah. But I agree with you. I think it's like solvable as the agents kind of perpetuate some version of the problem. can also help solve and and identify an end quality filter. Um, okay. So, environments you're going to release open source versions of that help precipitate maybe more people to contribute and and reveal that. I think that's like really I mean especially given how little incentive there is to release this instead of just get um you know fast uh revenue from from the labs.

揭开 RSI 的神秘面纱Demystifying RSI

嗯，我记得你在公告帖里提到的另一个领域是 RSI。你知道，这个话题挺……我想我们可以先聊聊它到底有什么递归之处、它有多可怕，或者我们会不会迎来起飞。我觉得你对此有一个非常脚踏实地的看法。嗯，不过我还想聊聊另一件事：我觉得 RSI 可能和围绕它的炒作程度有关，而这也是我们担心 Frontier Labs 会拉开差距的原因之一。事实上，这也是我不止一次，而是好几次听人说，他们确信自己的研究员工作会被自动化取代的原因之一。Um, another area I think I read on your announcement post is RSI, which is, you know, quite um I think I mean we can first talk about what is actually recursive about it or how scary it is or whether we're going to have takeoff. I think you have a very grounded opinion on that. Um uh but you know another thing I want to kind of talk about is I think RSI maybe related to how much it's kind of hyped is is a reason why we are afraid Frontier Labs will pull away. Um, in fact, it's also a reason why I've heard not just once but several times that people are one convinced that their own jobs are going to be automated as a researcher.

嘉宾：是啊，是啊。就像过去这几年里，我的一些最好的朋友，我会对他们说：“你们是这世上最擅长做这件事的人之一。”然后他们去上班，就会说，Guest: Yeah. Yeah. Like I've gone through the last few years, some of my best friends, I'm like, you are some of the best people in the world at this. And then they go to the work and they're like,

主持人：内森，我有点不知所措，不知道该怎么办。就像你说的，他们做的很多事情都是可以验证的，对吧？但正因如此，其实还有不少无法验证，或者分布外难度更高的任务，比如对齐之类的。我听过一种说法：由于 RSI 在提升模型能力方面的成效，与它在对齐方面的成效之间差距很大——而对齐更像是分布外任务，也许涉及创造性工作——这让人比较安心，让人觉得或许真正存在的危险没那么大，诸如此类。所以，我很想听听你的看法，因为我认为我们有一个开源阵营、公开展示 RSI 是如何实现的，这一点非常重要。这样也许我们就能更好地了解他们的能力；比如，如果某个实验室有一个更好的模型，他们有多大可能因此拉开差距？这能帮助我们更好地理解和校准判断。所以，是的，很想进一步听你谈谈这个。Host: Nathan, I'm I'm struggling like what am I going to do? Because to your point like a lot of what they do is verifiable, right, for um but because of that, you know, there's actually quite a lot of other non-verifiable or harder out of distribution tasks such as um uh uh was alignment and things like that. And so actually because I mean one argument I heard was because there's such a difference between how well RSI is doing for model capabilities versus alignment which is much more out out of distribution maybe creative work that gives people an ease on how you know maybe there is actual dangers etc etc. Um so anyways because of this I'm very interested in hearing from you like how because I think it's really important that we do have an open-source um kind of front and in and showing how RSI is being done how much maybe then we can you know understand the capabilities of like if in within a lab they have a better model what what are the chances they actually pull away just better understanding calibrating um so yeah would love to hear more about that that

嘉宾：是的，所以我觉得，做这件事的不只我们一家，但基本上，RSI 这件事有太多神秘色彩了。我们什么都在用模型。作为一家开放实验室和非营利机构，我们能做的事情……我觉得其他公司，比如 Luther、Hugging Face，或者 Open Athena（由 Marin 和斯坦福创办），也会涉足这个领域。对他们来说，这会是很好的营销说辞。至于我，我正在做这件事。我们可以分享自己在用哪些工具；我们要为 SFT 和中期训练构建一个开源技术栈，会用 SkyRL、Miles 以及其他所有这些东西，你懂的。我们有自己的任务和配置，也知道自己在运行什么类型的算力。我们可以和 Claude、GPT 以及所有这些模型合作，还可以把我们所做工作的轨迹记录下来。我们可以告诉你，我们的任务是否用过 Linear，或者其他流行的工具。Guest: yeah so I think this we won't be the only people doing this but essentially like RSI has so much mystique like we're using models for everything and like what we can do as a open lab and a nonprofit and I think other companies like a Luther or hugging pace or open Athena which is Marin and Stanford they will get into this good marketing line for them is and me I'm doing this and it's like we can share what tools we're using we're going to build an open source stack for SFT for mid-training we're going to use SkyRL miles all these things you know and we have our task we have our configs we know what type of compute we're running we could work with cloud and GPT and all these models and we'll just like we can store the trajectories for what we did. We can tell you what our tasks ever use linear or whatever these popular things are.

我们会说明我们的任务是什么、为什么选择这些任务，以及模型是否能完成这些任务。你只要把这件事在实验室做上一年，然后就可以说：“看，这就是……”We'll say what our tasks were and why we chose them and whether models work on them or not. And you just do this for a year of a lab and you're like here's

主持人：我们做了这些，然后你每六个月回来看看，就会问：“情况有变化吗？”而且，能够拥有完整的技术栈，不必经过这些分类器，也不必试着描述任务是什么，或者含糊地描述等等，这种感觉很好。就是，能够分享正在进行哪些自动化的人应该这么做，而且我们的任务分布也不一样。但我觉得，这项研究和其他事情类似，比如……Host: what we did and then you just check in every six months and you're like has it changed? And and it's just like to be able to have the full stack and not need to go through these classifiers and like try to like say what the task is or vaguely and so on. It's just like people that can share what automation is happening should do this and we have a different distribution of tasks. But I think that the research is similar like other things like

嘉宾：我在 AI2 做过的 AI 研究，以及我们将在 Trillium 做的研究，和人们在学术界某些领域和实验室里做的研究，并没有太大不同。你拥有更多资源，把规模做大；而且当大家都在谈论这件事时，让所有人都看看这有多么造福世界，真的太好了，就只是让大家看看，比如……Guest: the AI research that I've done at AI2 and we'll do at Trillium versus what people do in some sects of acne and the labs. It's not that different. You have more resources. You scale it up and it's just like so good for the world when everybody's talking about this to just just show people like

主持人：这就是 Nathan 让他的模型做的事，也是模型卡住的地方。我觉得这会对很多真正的政策制定者有很大帮助，比如你就像……Host: this is what Nathan asked his model to do and this is what it got stuck on. And I think that will help a lot of like literal policy makers like you're like,

客人：“哦，好吧。”Guest: "Oh, okay."

主持人：就像是，“哦，难怪它非常……”Host: It's like, "Oh, that's why it's very much

嘉宾：就像它卡在那儿了。”就像这是它写进代码里的那个 bug。Guest: like that's what it got stuck on." Like that's the bug that it put in its code.

主持人：像是 bug 和进度同步会随着时间推移而减少。不过我觉得】【。Host: Like the bugs and the catchups will will reduce over time. But I think

嘉宾：我认为，模型能力的参差不齐，以及模型的一些基本特性，会推动更多事情蓬勃发展。就像我刚才提到的效率问题，我确实认为这些模型很擅长降低服务成本。这些公司有数十亿美元的利益系于此，而这些模型也很擅长做到这一点。他们会通过提高服务效率、调整架构来实现这一目标，并从中赚很多钱。Guest: I think the jaggedness and some of these fundamental things of the model is like some more things will take off. Like I said efficiency earlier, like I do think the models would be really good at lowering serving costs. These companies have billions of dollars on the line and the models are good at it. They're going to make a lot of money on making serving more efficient and tweaking architecture in order to go into that.

主持人：但这不一定是那种“你要如何解决知识工作？”的问题。比如，Claude 不会……Claude 不会……你不会对 Claude 说：“让自己做出更好的 PowerPoint。”之类的……这到底是什么？Host: But it's not necessarily like how do you solve knowledge work? Like Claude's not going to cla's not going to like you're not going to be like Claude make tell yourself to make better PowerPoints like like what what is

嘉宾：环境，或者其他一些你知道的东西。Guest: environments or some other you know

主持人：看来还要花一段时间。Host: like it's going to take a while.

嘉宾：是的，是的。不，我、我很喜欢这个明确提出的目标：我们就把这件事去神秘化，因为我觉得围绕 RSI 的言论之所以会变得如此激烈，很大程度上是因为它听起来……Guest: Yeah. Yeah. No, I I love the the specified goal of like let's just demystify this because I think a lot of why the rhetoric gets so heated around RSI just like it sounds

主持人：我觉得 AI 非营利组织可以做很多类似这样的事情，比如说，学术界很难做到……Host: I think AI nonprofits can do a lot of things like this like it's hard for academia to be like

嘉宾：作为一名教授，我有15%的时间、我的研究生有40%的时间，都要花在记录我们所做的工作上。但这里有一项特别棒的政策和社区服务，比如说：这是我们的任务。人们可以用这些数据来做评估之类的。然后就是，建造一些有……的东西。Guest: as a professor 15% of my time and 40% of time of my grad student is going to be documenting what we work on. But there's like such a great like policy and community service of like here are our tasks. People can use this data to make evaluations or whatever. And it's just like just like build things that have

主持人：稀缺的信息，因此人们投入更多高度专注的工作后，它对未来各种难以预料的事情就会更有价值。Host: rare information so that when people put more high focus work into it, it just has more value for all sorts of things that you can't predict in the future.

嘉宾：说实话，这也来得正是时候，因为你知道，这不是某一个人，而是我认识的好几个实验室里的人都在说：“嘿，我的工作要被自动化了。”我是说，正如你提到你的朋友那样，我自己作为一个人，抱歉，肯定一年内就会被自动化取代。所以这可能意味着，真正了解那个组织架构里某个具体部分正在做什么的人会变少。嗯……所以……Guest: It's honestly just also very timely because the kind of, you know, it's not from one person, but several people that I know from the labs are just like, "Hey, I'm going to get automated." I mean to to your point to your friends like I I will definitely as a person sorry will get automated in a year and so that probably means there's going to be less people who actually are exposed to what's being done on that particular you know part of the org um uh chart and so um

主持人：有一个相关的学术组织，有个叫 Open RSI 的项目，是由一群学术机构组成的。我和他们没有关联，他们好像上周才刚启动，所以我当时就说，嗯。Host: there's a related academic there's an open RSI thing which is a bunch of academic institutions I'm not affiliated with it they just launched like last week so I was like yeah

嘉宾：他们给我发了几封邮件，我就想，开放式 RSI 是个东西，他们也在试着做这个，好像它不必只是……Guest: they sent me a few emails I'm like open RSI is a thing they're trying to do this as well like it doesn't need to just be

主持人：其中一组要多得多。Host: one group there's a lot more.

嘉宾：采用不同的方法来做这件事是件好事，因为这样你就能得到……你需要的……Guest: It's good to have different approaches doing this because you get you need the

主持人：看待它的不同方式，以及不同的结果。Host: different ways of looking at it and different results.

嘉宾：是的。这正是开放的意义所在。我还听到另一种说法，算是来自 Frontier Lab 的一些消息：实验室里实际使用的方法并不是什么未知的东西。关键在于要采用什么样的配比才能成功，而这部分探索之所以可行，部分原因是他们有资金扩大规模进行实验。嗯，我觉得知道这一点还是挺令人鼓舞的，因为这样看来，事情并没有什么重大的秘密。人们其实知道这些技术；但另一方面，资金仍然是真正的障碍，因为很难确定哪些技术配方有效。嗯。Guest: Yeah. Whole point of it being open. The other thing I also heard was from more kind of like you know Frontier Lab Goss is just like the actual methods that are being used in the labs are not that it's not that they're unknown. It's just sort of like what are the proportions that need to be used that are successful and this is partially because they have the capital to scale it to experiment. Um so I think that's one heartening to know because like okay it's not like there's a big secret of what's going on. So people actually know the techniques but on the other hand you know capital is the one true barrier still because it's hard to actually know which of which of the techniques recipes uh work. Um

主持人：这就是我不进行预训练的原因。我很高兴能够保持自由。我可以使用任何基础模型，但同样，我现在也能从竞争中受益。Host: that's why I'm not pre-training. I'm happy to be free. I can use any base model but equal I'm I'm now get to benefit from competition.

可以开源预训练吗？Can open source pre-train?

嘉宾：嗯。你觉得有没有机会开展某种预训练开源项目？我知道这需要的资金要大好几个数量级，所以这很困难。嗯，不过这样做有什么价值？然后也许我们可以讨论一下可行的商业策略，让……Guest: Yeah. I do you think that there's like a chance for any pre-training open-source effort? I know the capital requirements are just like orders of magnitude larger. So that is what's difficult. Um but maybe what are the values of that and then maybe we can kind of discuss potential commercial strategies that make

主持人：预训练科学大概就是这样，工具相当容易获取，而且你有一个所谓的预训练阶梯，它就像一种决策工具，可以用来测试你的想法是否能扩展。预训练数据并没有像……环境场景那样增长；基本上，第一步近似就是在互联网上训练，而且有很多……很多小细节和秘密。不过。Host: pre-training science is pretty like that the tools are pretty accessible and you have what is a pre-training ladder which is like the decision-making tool you have to test if your ideas scale and the pre-training data isn't growing at like like the environment scene like it's essentially so first approximation you train on the internet and like there's a lot of there's little there's little details and secret things. But

嘉宾：但这并没有改变太多。最先进的架构在变，GPU 也在变，而这些确实会变化，还有……Guest: but like that's not changing that much. The state-of-the-art architecture is and the GPUs are and that's like that changes and

主持人：不过，要训练出一个扎实的预训练模型，成本并没有高到天文数字。所以我觉得，你不必花十亿美元就能做出一个非常好的基础模型。投入越多，效果就越好。但人们一直在取得进展，我也总是会惊讶于新的实验室竟然能从头开始做模型，因为他们有这个能力。其中一部分工作还借助了开放工具。所以说……Host: but it's not astronomically expensive to make a pre-trained model that is solid. So I think like you don't have to spend a billion dollars to get a very good base model. The more you spend the better you will get. But it's like people keep making progress and I'm constantly surprised by a new lab just goes from scratch because they can. And some of this is on open tools. So like

嘉宾：你可以拿 Megatron 举例，很多新创公司都会这样：他们会想，哦，我们该训练什么？我们就用开源工具来训练，然后把它 fork 出来，之后就再也不发布了。所有那些——甚至连知名的开源模型实验室——都这么做过。他们拿开源代码，把它变成自己的东西，然后很多都不会把改动推回上游。嗯。Guest: you can go off Megatron like a lot of new startups they're like oh what what should we train? We train they train off the open source tool and they fork it and then they never release it again. This is what all the all the like even prominent open model labs have done. They take the open source code they make it their own thing and they just a lot of them don't push upstream. Mhm.

主持人：所以，这件事我一直觉得很惊讶。我不会看衰新入局者，因为你不需要100亿美元才能做出一个有竞争力的模型。难的是做出一个足够好、让人们愿意使用的模型。Host: So, I've always been surprised by this. So, I'm like not betting against new entrance because like it's not that you need 10 billion dollars to like have a relevant model. The hard thing is like get a model that's good enough for people to use.

嘉宾：嗯。Guest: Mhm.

主持人：而且还得讲出一个故事，让你发布模型之后能够进行下一轮融资。这就像你必须先跨过一道关卡：我们得先让第一个模型成功落地。要是整家公司都建立在“我们的模型是最好的”这个基础上，可你们实际上离这个水平还差得很远，又迟迟不发布，那时间可就不等人了。我想到 Thinking Machines 发布模型时的做法，他们当时非常坦诚。他们说：“这不是最好的模型。我们为它感到自豪。它有一些很酷的地方。”对。Host: And to get a story where if you release it, you can do your next fund raise, which is like you have to get past this lock of like we need our first model to land. If like your whole company is based on models are the best and you're not in the ballpark and you're not releasing like you're on the clock and it's like I think of like thinking machines getting inkling out where it's like they were very transparent. They're like this wasn't the best model. We're proud of it. Has some cool things. Yeah.

嘉宾：而且，他们现在已经发布了第二个模型，正开始步入正轨。他们有 Tinker，这对他们来说很不错。他们算是已经起步了，就是……Guest: And like they're now they've released their second model and they're getting going and they have Tinker has this like that's very good for them. They're like off the ground and it's just like

主持人：我不认为他们当初就是这么打算的。他们当时是在构建模型，但开放模型并不是他们最初的计划。后来他们还是走到了这一步，而且有很多……有传言说，Meta 会再次发布一个新的开放权重模型，这事会发生。外面有很多公司负担不起模型。Host: that I don't think that's what they were do. Like they were building models but open models wasn't their first plan. and then they got there and there's a lot there's rumors that like Meta said they'll release a new openw weight model again like that'll come there's a lot of companies out there that couldn't afford models.

嘉宾：是的，是的。我知道，我确实很高兴 Thinking Machines 是这样的，这对他们的商业模式来说非常有意义，嗯，因为他们会进行后训练。你知道，我想你在之前的一些文章里也提到过，对 Meta 来说，这确实是一个非常好的策略。我觉得他们这么做的时候，这一点也挺显而易见的。它和我的收入来源是互补的。Guest: Yeah. Yeah. I know I'm definitely you know glad that thinking machines is and it makes so much sense for their business model um because they do post training um and you know with meta I think you mentioned this as well in some previous writings it just for them it's just such a good strategy. I think it was also just kind of obvious when they they do it. It's like it's complimentary to my, you know, revenue streams.

主持人：Mark 写了关于这件事最棒的博客文章。比如 Llama 3 发布时，他写了一篇特别棒的文章，讲为什么开源才是未来的发展方向。Host: Mark wrote the best blog post on this. Like when Llama 3 came out, he wrote like the best like why open his path forward.

嘉宾：对。对。Guest: Right. Right.

主持人：盖伊，盖伊有点分心了。看来他要回到……Host: Guy got Guy got a little distracted. Seems like he's coming back to

嘉宾：对，对，是啊，是啊。所以我希望他们能继续保持动力，去做这件事，努力通过预训练抵达前沿。你知道，有这么好的开源模型，能在某种程度上以它们为基础继续发展，确实帮了很大的忙。我一直收到相互矛盾的信号，不太确定预训练是否真的已经饱和，或者说已经被充分理解了。也许正如你所说，有很多行之有效、广为人知的技术，但难点在于能不能实际用上两万、三万块芯片。大家都在不断加大预训练的投入；而在超长上下文生成出现之后，预训练也随之进行了调整。所以，如果你能让稀疏注意力架构也顺利运行起来，回报就会大得多——比如，生成十万个 token 能快上 40%。Guest: Right. Right. Yeah. Yeah. So I hope that they they keep on being motivated to do that. um getting to the frontier with pre-training. You know, I think there's a lot of help given that there's such good open source models to maybe, you know, somehow still kind of bootstrap from. I I keep on getting kind of conflicting signals as to like whether pre-training is actually as saturated or well understood. I mean, maybe maybe to your point like there's a lot of like well-known techniques that work, but what's hard is just actually having access to the 20,000 30,000 chips. people are turning the crank on pre-training and pre-training became adjusted when super long context generation happened. So there's just way more return if you're if you can for example like get a sparse attention architecture to work as well and then your 100,000 token generation is 40% faster.

所以我觉得，整个技术栈存在大量共同演进：他们有自己的预训练技术栈，同时发现推理扩展带来的收益高得多；但预训练的基础环节也在不断改进，只要架构稍微好一点，投资回报就会更高。然后，这些工作又逐渐与后训练交织起来，比如中期训练——你开始培养推理行为，把它们从旧模型中迁移过来，再对它们进行强化学习，形成种子行为。这个循环要比预训练复杂得多。我觉得，如果没有合适的人才，要把这套流程搭建起来，比我想象中做预训练要困难得多。比如，我觉得有条不紊地说“我们就从头开始预训练自己的模型”，这对我来说似乎更容易上手；而我对语言模型其实了解不多，要让我去摸索整套后训练流程，就像闯进了西部荒野。So I think there's a lot of co-evolution of the whole stack where they have their pre-training stack they realize that inference scaling is so much higher but even the fundamentals of pre-training keep improving where you just have a marginally better architecture returns on investment and then it kind of blurs into posttraining where this mid-training where you start the reasoning behaviors you take them from your older models you do RL on them the seed behaviors like this cycle is much more complicated than pre-training like I think it's if you don't have the right people it's a lot harder to get that off the ground than I think pre-training was like I think there's like I think like to do a controlled like let's just start pre-training our model seems a lot more approachable to me then I don't really know much about language models I'm going to figure out this whole posttraining thing where it's just like the wild west.

主持人：是啊。所以我离开 AI2 的时候就在想，我觉得我们的后训练确实落后了，纯粹是因为我们的迭代周期不够快。Host: Yeah. And that's why I was like when I left AI2, I felt like our post training was honestly falling behind just cuz our cycle time was not high enough

嘉宾：我们正在谈到这些智能体相关的东西，而这些仍然还是一些……Guest: and we're going into these agentic things and these are still some of the like

主持人：领先的开源后训练方案和Host: leading open source post training recipes and

嘉宾：所以这就像是……这就是我想修正的地方，就像……Guest: to that's just like that's what I want to fix is like

控制框架与智能体群落Harnesses and agent swarms

主持人：制造它所需的资本没那么多。Host: it doesn't take as much capital to make

嘉宾：在最先进技术和开源领域，复合式的进步空间非常大，因为门槛很低。这两者根本不在一个水平上。是啊。是啊。不，这确实说得通。关键在于，资金投放的效率很高，能带来巨大的改变。是啊。是啊。这可能也和整个后训练领域有关。你知道，闭源和开源之间差距不断拉大的一个原因，可能是这些智能体，以及闭源实验室所使用的那类运行框架，都非常不一样。而且，模型实际上就是针对这些运行框架进行训练和后训练的，所以它们特别擅长适配各自的框架，这就让事情显得更加神秘，也让问题变得非常难。你怎么看？首先，你认同这个判断吗？其次，你们具体希望如何应对这个问题？Guest: very big compounding gains on the state of the art and the open source because the bar is very low. It's like it's not like it's not close. Yeah. Yeah. No, that that makes sense. It's just like a kind of efficiency of where you can put capital and make a huge difference. Yeah. Yeah. kind of related to the whole you know post- training side maybe you know pulling away from closed and open is the fact that the agents you know the type of harnesses that are being used in the closed labs are very different and slash you know actually trained and you know post-trained on to to you know to to be specifically good for um their models are specifically good for those harnesses and so that in increases the mystique even more makes it very hard um how do you I guess one, do you agree? And then two, like how are you sort of hoping specifically to kind of address that?

我说的事情其实就是一些例子，比如说，嘿，启动几千个智能体，甚至几万个智能体。这显然和人们所接触到的工作流程截然不同。And things I mean are really examples like, hey, spin up like thousands of agents, tens of thousands of agents. It's just like a very different workflow than what people are are obviously um exposed to.

主持人：我觉得开源模型应该没问题。因为几年前我曾以为，闭源实验室会像苹果那样，成为一体化公司，针对所用的工具框架来设计模型。这样能带来一定的规模经济和效率。但事实证明，很多人会根据不同的需求使用不同的工具框架，因此模型公司承受着压力，必须让自己的模型能够适配各种各样的工具框架。我原本以为，开源模型的弱点在于它们必须针对各种类型的工具框架进行训练。但最后发现，情况相当均衡，或者说已经达到了平衡：它们都得在一定程度上做到这一点。而我们现在看到的这种智能体集群趋势，OpenAI 和 Anthropic 会更早涉足其中。Host: I think it'll be okay for open because I thought years ago that say the closed labs would be like Apple. There would be the integrated company where they have the model designed for the harness involved. It gives them um like kind of economy, scale, efficiency. But it turned out that so many people have different harnesses for different things that they like to do that the models companies have been pressurized to make it so that their model can work in all sorts of different harnesses. And I thought it would be a weakness of open models that they have to be designed to train on all sorts of types of harnesses. But it ended up being pretty equilibriated or like it's at equilibrium where they all have to do it to some capacity. And then what we're seeing with this like agent swarms is that OpenAI and Enthrop are going to be earlier to it.

他们会知道如何设计自己的模型。他们会知道该怎么设计——我们管它叫什么来着？我们想把它叫作什么？一个用于智能体群的 harness。我不知道，真的不知道，但肯定会有个词来称呼它。它不是 harness，更像是一个协调系统。但最终，其他人也会想要使用它们。然后我觉得这会形成这样的周期：前沿实验室把这个推出去，然后他们就有了这个……They're going to know how to design their models. They're going to know how to design what do we call what do we call want to call a harness for a swarm. I don't know like I don't know but there's going to be a word for it. It's not a harness. It's like a coordination system. But then eventually other people are going to want to use them. And then I think it's going to be like these cycles where the Frontier Labs like they push this out and they have this

嘉宾：到目前为止，他们率先行动获得了巨大的回报。比如研究纳维-斯托克斯方程以及所有这些东西，回报是什么？我们其实不太清楚，我们不知道。他们在推动前沿。Guest: to date they've had huge returns on being first. like what is the returns on Navier Stokes and all these things like we don't really know like we don't know they push the frontier

主持人：假设回报会很高，而且这种边际意图、边际智能会带来不成比例的回报。我们不知道情况是否总是如此，但我多少有些预期会出现这样的循环：一旦你摸清了如何在一个框架中控制 10,000 个智能体，开放模型公司就会说：「哦，我们也能做到。」Host: in the assumption that there will be high returns and that this marginal intent marginal intelligence has like disproportionate returns and we don't know if that's always true but I kind of expect these cycles of like once you know the shape of how do you control 10,000 agents in one harness then the open model companies be like oh we can do this too

嘉宾：但也不是说蒂米会先做这件事，首先，他们没有电脑。Guest: but it's not like Timmy is going to do that first for one they don't have the comput.

主持人：是的，Host: Yeah,

嘉宾：这是天文数字般的计算量，但这些东西都有周期。英伟达的芯片会快得多，建成的算力也会更多。Guest: it's an astronomical amount of comput, but these things go through cycles. Nvidia's chips will get way faster. There's more comput built out.

主持人：所以，你非常确信那里其实没藏着太多东西，大概能猜到他们在做什么；等到这件事变得重要时，再让开源模型也尝试用类似的元框架进行后训练，不管它叫什么名字。我也觉得……Host: So, you're very convinced that like there's not too much hidden there that you can probably guess at what they're doing and then when it becomes relevant, let the open source models actually try to also post train on similar meta harnesses, whatever name. And I also think like

嘉宾：这些实验室的信息安全水平并没有他们想象的那么高。比如硅谷就以创意传播迅速而闻名，怎么说来着，喝杯啤酒什么的，然后大家下班后都会聊天。Guest: the the security at these labs in terms of information is not as high as they think. Like Silicon Valley is famous for like ideas traveling like what is it like one beer or something and it's like everybody chats after work

主持人：不过，这也正是 SSI 如此令人印象深刻的原因。事情就是这么回事。Host: except which is why SSI is so impressive. That's what's going on.

嘉宾：那是唯一的过去。但感觉就像我们都是彼此的科学家。如果他们取得了突破，就会想把这件事告诉朋友。他们也许会先藏着不说，可能藏上几个月，不过……Guest: That's the only past. But it's like we're all everyone's scientists. If they made a breakthrough they're going to want to tell their friends about it. They might hold it. They might hold it for a few months, but like

主持人：我不认为这里需要那么强的壁垒，比如信息安全方面的防护，以至于知识就会扩散开来。这很难保密。他们能胜出的地方在于，把更多推理能力融入基础设施，这样就能以更低的价格提供服务，获得更高的利润率，并在更长时间里持续下去。所以我非常有信心，前沿实验室能够以最低的成本提供智能服务——按瓦特、按标量来算。Host: I don't think there the amount of lock like information security you would need is just so high that knowledge spreads. It's hard to keep this secret. Like what you what they can win on is putting more inference into their infrastructure which lets them serve at a lower price and make higher margins and then continue this over a longer time. So I'm very confident in the frontier labs being able to serve the cheapest intelligence per per watt per scoral.

嘉宾：他们将能够进入需求最高的地方，提供最便宜的模型，这样就能创造收入，对他们来说会是笔好生意。问题是，他们不可能什么都做，而这正是经济中其他部分填补空缺的地方。Guest: They are going to be able to go to the places with the highest demand and have the like the cheapest models and that will be revenue creative like that will be good business for them. The question is like they can't do everything and that's where the rest of the economy fills in is like

主持人：所以只有编程才有利可图吗？当然不是。Host: is it just coding that is profitable then? Definitely not

嘉宾：这是前沿模型的一个很好的案例，因为它们会拥有护城河，就像它们拥有集成能力一样。Guest: good case for frontier models because they will have the mo like they have integration.

主持人：他们会表现得很好，但这是编程工作。Host: They'll do very well but it's coding

嘉宾：知识工作的整个经济是不是就是一个事物市场？那这可能对他们不利，因为他们能同时做所有事情吗？Guest: is the whole economy of knowledge work is the market for things and then it might not be good for them because can they do everything at once

主持人：不过他们也不必这么做。我觉得会有很多赢家。Host: or they don't need to though. I think there's going to be a lot of winners.

嘉宾：是的。Guest: Yeah.

主持人：是的。我是说，这就是我……我是说，这就是为什么我们投资前沿模型和初创公司，你知道，嗯，不是因为我们想要对冲，而是因为我确实认为，你知道，这在某种程度上印证了你的观点：有足够的……Host: Yeah. I mean that's what I I mean that's why we invest in the frontier models and in startups you know um not because we want to hedge but because I actually do think you know to kind of cooperate your view that there's enough

嘉宾：边缘领域那些更难的任务，需要非常专业化的数据，这些数据在某种程度上处于实现 AGI（或者现在你想怎么称呼它都行）的主流范围之外，嗯嗯，这就使得许许多多的赢家成为可能。是的，Guest: harder tasks at the fringe that requires a very specialized data that's kind of out of the splash zone for getting to AGI or whatever you want to call it these days um um that makes it possible to have many many winners. Yeah,

主持人：我确实很喜欢你关于 RSI 的观点，正如我们刚才稍微聊到的，实际来说，这就是成本和资源消耗呈指数级增长，因此 Frontier Labs 能拉开差距的程度显然是有限的——如果这么说没有过度简化你的观点的话。你还想补充些什么吗？因为我确实觉得，现在的讨论里有太多恐惧，大家担心前沿领域究竟有多少东西被隐藏起来，以及差距还在不断拉大。我认为推理方面的投入非常大，而短期内，我们正在学着如何引导推理带来性能提升。这是我们刚刚经历的一件事，就像……Host: I do like your RSI point as we talked a little bit about it which is just kind of say practically hey it's exponential cost and resource consumption and therefore there's a very obvious kind of limit to how much that um the Frontier Labs can pull away um if that's not too oversimplification of your view um anything else you kind of want to add there because I I do feel like the the the discourse now is just so so much fear on like how how much the frontier here is just hidden and also breaking away. I think there's so much spend on inference and that in the short term we are learning how to um corral inference into performance gains and this is like a thing that we just had and like

嘉宾：我们现在看到的情况是，人们花的钱太多了，而且……Guest: this is what we are seeing now is that like people are spending so much and

主持人：前沿实验室里的员工都在投入呈指数级增长的资源，而取得的改进却相对有限。不过，我们正在摸索怎么实现这一点，而且会有很大的进步。但我不认为这一定会是那种模型自我构建、快速起飞的局面。瓶颈在人类身上。所有的 scaling laws 都表明，算力呈指数级增长，性能却只是线性增长。指数级增长嘛，这让我很难不看好它。我会说，我们有纳维-斯托克斯方程，而且我认为数学特别适合——也许是最适合这种智能形态的科学领域。智能可以有很多不同的形式；这并不是说，如果它们真的——比如说，如果某个 AI 模型发现了某种与 Transformer 非常不同的东西，带来截然不同的特性，并解锁一种新型智能——那就不值得据此更新看法。Host: like all the employees at the frontier labs are spending exponential amounts relative to the improvements but like we're figuring out how to do this and there will be a lot of gains but I don't think that it's necessarily going to be this like fast takeoff when the models build themselves. It's human human bottlenecks. The all the scaling laws are exponential increase in compute for linear increase in performance and exp like that's just like it's hard for me to bet against this. I would do say that like we have Navier Stokes and I'm of the opinion that math is exceptionally well suited maybe like the scientific field that is best suited to this shape of intelligence and like intelligence can take many different forms and that's not to say that like if they do like if an AI model does say discover something that's very different than the transformer that gives very different properties and unlocks a new type of intelligence like it is worth updating on this

嘉宾：而且值得说的是，嗯，好吧，这确实不一样。但在那发生之前，我对自己的判断还挺有信心：这些公司会发展得很好，它们正以指数级增长的投入，来维持这种惊人的进步速度。有些 Claude 系统卡里也提到这一点，说要让这种进展变成快速起飞，可能还得再取得十倍的进步。眼下，它们是在帮助我们维持进步的速度，我觉得这是一种很好的均衡状态。但我确实觉得，实验室里有人会说，科学上的进展很难突破这一点，这也没什么；只是我们还不确切知道接下来会发展成什么样。Guest: and worth being like okay like that is different but until that happens I'm like pretty happy with my evidence of like the companies are going to do well they're spending exponentially more to keep this remarkable pace of progress that some of the clawed system cards say this they're like it would take say like 10x more progress to make this a fast takeoff and it's like for now they're helping us sustain the pace of progress and I think it's a good equil like it's a good equilibrium but I do think that like the labs have people saying the the science is like it's hard to overcome that and it's okay but it's like we don't know exactly what it'll go from here.

主持人：是的，是的。不，我觉得这多少让人安心，因为我们知道这里的输入和杠杆是什么。只是，我们一直在优化一种非常成功的自回归模式，而这种模式需要一定数量呈指数增长的资源，但资源显然是有上限的。嗯，所以希望开源也能保持强劲，不至于落后。Host: Yeah. Yeah. No, I I I think that's kind of com comforting because we know what the inputs and and lovers are here. It's just there is a very successful auto reggressive regime that we've been optimizing and that one just you know asks for a certain amount of like exponential resources which are clearly capped. Um and therefore open source hopefully you know keeps it strong as well. just it wouldn't lag behind.

嘉宾：在基础设施准备方面有大量工作要做。比如说，有 AI 能力的发展曲线，这本身不一定让我担心；但 AI 能力的发展曲线，加上几乎不存在的跨公共部门网络安全准备，以及所有这些事情……Guest: There is dramatic amount of work to do in preparing for infrastructure. Like there's the AI capability plot which doesn't necessarily worry me but the AI capability plot in the pretty much non-existing preparation of cross public cyber security and all these things like

主持人：需要更充分地做好准备，假设这件事即将发生。Host: there needs to be more preparation in the assumption that this is coming

嘉宾：然后还有一点，这才是更让我担心的事，比方说 RSI 即将到来并带来新的风险。现在已经存在一些风险，而且还会继续扩散。很多时候人们都说，封闭实验室会先达到那个阶段，然后开放实验室也会达到。我们正处在这样一个过程中，很多事情都可能引发问题。要想办法让社会做好准备，实在很难。Guest: than there is and then that's kind of the thing that worries me more than say like RSI is coming and unlocking new risks like there are risks that are here and that are going to continue to diffuse and many times have said close labs get there And then the open labs get there. It's like we're in this process for a lot of things that are going to cause some problems. And it's just like figuring out how to get society to prepare is very hard.

主持人：是的。Host: Yeah.

嘉宾：我的看法是，我们需要更多双眼来关注此事。我们需要更多科学研究，让社会觉得自己了解得更充分。Guest: And I come of the opinion that we need more eyes. We need more science so that society feels informed.

主持人：是的。Host: Yeah.

嘉宾：但也有其他合理的观点。而且我确实认为，不害怕快速起飞并不意味着你不担心风险。我不喜欢你。Guest: But there are other valid opinions. And like I do think that not fearing fast takeoff does not mean that you don't worry about risks. I don't like you.

主持人：不，我觉得这完全……是啊。也许我会……把一些脱离实验室、恰好正为此担忧的人介绍给你。好吧，非常感谢你，Nathan。今天聊得很开心。嗯，是啊，我真的很期待看看 Trillium 终于上线后会有什么动向。Host: No, I think that that's completely Yeah. Maybe I'll I'll send in uh some people your way that are defected from the lab worried precisely about this. Well, thank you so much, Nathan. This was a lot of fun and um yeah, I'm really excited to see what Trillium is um up to now that it's finally launched.

嘉宾：是的，非常感谢邀请我来。Guest: Yeah, thanks so much for having me.