AI 账单太高了：Eugene Cheah 讲透模型、算力与组织流程的成本优化

简要描述Brief Description

The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah， Featherless AI

目录Table of Contents

• Intro: Eugene Cheah, Featherless AI
• “规模”是什么样：每天 2 万亿个 token
  What "scale" looks like: 2T tokens/day
• 免责声明：适用对象
  Disclaimers: who this is for
• 这 4 个动作
  The 4 moves
• 步骤 1：衡量（停止多人共用同一个 API 密钥）
  Move 1: Measure (stop sharing one API key)
• 智能体型工作负载与流程型工作负载
  Agentic vs. process workloads
• 第 2 步：为模型选择合适的规模
  Move 2: Right-size your models
• 分类器：一种新型的低成本模型
  Classifiers: a new class of cheap models
• 用智能模型进行规划，用廉价模型进行构建
  Plan with the smart model, build with the cheap one
• 没人会为了省钱而切换模型。
  Nobody switches models to save money
• 向团队提供模型标签，而非模型名称
  Give teams model labels, not model names
• 启动你自己的微调模型
  Bootstrapping your own fine-tuned model
• 持续改进的影子流量
  Shadow traffic to keep improving
• 第 3 步：购买 GPU 算力
  Move 3: Buying GPU capacity
• 速度翻倍 = 成本变为 3 倍
  2x the speed = 3x the cost
• 突发流量与专用容量
  Bursty traffic & dedicated capacity
• 步骤 4：检查你的流程
  Move 4: Check your processes
• Token 并不代表完成的工作
  Tokens are not work done
• 别扔垃圾手榴弹。
  Don't throw slop grenades
• 问答：微调与数据集
  Q&A: Fine-tuning & data sets
• 问答：黄金追踪记录
  Q&A: Golden traces
• 问答：摆脱线束的束缚
  Q&A: Escaping harness lock-in
• 问答：缓存与自动路由
  Q&A: Caching & auto-routing

Intro: Eugene Cheah, Featherless AI

主持人：呃，感谢大家来到这里。呃，不太确定看到这么多人都在为 token 账单发愁，我该高兴还是难过。这是一场满员的工作坊。呃，没错，我们今天要聊聊代币经济学。先简单介绍一下背景，作为开场，好吗？我是 Eugene，来自 Felis AI，也是公司的 CEO 和创始人。呃，我还是阿拉伯语 AI 开源项目背后团队的一员，这是 Linux 基金会旗下的第一个 AI 模型。我们致力于开发下一代 AI 架构。呃，这就是我的一点学术背景。我的 Twitter 账号是 PicoCreator。不过，更重要的是，简单介绍一下 Felis AI：我们的使命是让每个人、每种语言、每种方言、每个国家都能使用 AI。我们通过打造全球最大的开放模型平台来实现这一目标，目前平台上有超过 40,000 个开放模型。Host: Uh thank you for everyone for coming here. Uh not too sure whether to be happy or sad that so many people are suffering from the token bill. It's a full workshop. Uh Yeah, that uh we're going to talk about tokenomics. So, a little bit of a a background with the kickstarting it, right? I'm Eugene from Felis AI. I'm CEO and founder of of it. Uh and I'm part of the team behind the Arabic AI open source project, the first AI model under the Linux Foundation. And and we work on next generation AI architecture. Uh that's a little bit of my academic background. And my Twitter handle is PicoCreator. But, but more importantly, right? Uh uh a little bit of background for Felis AI is that our mission is to make AI accessible for everyone, every language, every dialect, every country. And we do this We do this by creating the world's largest platform of open models, of over 40,000 open models.

这还包括像 Sunflower 这样的专门微调模型。它是乌干达首个专门针对当地语言的语言模型，此外还有许多其他专用模型。所以，我们谈的不只是最大的开源模型，还包括专用语言模型或针对特定用途的模型，呃，其中一些我稍后也会提到。而且，当我们能够在模型种类上实现规模化时，Felis AI 也能大规模服务客户。我们的一些最大客户每天实际处理的 token 数高达 2 万亿。他们的处理价格约为每百万个 token 0.3 美元，也就是每天 6 万美元。他们使用的约 512 张 AMD MI215 GPU。呃，针对这位客户，我们提供的是一个私有的 400B MoE 模型。呃，其性能大约达到 Opus 4.8 级别。And this includes specialized fine-tune like the Sunflower model, which is Uganda's first language model, specifically for their language, and a lot of other specialized models. So, we're not just not talking about the biggest open source model, but specialized language models or specialized use cases, uh some of it which I'll touch as well. And when we when we can go on scale in terms of variety of model, Felis AI also we serve customers at scale. So, some of our largest customers are literally serving 2 trillion tokens a day. And and they're And and they're serving it at around 0.3 per million tokens or 60K a day. And they're doing it on around 512 AMD GPUs, uh MI215. Uh and for this customer in particular, it's a private 400 Android BM OE model. Uh and they're serving it around Opus 4.8 class performance.

所以，它有很多……So, it's lots of

“规模”是什么样：每天 2 万亿个 tokenWhat "scale" looks like: 2T tokens/day

并发请求，以及大量 token 用量。这就是我们所说的规模。如果你看一下，每百万 token 的成本是 0.03，且属于 4.8 以上级别，这并不是你能从公有云市场上获得的东西。这是非常专业化的配置，专为我们所说的这种规模运行而设计。那么，这就引出了两个主要的免责声明。concurrent requests, lots of token volume. That's what we mean by scale. And if you look at it, 0.03 per million token at over 4.8 class, it's not something that is not something that you can get off the public cloud market. This is specially very specialized setups that is meant to run at the scale that we are talking about. So, that kind of uh leads in into two major disclaimers

免责声明：适用对象Disclaimers: who this is for

我先从这里开始说。第一，我们销售推理服务。所以，我在这里讲的所有内容都会涉及任何一家服务提供商。这里没有任何内容是 Fiddler 独有的。不过，为了说清楚，我们销售的是推理服务。因此，这些内容也分别来自我们的数据和经验。好吗？另一点是，这场研讨会会涉及很多大规模优化的内容。如果你所在的组织每月支出不到 1 万美元，这场研讨会可能就不太适合你。呃，我给低于这个指标的人一个建议：这个指标是随意设定的。如果你的支出不到这个数，你最好把精力放在为组织寻找有力的应用场景上，而不是试图优化 Token 支出账单。因为一个有力的应用场景能为你的组织带来的价值，远不止每月 1 万美元。that I'm going to start off. One, we sell inference. So, everything I cover here will cover the will will touch on any provider. Nothing here is specific to Fiddler's. But, to to for clarity, we sell inference. So, so this is from our data points, from our experience, respectively, as well. Okay? The other thing is that this workshop is going to cover a lot of optimization at scale, and it probably will not fit you if you're like spending less than 10K a month in your organization. Uh my word of advice for those who are below this this uh metric mark, and this is an arbitrary metric, is that if you're spending less than this, you're probably better off focusing on like trying to get strong use cases within your organization, rather than trying to optimize the token spending bill. Because a strong use case can bring your organization way more than 10K a month.

而且，而且那应该是你们主要关注的领域，而这不是我这次演讲的关注重点。所以，我们接下来要讲的是这次……And and that's and that's what should be your main area of focus, and that's not my area of focus for this talk. So, what we are going to cover in this

这 4 个动作The 4 moves

这个研讨会分为四个步骤，按四个主要顺序：衡量、合理调整规模、如何购买容量，以及如何……workshop uh is four moves in in in four major order: measuring, right-sizing, uh how do you buy your capacity, and how

第 1 步：衡量（停止共用同一个 API 密钥）Move 1: Measure (stop sharing one API key)

检查你的流程。所以，要进行衡量。没错。你会惊讶地发现，这有多重要。呃，你们有多少人知道自己的 token 总开销？能举个手吗？哇，只有一半人举手。好，那有多少人能按部门衡量？举手的那些人，请继续举着。看来又有差不多一半人放下了手。是啊，越往下看，能衡量的人就越少：到团队层面，再到用例层面。呃，我们接触过的客户中，至少有 30% 到 40% 一开始全组织都共用同一个密钥。然后他们就会问：“呃，我们该怎么衡量？从哪里开始？”所以，这不是某个供应商特有的问题。你得把密钥拆分开，按部门拆分，再按主要用例拆分，这样才能知道 token 都用在哪里。you check your process. So, measuring. Yep. You'll be surprised at how essential this is. Uh like, how many of you know your total token spend? Can I have a raise of hands? Wow, only half the room raised hand. Okay. How many of you can measure at the department level? Those those who have their hands raised up, leave your up. And so, I see about another half went down. Yeah. And it gets lower and lower down to the team level, to the use case level. Uh we like easily 30 40% of our customers that reach out to us, the first thing they had was a single shared key that everyone in the organization used. And that was like Uh uh how do we even measure that? How do we start from there? So So this is not and a vendor specific like you need to start splitting up your keys, start splitting up by departments, start splitting up by major use cases so that you know where all your tokens are.

嗯，从那之后，这些信息就可以用来指导你的工作负载了。所以，没错，发放 API 密钥不应该太难。嗯，了解这一点之后，对吧，你就可以相应地合理调整资源配置。比如，哪些团队和工作负载在消耗资源？用户在等什么？有些团队确实需要每秒生成更多 token 的速度。有些在午夜运行的后台代理，可能并不需要你付费购买的那种每秒 token 生成速度。嗯，没错，别给后台代理付费买 Ultra，这太蠢了。嗯，你也可以据此查看使用情况，从而更好地为你计划购买的专属容量提供建议，不管是用于开源模型，还是闭源模型。嗯，对了，你也可以查看哪个团队消耗的 token 最多，以及他们的工作效率如何。我也会谈到这一点。Um and this is then from then on it can inform your workload. So yeah, API keys shouldn't be that hard to issue. Um once you know that, right, you can start right-sizing things uh respectively. Like like which team is uh and workloads spend on it? What was the person waiting for? Like some teams deserve the tokens per second. Some background agents running at midnight probably can don't deserve the tokens per second speed that you're paying for it. Uh yeah, don't pay for ultra on a background agent. That's just dumb. Um and and from that also you can also look at your usage to better advise like uh dedicated capacity that you may purchase, be it for open models or even close models. Uh yeah. And also you can also check which team spend the most token and whether they are productive. Uh I'll touch on that as well.

关于 AI 工作负载，有一点你们应该知道，对吧，呃，还有你们所有人……One of the things that you should know in AI workloads, right, uh and you all

智能体工作负载与流程型工作负载Agentic vs. process workloads

你们可能已经有所体会了，对吧？这里有两条主要路径，它们的经济效益截然相反。一类是智能体工作负载。当然，两者之间也有一些情况，比如长期运行的多租户智能体。编码智能体就是一个典型例子。有些开发者可能会让编码智能体连续运行很长时间，甚至一整夜，并进行多次上下文压缩。这类任务会最大限度地利用上下文。另一端则是流程自动化。想想呼叫中心、邮件路由、会议记录转写之类的场景。总体而言，这些任务的上下文要短得多，尽管也有很多例子不符合这一说法。但它们非常依赖流程，任务之间高度隔离，本质上是一场拼规模的游戏。may already experience, right, is there are two major tracks with very opposite economics uh respectively. So there is the agentic workload and obviously there's things in between where it tends to be like long-running multitenant agents. Coding agents are a particular example of this. Some Some Some may have some devs like run coding agents like literally overnight over a long period of time with uh with multiple compaction. And and these things maximize the context uh usage. Process automation is the other extreme. Uh think call centers, think email routing, things your meeting notes transcribing, and things like that. It's all It's much shorter in context in general, though there are a lot of examples that defy this statement. But, it's very process-driven, very task-isolated, and it's it's a volume game, essentially.

那么，嗯，我说的使用量增加是什么意思呢？这可能是许多组织最先面临的一次重大激增。尤其是当他们赶上开放云的热潮，或转向某个云平台的克隆版时。比如说，我们支持的各个开发团队中，有一个团队只有五名开发人员，但他们平均会持续运行大约八个智能体。这是我们合作的一家公司里的情况。他们之前一直在使用 Opus 4.8 API，每个月持续运行这些智能体，花费基本上达到 15 万美元。其中一些智能体用于编写代码，另一些则会在夜间启动。他们算是稍微有经验一些，已经知道应该在晚上、没人工作的时候启动任务。So, well, what do I mean by more usage? And this is probably one of the first major explosion a lot of organizations face. In particular, when they jump onto the open cloud hype or one of the cloud clones. So, you like like from from from various developer teams that we we support, like for example, we sampled one of the team, and it was only five devs, and they were averaging around continuously eight running agents for one of the companies we work with. And And so, they were previously running Opus 4.8 of the API, and they were essentially spending 150k a month continuously running these agents. Some of it is the agents that they use to code, some of it is the agents that kick in at night. They They were slightly more advanced. They already knew that they they they they should kick things off at night when no one's are working.

但本质上，他们每个月要消耗 1160 亿个 token。我们还真的帮他们切换到了 GRM 5.3 实例。之前用的是 5.2。他们已经升级到了 5.3，而且当时只用了单节点。每月费用不到 1 万。他们也没有感受到任何中断或影响。因此，了解你的团队，就能帮助你做出相应的决策。正如我刚才强调的，了解流程和工作负载，也能让你把任务拆分开来。比如，他们之前全天都在运行所有任务，后来开始安排一些后台代理在团队休息时运行，并让它们整夜执行。这样就能更合理地利用他们的硬件。正如我刚才简单提到的，流程自动化的另一面纯粹是靠规模取胜。But, essentially, they were spending 116 billion tokens a month. And we literally helped them switch to GRM 5.3 instance. Previously, it was 5.2. They already upgraded 5.3, and that was like a single node. That was less than 10k a month. And they were And they didn't see any disruption or impact. So, knowing your team allows you to make these decisions respectively. And like like I highlighted, like knowing processes workload also allows you to split things off. So, like, for example, previously, they were running everything throughout the day, but they they started scheduling some of the background agents when the team was asleep, and it runs through night. So, this allows you to utilize their hardware respectively. The As I I touched on briefly, the other side of process automation is purely a volume game.

高度预定义。我可以私下给你一个更具体的例子，但你可以把它想成每天数百万甚至数千万次的决策：呼叫中心的决策、扫描代码行、过滤电子邮件和短信。比如，我们合作的一家电信公司，单分钟的峰值消息量轻轻松松就能达到 1000 万条。这也不足为奇。新年快乐那阵子最糟糕。对，对，对，这些……而且它们……它们之所以过滤消息，是因为它们提供了一项功能，比如为孩子提供安全手机。所以它们需要用 AI 扫描消息，判断其中是否含有脏话、色情内容等等。所以，这实际上不是长上下文的问题，而是吞吐量的问题。其他情况还有制单-复核流程、审批流程、文档处理。Extremely predefined. I I can give you more specific example off the off the record, but you think of it as like millions per day or even tens of millions per day, decisions in call centers, lines of code scanned, emails and text messages filtered. Like, one of the telcos we work with, like, they easily hit peak 10 million messages in a single minute. And this shouldn't come as a surprise. Happy New Year is the worst. Yeah. So So So these And they And they help And the reason why they filter messages is because they provide a feature about like providing safe phone for kids. So they needed an AI to scan like basically is it profanity, sexual content, etc. etc. So So this This is really not a long context, but a volume game. Other things like Other things is like maker-checker processes, approval processes, document processing.

这些都是面向多个组织的规模化业务。所以我怀疑，对于这里的金融机构来说，毕竟我们在纽约，很多 OCR、很多文档、很多审批，以及很多交易风险都属于这一类。好吧。是的。如果你还没有在跟踪自己的 token 支出和 token 视图拆分，那么我们正在开源一款 tokenomics 跟踪工具。这只是一个听起来很 fancy 的说法，指的是拆分你的 API 密钥，并跟踪每个 API 密钥消耗了多少；这和今天的演讲是分开的。是的。所以，如果你想再次更新它，欢迎扫描这个二维码，分别……如果你想获取这场演讲的幻灯片，并在 AI Engineering 发布录像时收到有关这场演讲的通知，也可以填写那里的表单。好吧。These are all volume game for multiple organizations. So I suspect for the financial institutions here, since we are in New York, a lot of OCRs, a lot of documents, a lot of approval, a lot of transaction risk the is is under this category. Okay. So yeah. If you're not yet tracking your token token spend and token view split, right, we are open sourcing a tokenomics tracking tool, which is just a very fancy way of like splitting up your API key and tracking which API key spends how much, which is separate from this talk. Yeah. So if you If you want to again update it, feel free to to scan this QR code and respectively. It also if you want to slides for this talk and when the recording is released from AI engineering to be notified about this talk, you can also get fill up the form there. Okay.

好。好。所以，是的。刚才讲的是衡量。嗯，是的。如果你还没有进行衡量，那基本上可以忽略我接下来要说的所有内容，先从衡量开始，因为你能获得的最大投资回报率，基本上都来自衡量。不过，一旦你开始衡量了，对吧，接下来要问的问题就是……Okay. Okay. So yeah. That was about measuring. Um yeah. If you are not measuring, you can pretty much ignore everything else I say and start with measuring first because all your biggest ROI is just by measuring. But once you have things measured, right, the question is

步骤 2：为你的模型选择合适的规模Move 2: Right-size your models

如何为它确定合适的规模？这时候事情就变得更复杂了。嗯，总的来说，随着 AI 在组织内部大规模部署，我们开始观察到一种现象：因为 AI 是一把闪亮的新锤子，我们就想把它用在所有事情上。尤其是在创新实验室里，我们恨不得把它用在每一件破事上。我见过一些组织，真的把 Fable 用在了最无聊的事情上。所以，模型很多，其中不少模型的成本也非常高。弄清楚该怎么做、该用在哪里，是整个流程中非常重要的一步。我知道开源模型——我想你们很多人可能都听说过开源模型，而这正是我们的业务领域。是的。所以，很多公司来找我们，比如想找到适合自己用例的开源模型，以及可以直接替换现有方案的模型。how do you right-size it? And this is where things gets more complicated. Uh In In general, right, one of the things that we start to observe as as AI gets deployed at scale within organization is that there tends to be a pattern of like because AI is the shiny new hammer, we try to apply it to everything. Especially within the innovation labs, we just try to apply to every damn thing. And we tend and and I have seen organizations literally apply fable to really the silliest of things. So so you there's a lot of models and a lot of models are really expensive and and and and and figuring out to do and where it fits is is is important is an important step of the process. I know open models I mean a lot of you probably heard about open models and that's our line of business. Yeah. And and so a lot when a lot of companies come to us for example to like find open models that fit their use cases and drop in replacements.

我们通常会根据他们的使用场景来提供指导。这些只是一些通用经验法则，可能每个月都会更新，因为开放模型每隔几个月就会发布一次，闭源模型也是如此。变化很快。不过，真正令人兴奋的是，我们发现今年以来情况开始发生变化。去年，最好的闭源模型几乎总是优于最好的开源模型。但随着今年的推进，我们开始看到差距正在缩小。有些情况下，某些开源模型甚至不相上下，乃至超越闭源模型。比如，在 Opus 5.5 更新发布之前，Kimi K3 在一些 UI 基准测试中的 UI 设计表现就胜过了 Opus。而且这些都是非常公开的基准测试。This is we generally guide them along their use cases. So these are these are general rule of thumb that is probably going to be updated month because open models release every few months and same thing close source models. Things change rapidly. But but there but the thing is what we find right exciting actually within this year is things have started to change. Like in the previous year the best close source model is without better than the best open source. Almost. But as this year approach right we are starting to see the gap close. We are starting to see situations where where some of the open models are even on par or even surpassing. So like for example before the Opus 5.5 update came out Kimi K3 was beating Opus in UI design in some of the UI benchmarks. And and this is like very public benchmarks.

在客户基准测试中，我们已经开始看到，开源模型在特定用例的专业基准测试中表现出色。比如，值得特别提一下的是 Step 1.5 模型。它在金融基准测试和金融用例中的表现，达到了接近 Opus 和 Extra 的质量水平，尽管它的规模小了 10 倍。它是最好的编程模型吗？不是。但在那个用例中，它确实非常强大、能力出众。因此，整体格局正在慢慢变化：许多用例都可以找到即插即用的替代模型，有时成本甚至能低 10 倍——前提是你了解自己的工作负载，并且能够相应地替换模型。还要特别提一下 DeepSeek-V4 和 Flash。不知为何，这个模型在各种工作负载中都非常受欢迎，因为对很多公司来说，它很容易提示。Within customer benchmarks right we were already starting to see open source models beat very specialized benchmarks in specialized use cases. So like A particular one to highlight is like for example, step one five model. They it was able to achieve near Opus and extra quality despite being like a 10 x smaller model in financial benchmarks uh financial use cases. And this is it the best coding model? No, but in that use case, it's actually extremely strong and competent. So so so the landscape is slowly changing where where where you can have drop-in replacement for a lot of use cases that are sometimes even just 10 x cheaper respectively for the uh uh if if you know your workload and you can drop it in respectively. Special shoutout is also DeepSeek-V4 and Flash. For some reason, and this model is extremely popular across all workloads respectively because it's easy to prompt there uh for a lot of companies.

所以，嗯，我……我列出的其余模型，在各自类别里基本都是默认的优选。DeepSeek-V4 和 Flash 似乎都挺受欢迎，不管具体情况如何。是啊。呃，不过基准测试分数似乎并不能解释它们为什么这么受欢迎。是啊。嗯，还有一点值得强调，对吧？最近，也就是几周前，出现了一类新的 AI 模型，它们已经……So well I well the rest of the models I have listed there they are they are like default for good in their categories. DeepSeek-V4 and Flash seems to like be quite popular regardless the case. Yeah. Uh though benchmark score doesn't seem to justify why why is it so popular. Yeah. Yeah. Uh another thing to highlight, right, is that recently like and this is like for a few weeks ago, uh the a new class of uh AI models have

分类器：一类新型低成本模型Classifiers: a new class of cheap models

已经出现了一种叫作分类器的东西，呃，一次性分类器，最广为人知的是 JAF。开源项目已经开始兴起，比如 Simple JAF。这些模型完全专注于对任务进行决策和分类。呃，主要优势在于，单从技术层面来说，它们通常比现有模型便宜 10 倍，甚至速度也更快。呃，缺点是它们无法生成聊天回复。所以，所以，所以这些都是正在发生的变化，你们可能会想开始研究如何改进自己的 Token 经济，比如从不同的细分方面入手。对。不过，回到 Token 经济这部分，这里有两条主要路径，重要的是把它们区分开来，并分别跟踪。been introduced called uh classifiers one-shot classifiers most popularly known as JAF. Open-source projects have already started uh taking off like simple JAF. And these are models that are entirely focused on just making decisions and classification on on tasks. Uh the main advantage is that because on a technical level only, they are commonly 10 x cheaper than existing models or even faster at the same time. Uh the downside is it cannot give a chat response. So so so these are things that are changing and happening that you all may want to start looking into on like on like segments of how you want to improve your your token economy. Yeah. But but back to the The the the token economy part, there is two major tracks and it is important to separate out and track this track them respectively.

在智能体方面，正如我之前所说，我们讨论的是长时间运行、长上下文的工作负载。你得明白，这类工作负载的缓存开销非常大，输入预填充（prefill）占比也非常高，而输出只占很小一部分。我们观察到，在生产环境中的各类智能体用例里，输出 token 只占请求的 1%。这很合乎逻辑，因为它需要多轮交互，反复进行。不过，这也会影响你的 token 成本。你知道 token 成本，对吧？部署成本，对吧？输出的单价可能最高，但即使输入单价低得多，或者缓存成本低得多，在这类用例中也会被大量重复使用。是的。这不是一场关于编码智能体的研讨会，不过这是我们在很多组织中看到的最常见模式之一，对吧？It on the agentic side we're talking about long running long context as I said earlier. And to and you have to understand this workload tends to be extremely cash heavy, extremely input preview heavy and cash heavy and there's only a small percentage of output. We see across like agentic use use cases in employment only 1% of the request is to the output tokens. And and and this is this is very logical because it takes multiple turns, rinse and repeat. But this also decides your token cost because like if you've seen token cost, right? Deployment cost, right? Output is may have the biggest number, but even your input value is much smaller or the cash is much smaller, it it gets repeated heavily in in this use case. Yeah. This is not coding agent workshop, but one of the most common patterns that we we see in a lot of organizations, right?

这通常是指试图在编码团队内部扩大他们的 AI 智能体部署规模，对吧？Is that is that typically typically in terms of like trying to scale up their AI agent deployments we within within the coding teams, right?

用智能模型进行规划，用低成本模型进行构建Plan with the smart model, build with the cheap one

他们往往会让最强的智能体负责规划，再让一个更快、更快也更便宜的智能体负责实现，之后可能再让最强的智能体进行审查。这种分工在各种智能体框架中都很常见，是有充分理由的：这样可以兼得两者之长，因为实现阶段往往是一个非常消耗 token 的过程。很多操作会来回重复，比如修改文件、测试、检查结果，周而复始。而且，这些智能体中的大多数，尤其是 Gemini 5.3 Flash，都完全有能力以与强得多的智能体（比如 Astra 或 Opus）相同的质量水平完成任务，前提是由那些智能体负责规划。我们曾与一些组织合作，他们实际采用的做法是：由最强的前沿级“肌肉模型”负责规划和审查，而负责实现的工作模型则是 GRM 5.3 Flash，或者 Deep Seek 4.1 Flash。They tend to have the best agent do the planning and a faster a faster cheaper agent do do the implementation and then maybe the best agent do the review again. There's a good reason why this split is very common across all the agent frameworks is is that you get the best of both worlds because like the implementation phase is tends to be tends to be very token heavy process. A lot of processes repeated back and forth like key modifying the file, testing, checking for results, rinse and repeating. And and you and most and most of these agents are especially this especially like Gemini 5.3 flash is more than capable of doing a task at the same quality level as a much more stronger agent like I say Astra or Opus, uh respectively assuming those agents do the planning. We have worked with some of the organizations where literally the planning and review agent is the best frontier close muscle model, and then the workers that implement it is the GRM 5.3 flash as an example, or Deep Seek 4.1 flash.

是的。我们从这种、从这种分裂模式中观察到的一点，对吧，是……Yeah. One of the observations that we made from from this from from this kind of split pattern, right, is that

没人会为了省钱而切换模型。Nobody switches models to save money

遗憾的是，尽管我们都希望每个人都这么做，但实际上没人会为了替公司省钱而切换模型。嗯，人们通常懒得切换模型。你得提供激励，促使他们切换。或者，如果你的代理框架足够好，比如某些像 M 这样的框架，就能把这个流程自动化。比如，它会把较弱的模型用作基线，然后在需要更聪明的咨询时，自动切换到更大型、更聪明的模型。这能相应地减少摩擦，并有助于在用户规模下改善你的 token 成本效益。比如，经过几次实践后，我们现在给客户的一条主要经验法则是：你需要更聪明的模型；但既然它作为基线已经很贵了，你也希望更快的模型——也就是更便宜的模型——速度更快。unfortunately, as much as we wish that everyone does so, nobody really switches their model to save the company money. Um this people are lazy to switch the models in general. You you want to give an incentive to switch. Or or alternatively, if your agent harness is good enough, so like for example, some harness like M, they automate the process. Like like it uses a the weaker model as a baseline, and then it uses a big the smarter model to automate switching when it needs a smarter consultation. This This helps reduce the friction respectively, and helps improve your your token economics at user scale. Like a a major rule of thumb that we we now give our clients after going through a few times is that you want the smarter model, and since it's already so expensive as a baseline, but you but you want the the faster model to the cheaper model to be faster.

这自然就形成了一种激励。人们想尽快完成工作。如果模型足够聪明，能把事情做好，而且价格更便宜，他们就会用它，因为它速度更快，而不是因为它更便宜。嗯，我们遇到过一些团队，他们使用的两个模型，一个更聪明，另一个更便宜，而后者的每秒 token 速度也保持不变。只要他们的 token 预算充足，对吧，就没人会切换。反过来，我们也见过处于另一个极端的团队：他们把预算设得极低，所以一旦使用更聪明的模型，对吧，一天之内就会把预算用光。是啊，于是大家就都切换到 Dharma，也就是那个更快的模型，因为他们的预算就那么多。这几乎就像是更聪明的模型根本不存在一样。And that naturally sets an incentive. People want to get their work done fast. If the model is smart enough to get it done and it's cheaper, they will just use it because it's faster, not because it's cheaper. Um we had teams where where where both models were the smarter one and the cheaper one was maintained at the same tokens per second speed, and it and as long as they had the token budget in excess, right, no one switched. And it and on the flip side, we have seen teams, which is the other extreme, where they set an extremely low budget, and so that if they use a smarter model, right, they just burn it in one day. Yeah, then everyone just switch to the to the Dharma into the faster model Dharma and faster model because that's their budget. It's like almost as if the smarter model didn't exist.

所以，要留意团队在切换时的动态。最好的权衡方式是速度，因为人们自然会受到速度的激励。所以，他们会分别切换到更快或更便宜的方案。是的。再提醒一次，要进行衡量，并且要分别衡量两个独立的 token key。呃，不过这也不只是衡量的问题。比如，CI/CD 现在是一个很常见的应用场景，会有任务在后台运行。我们帮助一些团队设置过不同方案：面向用户，也就是人类使用者的任务，采用每秒 token 数更高的方案；而 CI/CD 流水线则采用每秒 token 数低得多、但大规模运行时便宜得多的方案。呃，这本质上取决于不同任务各自的优先级。是的。你可以在 key 级别控制这一点。So, like be aware about your team dynamics on how they switch. The The best trade-off is to speed because people are naturally incentivized for it. So, faster, cheaper, they will just switch it to it respectively. Yeah. Once again, reminder, measure, and you measure two separate token keys. Uh and but this also goes beyond just measuring. For example, like CICD is a very common use case right now of things running in the background. Uh we we have helped set up for teams like faster tokens per second that that is running for the the users, like the humans, but much slower tokens per second um speed, but much cheaper at scale for the CICD pipeline. Uh and and this is just the nature of like what's your priority respectively respectively across the task. Yeah. And you can control that at a key level.

所以，另一个重要发现是，我们有面向团队部署的经验，对吧？呃，举个例子，有些公司里，有多少人有幸或不幸地听过 CEO 说：“现在每个人都必须实施 AI，而且每个人都必须有一个第二大脑。”是的，已经有多个组织开始这么做了。呃，在某些情况下，比如甚至在阿联酋，这件事实际上成了国家法律强制要求的。所以，很多部门都还没准备好去了解各自需要做出什么权衡。一个不错的经验法则是，对吧？开发人员通常知道，好的，这些智能模型，比如 DeepSeek Flash 和 GR553 之间有什么取舍？但你们的大多数财务部门完全不知道。你们的支持部门也完全不知道。So, another major finding is that we had experience with deploying for for teams, right? Uh uh So, like for example, for some companies, uh how many of you had the fortunate or unfortunate experience of the CEO saying, "Everyone must implement AI now, and everyone must have a second brain now." Yeah. Multiple organizations have started this. Uh in some cases, like even in in the UAE, it became literally mandated by by by by the law of the country. So, so, like a lot of departments are not prepared to know what is the trade-off respectively. So, a good rule of thumb, right? Well, developers tend to know, okay, all these smart models, what's the trade-off uh between deep seek flash and GR553, for example? Most uh your your finance department have absolutely no idea. Your your support department has absolutely no idea.

他们，他们大概会随便选第一个模型，然后就直接用它了。所以They they'll just pick whatever's the first model and probably just go with it. So

为团队提供模型标签，而非模型名称Give teams model labels, not model names

所以，我们通常的经验法则是给部门模型贴上标签，而不是使用模型名称。这样在切换模型时，你也能获得更多控制权。呃，就像前面建议的那样，可以区分快速型和智能型；但在某些情况下，也可以使用财务智能体或 Excel 电子表格智能体。这些都是针对相应任务微调的高度专业化智能体。这样也能让你更好地掌控局面：可以更换模型，也可以拆分流量，在不同模型之间进行测试，而不必告诉用户：“嘿，请试试这个。”缺点是，如果你这么做时搞砸了，用户有时也会抱怨。不过，拥有这种控制权是件好事。呃，我们还看到另一种模式：即便你这么做，可能还是会用某种带有“-preview”后缀的版本。So, our general rule of thumb is that give department models labels, but not the model names. And this also gives you a lot more control as you switch it out. Uh what like like advised earlier, fast versus smart, but in some cases also like finance agent or Excel spreadsheet agents. Um these are hyper-specialized agents that were fine-tuned to those tasks respectively. And and this also gives you a lot more control. It allows you to swap up the model. It allows you to split split the traffic and test in between with without without telling, "Hey, users, please try this." The downside is also users may sometimes complain when you do that and you screw that up. But having that control is a good thing. Uh another pattern that we see, even though you do this, is maybe some of a dash preview version.

所以，这是它的一种变体，但它能让你掌控局面，同时又不会增加整个组织的沟通负担。这极其重要，因为我们合作的一些团队就是这样，对吧？我们真的看到有成员还在运行 Llama 2 模型——我们今年谈到这个问题时，竟然还有人在用，因为没人切换选项。是的，所以你确实、确实需要，呃，需要把这件事落实到位。好。流程赛道。流程赛道。我之前提到的很多事情，对流程赛道来说其实没那么适用，因为流程赛道往往与应用和工作流高度集成，分别如此。嗯，这主要是由流量的庞大规模决定的。嗯，是的。而且在流程赛道的所有团队中，对吧？呃，他们通常从一开始就更注重合理配置，因为如果你要处理数万亿个 token，大多数部门都不会批准每百万个 token 花 10 美元。So, that's a variant of it, but it gives you control without without taxing the communication burden across the organization. And this is extremely important because like like some of the teams that we work with, right? We literally saw members running Llama 2 models when when we are talking when talking about like in this year because no one switched the option. Yeah, so you really really need to need to uh need to set this in place. Yeah. All right. Process track. Process track. A lot of things that I mentioned earlier don't really apply as much to process track because process track tends to be a lot integrated into the applications, into the workflows, uh respectively. And and that's just by the sheer volume of the traffic. Um yeah. And And of all of teams in the process track, right? Uh they tend to be a lot more focused on focused on the right sizing already by default because because if you're running like trillions of tokens you most departments will not approve it like $10 per token million token.

这就是不可能的事。所以，所以他们会很早就开始做评测、跑基准测试，然后再找到一个规模合适的模型。嗯，对。所以，常见的问题是：我怎么知道小模型够不够好？你不需要……这不是该问的问题。你只需要确保设置好测试用例、评测和示例，然后再从这一步开始搜索。像我们这样的公司很乐意帮你，因为我们有超过 4 万个模型。不过，如果你愿意，直接从 Hugging Face 下载 10 个模型试一试也没那么难。嗯，所以你只要搜索，找出价格最低、能通过大部分测试的模型，然后看看接下来该怎么迭代、怎么微调。It's just not possible. And and so so so they start they start early with the evals, the benchmark, and then subsequently finding the model that right size fits it. Uh yeah. So So the thing is like the common question is how do I know the smaller model is good enough? You don't need to That's the wrong question to say. You just need to make sure you set up your test cases, your evals, and your examples. And then subsequently just do the search process from there. Uh companies like us will be more than happy to help you because we have over 40,000 models. But then But then if you're It's also not that hard to just download 10 models from Hugging Face and just try. Uh So So from that you you just do the search, find the cheapest model that kind of like pass most of it, and then you just see how you iterate from there, fine-tune from there.

嗯，我们还有一个更具体的流程，我觉得这并非我们独有，我们也会介绍一下。比如说，我们合作的一家大型电信运营商，对吧？我们通过这样一个流程，帮助他们的团队把推理成本从每月 25 万降到了 7,600：他们原本使用的是闭源模型。流程大致是：“好，先用最好的模型跑起来。看看有哪些模型可以拆分出去，接下来看看哪些模型可以容纳进去，然后再看看哪些模型可以微调。”这是为了满足客户支持等使用场景。是的。再提一下 Jeff 和 Simple Jeff，这是我们做的一个开源项目，用来复现这个流程。你会惊讶地发现，你的 AI 流量中有多大比例其实是分类流量。Uh we also have a more specific process that I don't think is unique to us that we'll cover as well also. So for example, like in for one of the major telcos that we work with, right? We actually helped their team lower their inference cost 250k to 7.6k a month switching from existing close-source model just by going through this process of like, "Okay, you're running the best model first. Let's see what what models we can split off, and then subsequently what's the models that we can fit in in, and then what can we fine-tune?" And this was This was like to uh beat the customer support use cases and and so on and so forth. Yeah. And to re-highlight back Jeff uh and simple Jeff, which is a open-source project that we did uh that replicates it. You'll be surprised at how many mpercentage of your AI traffic is classification traffic.

嗯，说到底，对吧？就是，没错。当然，回答客户支持问题是文本形式的。但即便如此，对吧？即便如此，对吧？很多团队，对吧？已经有预先定义好的文本输出，会把这些内容放进模型的上下文中，让模型在大多数情况下输出这些内容。而且你可以在这里使用分类器。还有很多……这是 AI 领域内一个全新的新趋势。而且这是这类流程中降低 token 费用的重要杠杆之一。是的。举一个这类模型用于微调的例子。很多模型和流程，对吧？嗯，在达到一定规模后，确实能从微调自己的模型中受益。嗯，另一方面，对于编程模型来说，也有一些机构会替你微调模型，比如 Devin。嗯，很多开发者通常就是想要最好的模型。Uh At the end of the day, right? Like Yeah, sure. Answering the customer support is text. But even then, right? Even then, right? A lot of teams, right? Already have predefined text output that they put into the context for the model to output for most cases. And and you could use a classifier there. And there's a lot This is a new new new shift within AI world. And that this is one of the big levers for this kind of processes in lowering your your token bill. Yeah. So, one one one example of such a model when it comes to fine-tuning. So, a lot of models and processes, right? Uh do benefit at this scale where you start to fine-tune your own model. Uh on the flip side, for coding models, well, there are organizations that will fine-tune models for you, like Devin, for example. Uh a lot of developers generally just want the best model.

所以，除非你身处 AI 领域，否则要跟上微调最佳编程模型的步伐确实有点难。但另一方面，针对你的领域，为具体用例微调一个领域模型也许值得一试，尤其是模型小得多的时候。比如 AT&T，他们微调模型时，使用的模型从开放类别一路缩小到大约 27B，甚至 7B。他们已经发布了这些模型。我们是他们的首批合作伙伴之一。你实际上可以分别试用他们的开放对话模型。这基本上就是他们在幕后使用的模型。这里关键在于规模。我想你们大多数人都了解 AT&T 的规模、呼叫中心等情况。嗯，是的。这是一个很好的例子，说明如何通过微调模型，让它们专门适用于特定用例。So, it's kind of unless you are in this AI space, it's kind of hard to keep pace in fine-tuning the best coding model. But on the flip side, for your domain, it might be worth it to fine-tune a domain model for your use case, especially if the model is much smaller. So, for AT&T, for example, when they fine-tuned their models, they they they went from being open class to like 27B to even 7B models. And they have published this. So, so we were one of their launch partners. And you can actually try their open dialogue models respectively. And and This is basically what they use behind the scene. And and it's all about volume here. Like, I I think I think most of you understand the scale of AT&T and the call centers and things like that. Uh yeah. But it's a good example of how you fine-tune models to to specialize use cases.

而一旦你达到那个阶段，像 Shopify 这样的大型组织也开始这么做了。而且，这不一定需要进行大规模微调。比如在 Shopify 的案例中，他们这么做是因为 Shopify 网店使用一种专有编程语言来配置网店，而现有模型在这方面表现不太好，所以他们就对模型进行了微调。呃，AT&T 也有这类案例，特别是如何规划基站布局之类的。对，这是非常电信行业特定的任务。我可能不是回答这个问题的合适专家。呃，不过他们确实有这类应用场景。所以，他们进行了微调。And once you reach that point, other organizations that are major example of like Shopify have started doing this. And and things and and it doesn't necessarily need to be a large-scale fine-tune. So, for example, in the Shopify use case, they did it because Shopify web store had a proprietary laprogramming language for how to configure the the web store. And the existing models just weren't that great at it, so they went to fine-tune it. Uh but and AT&T also they had examples of that uh in particular on like how to arrange the the cell towers and things like that. Yeah. Uh very telco-specific task. I might not be the right expert to answer that question. Uh but they had those use cases. So so they fine-tuned

引导启动你自己的微调模型Bootstrapping your own fine-tuned model

分别如此。你也可能有这些用例。那么，如何引导启动这样的模型呢？好消息是，如今你可以轻松地引导启动这类模型。因为假设你愿意——如果最好的模型虽然运行成本太高，无法大规模部署，但能完成这项任务，那么你就可以着手开始了。那有什么简单的方法呢？我们很常见的一种做法是，要么生成自己的数据，要么准备自己的数据，而这很难。不知为何，没人喜欢这么做。另一条常见路径是，先从一个昂贵的模型入手，小规模部署，然后用它生成数据集。我知道，有些实验室的服务条款禁止这么做，所以你可能需要咨询律师；但也有些实验室对此持公开支持的态度。respectively. You too may have that those use cases. And how do you bootstrap such a model, right? Is that the good news is that uh you can easily bootstrap things these days because if let's say you're willing uh if the best model that is too expensive to run at scale can do the task, you can already get started. So so so an easy way to do this, right? Uh that we see very commonly is like either you generate your own data or prepare your own data, which is hard. No one likes to do that for some reason. But that but another common path is that they start with an expensive model and then they deploy in a small scale and then they generate they they generate data set. I understand that some labs it's against their terms of service to do that, so you may want to check with your lawyers, but there are labs that openly embrace it.

比如说，Google Gemini，他们的前沿模型确实专门提供了一个 API 服务来做这件事。这让你能够基于强大的模型，针对自己的使用场景构建数据集，然后再把模型大幅缩小，比如缩到 7B 规模。或者，开源模型也完全可以用来运行和蒸馏，所以你也可以分别运行最优秀的开源模型。收集好数据后，你就可以对数据进行微调，然后分别微调不同尺寸的模型，看看哪种最合适。再比如 80/20 的例子，他们有 27B 和 7B 两种模型，分别用于呼叫中心的不同层级。像日常的、日常的来电，可能会由……等等。我想我不能说具体哪种尺寸对应哪一类，不过日常来电可以交给小得多的模型处理，然后再根据某些分类器触发条件升级到更大的模型，例如就是这样。So, for example, Google Gemini uh their their frontier models, they literally have a dedicated API service for this. Uh so so so this allows you to build data sets respectively on your use case on a strong model, which you can then which which you can then reduce dramatically to let's say a 7B scale. Uh alternatively, open-source models are more than happy for you to to to run and distill on, so you can run the best uh open-source models respectively as well. Once you collect them, you can then fine-tune the data and then you can fine-tune and respectively like different sizes to see what fits best. So, once again, like in the 80/20 example, they have the 27B and 7B for different levels of the call center. Like your routine your routine calls may come in at a Wait. I don't think I can say which size goes where, but your routine calls can come in for the much smaller model, and then it escalates up due to some classifier triggers as an example.

然后，然后，所以，所以这就是如何让流量流动、拆分流量，然后让它达到那个高效水平。呃，我们在做这件事时也经常与团队合作的一点是，即使他们经历了这些阶段，并将所有流量都切换到微调模型上，因为最优秀的模型仍在不断进步，呃，他们会得到……他们，呃，我们通常建议他们切换一部分他们的……Then then so so so that's how you get the traffic to flow, get the traffic split, and then you get it at that efficient level. Uh one of the things that we also tend to work with teams on on when doing this is that even even when they go through the stages and they completely split over their traffic to the fine-tuned model, because the best of the best models still progress, uh they get they uh we typically advise them to cut over a cut a part of their

用于持续改进的影子流量Shadow traffic to keep improving

将流量作为影子流量发送给最好的模型，以持续构建数据集。而且，这能让你跟上最好的开源模型的发展浪潮和进步。嗯，单位经济效益在这里其实非常重要，因为我想提醒大家的一点是，人们严重低估了 AI 的发展速度，尤其是那些可以以低得多的成本大规模运行的模型。比如说，就拿这台笔记本电脑来说。我喜欢衡量一下，在这里最贵的 MacBook Pro 上，我能买到的最好的模型是什么。如今最好的模型，无论是 31B、32B 还是 27B 规模的 MoE 模型，都已经远胜 GPT-4。也就是说，能在笔记本电脑上运行的这个模型，已经比你去年拥有的最好的模型更强了。大致如此。traffic as a shadow traffic to the best model to keep building the data set. And and and this allows you to ride the wave and progress of the best open models respectively. Uh and the unit economics is actually very important here, because like one thing that I would like to remind people, right, is people severely underestimate the the the speed that AI is going, especially for models that you can run much cheaper at scale. Like you take this laptop as an example. I like to measure, right, what is the best model that I can buy in the most expensive MacBook Pro here. And today's best models, be it MoE models at 31B, 32B, 27B size, is already much better than GPT-4. So, essentially, this model that can run on a laptop is already better than the best model that you have last year. Give and take.

人们喜欢争论具体细节，不过我们就笼统地说。针对你的任务对它进行微调后，这意味着如果你能在笔记本电脑上运行这个模型，那么就能在 GPU 服务器上大规模、更高效地运行它。是的。People like to argue on the specifics, but let's just say broadly. And when you fine-tune it to your task, that means if you can run that model on a laptop respectively, you can run it on on on your GPU servers at scale much more efficiently. Yeah.

步骤 3：购买 GPU 算力额度Move 3: Buying GPU capacity

所以，是的。这就引出了下一个话题：购买 GPU，而这显然正变得越来越困难。嗯，这正是我们这类公司发挥作用的地方。通过和许多客户交流，我们意识到，有些人相当惊讶地发现，规模扩展并不是线性的。呃，我的意思是，单个请求的每秒 token 数与请求本身的成本之间存在巨大的权衡。举个例子，如果你不在乎成本，可以在最好的硬件上运行最好的模型，比如一个固定的模型，使用全部八块 GPU，批量大小设为 1，这样每秒就能得到比如 100 或 200 个 token。如果你真的想要每秒数千个 token——这并不是我们的专长——那么像 Cerebras 这样的公司也很乐意提供这样的服务。So, Yeah. So, that brings in to the next topic, buying GPUs which which which is apparently increasingly a challenge. So, yeah, that's where companies like us come in. One of the things that we we realized through a lot of customer conversations, people are rather surprised that things don't scale linearly. Uh what I mean by that is like there is a huge trade-off between tokens per second on an individual request to to the cost to the cost of the request itself. So, for example, if you do not care about about the cost, you could run the best the model, let's say a fixed model, on the best hardware, on all eight GPUs, batch size one, and you could get, let's say, 100 tokens per second, 200 tokens per second. If you like if you really want and and and this is not our specialization, like you really want like thousands of tokens per second, that's where companies like Cerebras is more than happy to offer that as well.

而且成本还要高得多。所以，确实有些使用场景适合这么做。但很多人会陷入一个误区，觉得：“嘿，如果速度够快，成本可能就只有一半。”根本不是这么回事。比如说，按照一个经验法则……And it costs even more. So, there are definitely use cases where this makes sense. But, it's but a lot of people fall into the trap of thinking, "Hey, so if I have the speed, maybe it's half the cost." It's not even close. Like like a rule of

速度提升 2 倍 = 成本增加 3 倍2x the speed = 3x the cost

我有时会说这样一条经验法则：“速度每提升两倍，成本就会变成三倍，而且还会复合增长。”以这个 JM 5.3 的具体例子来说，你可以看到，同一台服务器上的一个 JM 节点，处理单个请求时的速度是每秒 112 个 token。但如果并行运行 16 个请求，每个请求的速度大约会减半，但总输出量实际上会提升 10 倍。这样一来，成本就会降低大约 9 到 10 倍。只需要调整你的目标速度配置即可。所以，很多客户问我们：“如何大规模部署？”有时我们首先会问：“你想要什么样的模型质量？希望达到每秒多少个 token？”然后我们就会开始建议需要多少 GPU，以及部署规模。thumb that I sometimes say like, "For every two x of speed, it's three x the cost and it compounds." In this specific example for JM 5.3, it's like one JM node is like you can see the same server, 112 tokens per second for running individual requests. But, if you run 16 requests in parallel, you get about half the speed for all the requests, but you literally 10x the output total output. You And and that that gives you like literally around 9 to 10x lower in cost. In just like changing your changing your target speed profile. And so, when a lot of customers ask us like, "How do we deploy things at scale?" Sometimes the first question is, "What's the model quality you want? And what's the tokens per second speed that you want?" And then we start recommending the amount GPUs, the amount deployment.

而这个目标每秒 token 数是一个重要的权衡。基本上，在更极端的情况下，对吧？如果你真的能让事情以每秒 10 个 token 的速度运行，几乎没人会这么做，但确实有一些奇特的使用场景允许这样做。相应地，你甚至可以获得更惊人的总吞吐量。通常，我们看到客户会在 20 到 30 之间选择，这大致相当于 ChatGPT 的速度，不需要太多思考；也可以一直选到 60，这差不多是最快的速度。如果他们想要每秒 120 个 token，通常也会落在这个范围内。大多数人会根据任务链有多长，在 20 到 30 或 60 之间选择。嗯，对。所以，嗯，我之前提到过，一般来说，只有当有人直接面对这项服务时，你才需要为每秒 token 数的速度付费——无论面对的是你的开发者，还是你的客户。And this this target tokens per second is an important trade-off. Essentially, um in more extreme cases, right? If you really, really can run things at 10 tokens per second, rarely anyone does, but there are some weird use cases that allows it. You can get even more incredible to- total throughput volumes, respectively. Typically, we see customers choosing between 20 30, which is kind of like uh your chat GPT speed without without too much thinking, all the way until like 60, which is like the fastest speed. And if they want 120 tokens like it tends to be this this range. Most stick between 20 30 to 60, depending on how how long the chain of task is. Yeah. So, yeah, uh I kind of mentioned you generally only want to pay for the tokens per second speed uh when when it's in front of someone. Uh whewhether it's in front of your developers or whether it's in front of your customers.

这些就是你想做出的权衡。而且即便在这些权衡之中，对吧？我们也见过这样的情况：哪怕只是稍微往下调一点，对吧？就能轻松节省两倍的成本，但对客户的影响几乎微乎其微。是的。再说一次，后台作业请不要以每秒 120 个 token 的速度运行。那样做毫无意义。是的。所以，呃，正如我之前说的，一旦你开始进行测量，对吧？你就能看到自己的流量模式。所以，很多……Those are the trade-offs that you want to make. And even within then, right? We have seen cases where even just a slight tweak down, right? Can easily save two times the cost, but has near minimal impact for the cus- uh customer, respectively. Yeah. And once again, background jobs, please don't run it at 120 tokens per second. It makes no sense when you do that. Yeah. So, uh once you have Previously, I said measure, right? You're able to see your your your traffic pattern. So, a lot of

突发流量与专用容量Bursty traffic & dedicated capacity

流量模式往往具有突发性，同时也有一定的基础负载，因为总有人会在某个时候运行某些任务，呃，尤其是后台代理。你也可以据此规划部署。我知道我们做金融领域的很多业务可能无法选择这种方式，因为它们只能采用专用实例部署。不过，在某些允许混合部署的情况下，如果既使用专用实例，也会在流量突增时扩展到公有云，那么就有了一定的灵活性。因此，目前市场上最大的成本节省方式，是在了解基础工作负载的情况下部署专用实例。只要充分利用它，就能节省一部分成本。不过，具体情况会因流量模式而变得复杂一些，而这些因素大多都可以通过测试和验证来确认。traffic patterns tend to be bursty in some nature with some base load because someone's always running something at some time, uh especially background agents. And and this is where you can you can also plan your deployments, respectively. Uh I know it since we're are doing financial a lot of financial may not have this as an option, but because they are only dedicated instance deployment, but if for some cases where people allow mixed where they have dedicated instances with some burst to the public cloud, this is where you have some flexibility respectively. So, like the biggest cost savings you have right now in the market is deploying a dedicated instance if you know your base workload. So, as long as you fully use it, you kind of have a savings there. But, it's a bit more complicated depending on your traffic pattern, and a lot of these things can be tested and validated uh respectively.

这基本上就是任何推理服务公司都会和你一起做的事，并非我们独有。我们会问：“嘿，你了解自己的流量负载吗？能分享一些数据吗？”之后，我们会说：“好，我们建议采用这样的配置，你可以试着切换到这种方案。”然后我们通常就能分别为你估算出可以节省多少成本。在一些极端情况下，如果流量峰值很多，你可能会希望把专属容量限制在一定范围内，再让超出的峰值流量转到公有云。不同的使用场景需要不同的容量配置，但只要一开始测量一下你的 token 负载，就可以开始探索这些方案，节省的成本可能非常可观。再举个例子，我们服务过一位处理了 2 万亿个 token 的客户，他们使用的是 4.8 Opus 级模型，每百万个 token 的费用不到 0.03 美元。And this is like essentially what any inference company will work with you. It's not unique to us. We'll be like, "Hey, this is Do you know your traffic workload? Can you share some data?" And then subsequently, we'll be like, "Okay, we propose this much. You can try switching to this." And then we we generate we can generally show you your savings respectively. And in some extreme cases where you have a lot of spikes, you may want to have limit your dedicated capacity to a certain amount and let the spikes spill over to the public cloud respectively. The different use cases uh have different right sizing, but this is something that by measuring your token workload once again at the beginning, you can start exploring these options, and the savings can be huge. Like once again, like for the customer that we served two 2 trillion token, and that was like uh 4.8 Opus class model, they are paying less than 0.0 $3 per million token.

就像你在别处根本拿不到那个价格。对。所以，简单说明一下，峰值确实很重要。呃，很少有人会在极端情况下出现短暂峰值，比如每个月都以那种规模运行一次工作负载，除此之外就没别的了。所以，通常都会有一定的基础工作负载，关键只是基础工作负载有多大。对，对。所以，最后一项是检查。Like you can't get that price anywhere. Yeah. So, yeah. Simple disclaimer is the spikes really matter. Uh it's very rare that someone has a short spike at the extreme end where they like run a workload every month at that scale, and then nothing else matter. So, so usually some base workload exists. It's just how big of a base workload. Yeah. Yeah. So, yeah. Then the last one is check

第 4 步：检查你的进程Move 4: Check your processes

你们的流程。呃，从这里开始，这就和推理优化毫无关系了。嗯，有些幻灯片听起来可能对 AI 非常批判，但我总体上是个乐观主义者。呃，我只是想先说明这一点，因为有些话可能会被断章取义。所以，我们开始看到的一个潘多拉魔盒式的问题，或者说棘手的问题是，很多组织开始问：“嘿，我的 token 开销增加了10倍、100倍，但我却没有看到同等规模的成果。”嗯，不过——比如说，我们开始看到各种组织的报告，比如 AI 只能带来有限的生产力提升，只有6%或7%，这并不是我们很多人被承诺或所设想的那种规模。遗憾的是，情况确实如此，因为我们正处在这样一个阶段：每个人都在拿着一把大锤，见什么就砸。your processes. Uh this has nothing to do with inference optimization from here onwards. Um Some slides may sound that very critical of AI, but I'm generally an optimist. Uh just this I just want to lay that out because things can be quoted out of context. So, one of the uh Pandora's Box question or problematic question that we are starting to see is that a lot of organizations starting to ask, "Hey, I'm s10x 100x in my token spend, but I'm not seeing results at that same scale." Um respect- like like we are starting to see reports from from various organizations like uh AI where AI has limited productivity only has 6% productivity or 7%, which is not the scale that a lot of us were promised or we envision. And and unfortunately, this is the case because we are at the stage where everyone's just taking a giant hammer and hammering everything.

嗯，并不是说 AI 没用，只是我们往往会到处滥用它。是的，有时候我就想说：“能不能别什么都搞成另一个 Copilot？”因为这类东西实在太多了。你需要想想自己该如何恰当地使用它。嗯，对。我们开始意识到的另一个主要问题是，随着 AI 以更大规模、更频繁地部署，人们创造东西的速度越来越快，但组织流程未必能跟上。编码环境中的第一个瓶颈会是 CI/CD，你们可能也听说过，CPU 会成为瓶颈。嗯，另一个问题就是现有的流程。我们该如何信任？该如何审查？我想上一位演讲者已经详细谈过信任问题。Um it's not that AI is not useful, it's just that we tend to just spam it everywhere. And yeah, if uh there's a sthere are times where I just want to like, "Can you please not do this as another co-pilot?" Because it's just too much of it. You need to think of how you use it respectively. Uh um yeah. And and another major issue that we start to realize is that well well AI's are are being deployed at a larger scale and more frequently, and people get faster in creating stuff, the organization processes may not go as fast. The first bottleneck for for coding coding environments will be the CICDs, and you all may heard that CPU becomes a bottleneck respectively. Uh another another thing is like is just the processes in place uh respectively. How do you trust? How do you review? I think the previous speaker spoke at length at trust.

是的，缺乏信任会拖慢进度，而且这可能确实有其道理。也许这个智能体——这个 AI 智能体——设计得很糟糕，不值得信任。但情况也可能恰恰相反。假设这个智能体做的一切都完美无缺呢？可人们还是不信任它，因为没有建立相应的测试和验证机制，也没有赢得信任。这会拖慢进度，而且这些因素的影响要大得多。所以，再说一次，务必谨慎看待锤子的使用方式，因为你需要据此做出相应调整。Yeah, the lack of trust is what slows things down. And this could be rightfully so. Maybe the agent the the AI agent was badly designed and shouldn't be trusted. But it also flip and inverse. What if hypothetically the that agent was doing everything perfectly? But people weren't trusting it because it didn't have the test in place, it didn't have the validation, and it didn't build that trust. That slows things down. And these things make and these things have a much bigger impact. So, once again, be very wary about how the hammer is being used because you need to make your adjustments accordingly.

Token 数量不等于已完成的工作量Tokens are not work done

所以，这又回到了刚才说的：token 不等于完成的工作。我想重点强调的一个主要例子是，这种情况发生得比我希望的更频繁：有时候，增加更多 agent、消耗更多 token，意味着同一项任务可能会被完成多次。 我们开始看到的一种反模式是，整个工程组织确实通过更快地解决更多问题、自动处理问题来提高了生产力。比如，QA 团队报告了一个问题后，agent 无需工程师介入就能自动修复，这些做法都很棒。但我们开始遇到的一个常见问题是，有些人很快就变得过于雄心勃勃。他们不再做小改动，而是开始做大改动，比如修改半个代码库，或者四分之一的代码库。So, and this goes back into saying like tokens are not work done. You have uh um um and one major example that I will highlight, right, is is this happens more often than than I would like, is that sometimes spamming more agents and more tokens sometimes means you get the same task done multiple times. So, one of the anti-patterns that we started seeing, right, uh we are well in overall the engineering organization improve productivity by clearing more issues quickly, automating clearing of issues, for example, like when someone like when a QA and and things like QA teams reporting an issue and the agent automatically fixing it without even an engineer, those are great. But one of the one of the common issues that we start facing is that people get ambitious really fast. And they instead of doing small changes, they start doing huge changes like changing like half the code bases or one quarter of the code bases.

有时候，如果在规划时进行了适当的测试，效果会很棒。但很多时候，变化的规模实在太大了，瓶颈就变成了审核者合并代码。由于所有事情的规模都大到人类无法处理，我们开始看到的一种常见反模式是：有人已经实现了……有些问题正在被处理。由于生成的代码太多，没有生成代码的问题就不会得到审核者的审查，也不会被发现。所以，需要有人用智能体生成代码。然后，当智能体生成那一大堆代码时——如果他们能好好审查提示词之类的内容，代码量本来应该少得多——审核者一看就会说：“呃，代码太多了。”Sometimes when this is planned with proper testing, it's great. But a lot of times, right, the change is happening at such a large scale, right, the bottleneck becomes the on the reviewer to merge the code. And because because everything is done on such a large scale where a human cannot process it, one of the common anti-patterns that we start seeing is that okay, someone has implemented the the there's issues being done. And because there's so much code being generated, issues that do not have code generated do not get reviewed and do not get picked up by the reviewer. So, someone needs to generate the code using an agent. And then when the agent generate that giant pile of code, which should be a lot less if they were to like properly review prompting and stuff, the reviewer look at it and it's like, "Huh, this is too many code.

我能不能让我的智能体标出哪些改动是相关的，然后再重新实现一遍？”然后，然后，最后他们开始和智能体一起手动重建。所以，他们实际上把第一个智能体做的工作全丢掉了。这种情况有很多合理的原因。比如，审查者想要掌控主代码库的质量，但这是一个非常常见的瓶颈例子：如果你没有把流程想清楚，就只是在不断增加更多工作，而速度却没有提升。所以，审视一下你的流程。一个很好的解决办法自然是建立更好的测试实践，这样你就能将测试自动化并筛选出问题。另外，也要提醒团队：“嘿，把你的合并请求拆分开，或者让它更模块化，这样审查起来会更容易。Can I ask my agent to highlight what changes are relevant and then re-implement it?" And and and in the end they started rebuilding it by hand with the with the agent. So, they literally discard the first agent's job. There's multiple good reasons why this happened. For example, like reviewers want to maintain control and quality of the main code base, but this is a very common example of a bottleneck where you where where if you didn't think the process through, you're just throwing more and more things. And respectively, and the speed is not increasing. So, review your processes. A good way around this is naturally to to to instill better testing practices and so that you can automate that and filter out. Also, reminding teams to like, "Hey, split up your your your merge or or make it more modular to make it easier for the review process."

类似这样的事情。这些都是人为流程，跟 AI 毫无关系。是啊，我们有时确实会对 AI 过于兴奋。所以，这不只是一个代码问题。呃，我们开始看到这样的情况：由于 AI 智能体正在被部署来制作演示文稿，这份演示文稿在设计方面完全是由 AI 生成的。我经历了多轮迭代。这不是一次生成就完成的。我至少把每一页都过了 30 遍，修改文字，但设计是由智能体完成的。而且，有时你会开始注意到组织里有些人一口气就生成了整套幻灯片之类的，然后他们就开始到处传，开始开会。接下来就得由其他人来筛查其中的不一致之处、偏差之处，以及为什么这两个数据点完全不同。Like things like that. These are human processes. It has nothing to do with the AI. And yeah, we tend to get too excited about AI sometimes. So, this is not just a code problem. Uh we are starting to see like because uh AI agents are being deployed to make decks, this deck was entirely design-wise was generated by AI. I went through multiple iterations. It's not a zero-shot. I I went through every page like at least 30 times to to change the text, but the design was done by agents. And and sometimes within the organization you're starting to notice people that just one shot from the entire slides, things like that. And and they start passing it around. They start doing meetings. And then it's on everyone else to filter what's the inconsistency, what's uh what's the misalignment? Why is this two data points completely different?

这也是另一个例子，和前一个例子差不多：人们生成代码、幻灯片或报告，之后核实这些内容的负担就落到了某个人身上。所以，要充分意识到这种组织变革。嗯，我为什么仍然乐观？因为我不认为这件事本身一定是坏事，问题在于每个人都需要学习。我们要做的就是据此开始调整流程。嗯，另一件重要的事是，虽然关于如何使用 AI 有很多常见的例子，但你们比我更了解自己组织所在的领域。你们会发现一些可以用 AI 来自动化流程的应用场景，例如，这可能为组织节省数百万美元，或者大幅提升生产力；这些只有你们自己才会知道，而不是我这样的 AI 专家能替你们发现的。This is also another example of like same as the previous example, people generating code or people generating slides or people generating report and the burden shifts onto a some human down the road to verify things. So, be very aware about this organization change. Uh why am I still an optimist is that I I don't think this is necessarily a bad thing in itself because the problem is everyone needs to learn. And this is where we just start adapting our processes accordingly. Uh and the other major thing is that well, there are a lot of common examples on how to use AI. You'll know your subject domain for your organization better. And there will be use cases where an AI could be used in place to automate a process, for example, that could save your organization uh millions of dollars or increase your productivity at a grand scale that that only you will know and not me as an AI expert will come in.

我喜欢举一个最著名的例子，沃尔玛的那个——是沃尔玛吗？我想应该是沃尔玛。他们真的开始在每一辆卡车里都放一个凳子，一个踏脚凳。这是最简单不过的改变，却让他们省下了数百万美元，因为这减少了搬运重物时的背部受伤事故。AI 的应用也可以有一些类似这样具体的做法。它甚至不一定要用最好的 AI 模型，却可能产生最大的影响。对，对。所以，接着我之前的观点说，其中一个被创造出来的术语……I like to point to the most uh famous example that Walmart had where was it Walmart? I believe it's Walmart. Where they literally started putting a stool a stepping stool inside every single truck. And that was like the simplest change, but it lowered their bill by the millions because it lowered back injuries across for heavy goods respectively. It it's some something specific like that with the use of AI. And and it may not even necessarily be a the best AI model, but actually can have the biggest impact. Yeah. Yeah. So, uh doubling down on my previous point, uh one of the term that that was coined

别扔泔水手榴弹。Don't throw slop grenades

最近流行的一种做法叫“垃圾内容手雷”（slop grenade）。也就是用 AI 生成一大堆东西，然后把它们丢给别人处理。别让这种“垃圾内容手雷”发生。这既是文化问题，也是流程问题。没错。我会制定一些指导原则，不过这些更多是主观判断，大家可能会有不同意见。 为了防止“垃圾内容手雷”发生，要确保每个人都对自己交付或生成的内容负责。比如这张幻灯片，所有生成的文字都由我负责。如果出了问题，责任在我，不是我的 AI 智能体出了问题。我不能把责任推给智能体。代码也一样，任何生成的内容都一样。此外，你还得衡量结果，而不是使用量。这部分会复杂一些，我不想在这里讲太久，但有一个简单的公式，我觉得大家可以把它当作经验法则：token 数量乘以价格，再除以实际有人使用的成果数量。by uh that that's been trending is called the slot grenade. So, generating lots of things in AI and then leaving it to someone else. Do not let slop grenades happen. Uh and this is a culture thing and a processing. Yep. So, the some guideline rules that I would set and this is more subjective. People people disagree. Um to prevent a slop grenade from happening, enforce that people should own what they hand over or generate. So, for example, this slide, I own every text that is generated. If there's an issue, that's on me. It's not my AI agent who who who has an issue. I do not get to hand it off to an agent. I must own the responsibility. Same thing for code, same thing for anything that is generated. And you also have to measure outcomes, not usage. Uh And so this gets more complicated because I I don't want to explain too too long to this, but like there there is like this simple formula that's that I think people take as a rule of thumb, tokens times price divided by outcomes someone actually use.

而且，而且，这有助于指导很多事情。比如更小、更高效的模型，更快的模型，更大的模型，等等。不过，我不喜欢的一点是，我还会用另一个比喻来说明 AI：可以把它看成是增强团队能力的补剂或类固醇。比如说，假设我用建筑施工来举例，它可以帮助工人增加肌肉量。比如说——这么做在伦理上并不妥当，我知道实际情况并非如此，但我们姑且把它当作思路中的一个例子。有些团队会用它，生产力也因此提高；有些团队则不会。但即使没有它，也很明显，有些团队比其他团队强得多。所以，AI 在这里就像一种补剂，对吧？它不一定会改变一切。因此，你需要意识到，衡量成果并不容易；不过，你可以开始衡量和观察，最重要的是，回到第一步。And and and this help guide a lot of things. Smaller, more efficient models, faster model, bigger model, yada yada. But, what I don't like about this is that another analogy that I'll use about AI is think of it more as like supplements or steroids that enhance the team. Think of it as uh like if let's say I use construction work as an example. It let's say I can use it to increase the muscle mass for the workers. For example, not that it's ethically to do so. I know that's not the case, but let's just say as an example in the train of thought. Some teams will have it and they improve productivity, some teams don't. But, even without that, it's quite clear some teams are much better than others. And and so the AI being a supplement here, right, doesn't necessarily change everything. So, you need to be aware that outcomes outcomes are hard to measure respectively, but it's something that you can start measuring and observing most importantly back to the first step.

衡量。你可以了解每个部门、每个团队消耗了多少 token，并开始看出哪个团队表现更好，以及他们用了多少 token。你也能开始看出哪些做法有效。是的。Measure. You know the tokens spent per department, per team, and you can start seeing which team is performing better and you start seeing how much tokens they use. You can start seeing what works. Yeah.

问答：微调与数据集Q&A: Fine-tuning & data sets

是的，正如我所说，这种衡量方式有缺陷。所以，嗯，这大致就是整体概述。要是你想在组织内部启动这项工作，可以来找我或我的团队聊聊；也可以先问自己这六个问题：哪个团队花了多少？要使用更便宜的模型，你必须测试什么？你的基础负载是多少？每个团队的默认模型是什么？哪些工作负载需要更快的响应，因为有人在等？还有，我们生成的内容有多少是有用的？这些问题可以作为指引，帮助你开始提问并优化你的 tokenomics。好，下面是问答环节。是的。Yeah, as I said, this is flawed measurement. So, yeah. Uh that's that's the high-level overview respectively. And if if you want to to like kickstart your process internally within your organization, you can you can either talk to me or my team or like you have six questions for you to start asking like who team spent what, what's the what what must you test to use a cheaper model, what's your base load, what's the default model for each team, and which workload has a person waiting that you need something faster, yeah. And how much of what we generate is useful. These are like guideline questions for you to start asking to refine your tokenomics. Yeah. So, Q&A. Yes.

嘉宾：是的，所以你提到微调模型时，是不是……Guest: Yeah, so when you mentioned fine-tuning models, is that

主持人：测试，测试。Host: Test, test.

嘉宾：呃Guest: Um

主持人：是的，我正在努力。Host: Yeah, I'm trying.

主持人：比如 SFT 这类微调，还是说你们会根据具体的使用场景或数据，采用其他类型的微调方法？Host: like SFT sort of fine-tuning or is there like other sort of uh fine-tuning methods that you have depending on what specific use case there is or what data there is?

嘉宾：嗯，好吧，总的来说，通常是 SFT，不过我不认为……嗯，其实我们确实提供微调服务，而且通常采用 SFT。目前有很多种方法。我知道 RL 现在很受追捧，尤其是在编程环境中，因为很容易在循环中生成数据。但我们发现，对很多组织来说，尤其是像纽约的金融机构这类组织，并没有真正适用于金融任务的 RL 环境。所以通常更多是以 SFT 为基础。还有一些其他组织，比如 Adapted Adoption AI，也开始自动化数据集的优化，让你可以用 SFT 基础模型或思考型机器模型来训练模型。把重点放在这上面，可能问错了问题。Guest: Uh okay, so in general, it tends to be SFT, but I don't think um Well, we do provide fine-tuning services and it tends to be SFT. I there is a lot of methods that are happening right now currently. I know RL is very sexy right now, especially in the coding environment, because it's easy to generate the data in the loop, but what we realize for a lot of organizations, especially like in New York, for example, financial institute, things like that, there's not really an RL environment for financial task, for example. And and so it tends to be more SFT base. And and there are also other organizations like adapted adoption AI where where they are starting to also automate the the the refinement of the data sets for you to train the model be SFT base or thinking thinking of machine base. Focusing on this might be the wrong question.

对我来说，总的来说，人们应该分别关注评测。举个很好的例子：Jeff 模型发布后，我们做了一个名为 Simple Jeff 的开源项目，用现有的开源模型复刻 Jeff API。很多人很快就发现，Quen 27B 在这项任务上表现非常出色。于是有好几个团队开始对 Quen 27B 进行微调。而我们这边，因为公司想采取的方式是接受任何模型，然后在其上构建 API，所以我们开始尝试另一种方法：我们看到了基准测试，也看到了现有模型，就想，为什么不改用提示词调优呢？这甚至都不算微调。两周后，Hugging Face 开始大规模开展各种基准测试，测试题超过 10,000 道，诸如此类。It to me generally what people should focus on is the evals respectively. So here's a good example for Jeff that came when the Jeff model came out and then we did the open source project called simple Jeff which replicates the Jeff API for existing open source model. A lot of people quickly realized that the Quen 27B was really good at this task. And and there were multiple teams that went to fine tune the Quen 27B. I on on our side because we are trying to take the approach like our company like we want to accept any model and then we just build the API on top of it. We started taking the approach of like hey we see the benchmark we see the existing model why don't we prompt tune it instead. It's not even fine tune. And then and then two weeks later as hugging face went to do all the benchmarks at scale like over 10,000 questions things like that.

不知怎么地，我的提示词调优版 Quen 27B 击败了大多数 27B 微调模型。所以这并不是在炫耀，因为我现在相当确定，我的模型已经被 Cloudflare 最近那个基于更规范方法训练的微调模型击败了。不过这更像是说，调优方法可能不是应该关注的问题；更重要的是，你们用了什么评测，以及得出了什么结果。Somehow my prompt tuned Quen 27B beat most of the 27B fine tune. So this this is not a flex because I'm quite sure now my model is beat by the recent Cloudflare fine fine tune model that was done on a more proper basis but it's more of like the tuning method might the tuning method might not be the right question it's more like what's your evals and what's the results that lead to it.

主持人：那么，要做到这一点，你需要……你需要为所需的数据集收集黄金轨迹吗？如果你在做 SFT 之类的事情？还是说可以——就拿你之前的任务来说，前沿模型的准确率大概有 80%，对吧？所以它可能有 20% 的时候会犯某种错误。如果你用这些数据进行微调，微调后的模型不也会有大约 20% 的错误率吗？是……Host: And follow for that do you need do you need to collect golden traces for the set of data you need if you're doing SFT or whatever it is or can you do it let's just say like your your prior task the frontier models get like 80% accurate right? So maybe 20% of the time it makes some sort of mistake. If you fine-tune on that, won't the fine-tuned model also get like 20% of it wrong? Are

问答：黄金轨迹Q&A: Golden traces

你是不是应该想办法把那些过滤掉，只留下用于训练的高质量轨迹？还是可以把所有轨迹都丢进去训练？you supposed to filter that out somehow and get only the golden traces for training, or can you just throw all the traces in and train?

嘉宾：所以，这更像是一种个人观点下的技巧，因为，呃，一般来说，我会尝试拆分数据集，因为大家都懒得筛选。嗯，所以我至少会尽量整理出一个黄金轨迹数据集。黄金轨迹数据集可能会相对小得多，然后我们会对它进行微调，并赋予它更高的权重。不过，已经有多篇论文，尤其是关于思考 token 的论文，指出即使是失败的任务，对更小的模型进行微调也往往能从训练中受益，因为它改善了思考过程，即使模型得出了错误的答案。嗯，这确实是一个很好的例子，说明应该先尝试再衡量，而且这种方法有 70% 的时候都有效。有时候，你只需要提高黄金轨迹的占比，但这只是一个一般性的经验法则。Guest: So, this is more of a personal opinion technique because uh so, the generally, I try to split the data set because people are lazy filter. Um so, I try to at least get a golden trace data set. And the golden trace data set maybe is much smaller uh respectively, and we just and we then we fine-tune that with a much heavier weightage uh as well. But, there have been multiple papers especially as especially with thinking tokens that even the nonuh the failed task, right? The as much smaller model, when you fine-tune, tends to benefit from the training because it improved the thinking process even if it got the wrong answer. Um this is something This is really a good case of like try and then measure, and it works 70% of the time. Sometimes, you just you need to increase the golden trace ratio, but it's a general rule of thumb.

我认为，对很多客户来说，当他们尝试对一个更小的模型进行 SFT 时，通常会遵循这样的经验法则：能否让性能保持在几乎相同的水平？评估结果可能下降 1%，甚至在某些情况下提升 10%，因为基础模型原本并未针对这些任务进行专门训练。更重要的是，这能否让我的账单降低 100 倍？这通常分别就是他们的动机。是的，如果你想做到超越基线的效果，黄金样本至关重要。有时这需要人们亲自查看数据。虽然我很希望客户能多看看数据，但大家并不喜欢这么做。I think for a lot of customers when they're trying to to SFT down a smaller model, the general rule of thumb they're trying to keep is can I keep to the near same level of performance, and maybe the eval is down 1% or even in some cases it improved by 10% because the base model wasn't specialized for these tasks. And more importantly, does it 100x reduce my bill? So, that's usually the motivation respectively. Yeah, golden samples are critical if you want to do beyond the baseline. And sometimes that requires humans to actually look at the data, which as much as I love my customers to look at the data more, people don't like to do that.

主持人：你好。嗯，你会如何处理主要来自 Claude Code 及其所有集成的代码支出？因为这样一来，你的模型就完全……Host: Hi. Um how would you deal with uh code spend that would mostly come from Claude's code and all the integrations there? Cuz then your model is fully

问答：摆脱线束的束缚Q&A: Escaping harness lock-in

我想，算是被困在 Claude 生态系统里了。对于这类情况，你有什么建议吗？locked into the Claude uh ecosystem, I suppose. Do you have any advice on how to deal deal with those type of situations?

嘉宾：呃，是的，很遗憾，Claude Code 就是完全被厂商锁定的一个例子。Guest: Uh yeah, unfortunately Claude code is the example of fully locked in.

主持人：所以，有时候，第一步实际上是让他们能够切换 harness。现在看来，harness 灵活得多。比如，我们见过一些组织完全锁定在某个平台上，比如只用 Devin 或 Mboni。我们也见过一些组织更加灵活，甚至会自己搭建 agent harness。我的反应就是：“为什么？为什么要从零开始自己搭建 agent harness？”不过，我们看到人们会切换到的一个常见选择是 Pi。这也是因为大家对它很感兴趣。有人展示过，即使使用相同的模型，由于 Pi 简单高效，每个 token 的成本也会低很多。我认为，Pi 编码代理 harness 的妙处就在于它很容易调整，因为它非常轻量。Host: So, uh sometimes sometimes the first movement is actually getting um them able to switch the harness. Um harness it seems to be a lot more flexible thing right now. Like we have seen organizations that are like fully locked in. Like I'm only going to use Devin only or Mboni. We have seen organizations that um that are more flexible. Like literally they are running their they are creating their own agent harness. And I'm my response is like, "Why? Why are you creating your own agent harness from scratch?" Uh but like a common one that we see people switching to is for example Pi. And this is also because there's a lot of excitement around it. People have shown that even with the same model, it gets a much cheaper cost per token because of the way it's simple and efficient. Um and from there on, the I think the beauty for the Pi coding agent harness is that it's easy to adapt because it's so siso lightweight.

我们见过很多团队一开始只是提示他们的智能体进行修改。一旦他们这么做了，就开始进入不断尝试和调整的循环。呃，让他们特别有动力去调整的一种常见方法，是引入一个速度非常快、性能也足够好的模型，比如每秒能处理 100 个 token。然后他们突然就开始尝试起来了。有些人一开始的转变是，仍然用 Claude Code 作为规划器，然后指示 Claude Code：“用这个模型驱动的 Pi agent 来完成实现。”于是就形成了规划与实现分工。确实更像是这样，这就是转变开始发生的地方。改变人们使用的智能体并不容易，不过，是的。We've seen a lot of teams that just start prompting their agents to make modifications. And once once they have that, then then they start getting the cycle of experimenting and changing. Uh a common way to get them to be very incentivized to change is to actually introduce a a a good enough model that is really fast, like 100 tokens per second. And then suddenly they start giving that a try. And some people start or like some of the first shift would be they still use their Claude code as their planner. And then they instruct the Claude code, "Use this Pi agent with this model to do the implementation." Hence the planner implementation split. It's more yeah, and that's where the transition started to happen. It's not easy to change people's agents, but yeah.

嘉宾：嗯，是的，比如说，可以把这个较小的智能体作为 MCP 暴露在云端，或者提供一些可以把任务交出去的工具。我想问，这是不是你觉得可行的做法？Guest: Um Yeah, so an example of these could be perhaps to expose this this smaller agent as an MCP in inside cloud or having these tools where you can offload I I suppose is that something that you would you you would say could work?

主持人：MCP 要棘手得多，因为除非配置得当，否则 MCP 代理无法访问你的文件系统。所以通常只是简单得多的操作，比如安装 pip，然后运行 bash 命令 pip install。对，通常就是这样。Host: MCP is more tricky because MCP agents unless done right don't have access to your file system. So typically it's just something way simpler like pies installed and they run the batch command pie this. Yeah, that that's usually the case.

来宾：谢谢。Guest: Thanks.

主持人：是的。Host: Yeah.

嘉宾：如果可以的话，能分享一个案例吗？Guest: Can you share if possible any case

问答：缓存与自动路由Q&A: Caching & auto-routing

有没有一些关于缓存解决方案的研究，深入分析缓存本身以及为特定客户带来的成效（即使客户信息经过匿名化处理）？studies about caching solutions being very thoughtful about caching and the outcomes that have been generated for specific customers even if it's anonymized?

主持人：你说的具体挑战是什么？你做了一个缓存，你……好吧，你的问题更像是想要匿名化……你……等等，你是不想把客户的私人信息保存到缓存里。Host: What's the specific challenge you're talking about? You do one caching you Okay, is your question more like you want anonymized you do Wait, you do not want customer private information to save in the cache.

嘉宾：不是，我是想问，你合作过的客户中，有没有可以分享的案例，说明他们通过审慎考虑缓存方式和采用不同的缓存方法，取得了哪些成效？Guest: No, so so like with customers that you've worked with have you got any case studies you can share about the outcomes that they've been able to generate because of being thoughtful about how they cache and different sort of caching methodologies?

主持人：嗯，我觉得这个问题我们可以线下再聊。不过一般来说，对于规模较大的客户，我们会为他们部署专用缓存。这样基本上能解决很多 PI 相关的问题，因为这个缓存只供该客户使用。嗯，当然还有提示词格式方面的问题，不过我可能需要更具体地了解一下你在这里想问什么。简短回答是可以，但这个问题有很多种理解方式。嗯。Host: Uh I think this one we can take offline, but in general for our larger scale customers we deploy them with dedicated cache. So it kind of solves a lot of PI issues because it's the the cache is only for that customer. Um yeah, and obviously there's prompt formatting, but I may need to dig in more specifically like what you're trying to ask here. The short answer is yes, but there's a lot of way to interpret this. Yeah.

客人：好。刚才不好意思。Guest: Cool. Sorry about that.

主持人：呃，你怎么看基于轮次或任务的自动路由？Host: Uh thoughts on turn or task-based auto routing?

嘉宾：哦，嗯，我通常更倾向于基于任务的方式。嗯，主要是因为在部门或组织层面更容易控制预期。所以，这又回到了这样一种情况：比如说，某个部门可能有一个标签模型，我可以为他们自动路由到该模型。呃，我知道现在大家对基于轮次的自动路由有很多炒作。基于轮次的自动路由有一个问题：你需要把它限定在比如最多两到三个模型之内。否则，你的缓存使用效率会非常低。嗯，比如说，我们见过针对 GRM 5.3 和 GRM 5.3 Flash 的基于轮次的路由。这说得通。但如果你放进 30 个模型，而且每一轮都不断切换模型，那么较低的缓存命中率就会开始破坏你的优化效果。Guest: Oh, um I'm a I'm generally more towards task-based. Um mostly because it's much easier to control expectation at a department organization level. So, this this is back to like, hey, a department may have a label model and I kind of auto router that model for them. Uh I know there's a lot of hype right now for turn-based auto routing. One of the issues with turn-based auto routing is that you need to lock down your your turn-based auto routing to let's say at most two to three models. If not, your cache will be extremely inefficient to be used. So, um so for example, we have seen turn-based for GRM 5.3 and GRM 5.3 flash. Make sense. But if you put in 30 models and it keeps changing your model every turn, your your low cache hit rate will start to wreck your your optimization.

主持人：是啊，我以前也见过这种情况。我看到它像是请求级别的东西，完全说不通，然后……Host: Yeah, that's what I've seen before. I saw I saw it like request level and it just didn't make sense and then

嘉宾：是的。不过两个模型通常还行。嗯，任务和组织这种分法最合理。对。Guest: Yeah. But two models tends to be okay. Uh task and organization makes the most sense. Yeah.

主持人：是的，是的，我同意。Host: Yeah. Yeah, I agree.

嘉宾：好。好的。非常感谢。嗯，如果有人想专门找我问更多问题，欢迎到前面来问我，或者等下一位讲者到了之后在旁边找我。好的。非常感谢。Guest: All right. Okay. Thank you very much and uh if anyone wants to find me specifically for more questions, feel free to just come up front and ask me or find me by the side once the next speaker's here. Yeah. Thank you very much.