怎样真正节省 Token 成本：Alvin Pane 的智能体评估与优化方法

简要描述Brief Description

如何真正实现 Tokenmaxx 并节省数百万——OutRival 的 Alvin PaneHow to Actually Tokenmaxx and Save Millions — Alvin Pane， OutRival

目录Table of Contents

• 合成提示与冗余工作的代价
  A synthetic prompt and the cost of redundant work
• 观察 token、延迟和提示词版本
  Observe tokens, latency and prompt versions
• 候选项分歧与生产环境中的差异
  Candidate disagreement and production variance
• 前缀缓存与利用 DSPy 和 GEPA 优化指令
  Prefix caching and optimizing instructions with DSPy and GEPA
• 较小模型与基准测试过拟合
  Smaller models and benchmark overfitting
• 影子评估中的三项比较
  Three comparisons in a shadow evaluation
• 人工审核与成对发布标准
  Human review and paired release criteria

合成提示与冗余工作的代价A synthetic prompt and the cost of redundant work

大家好，我是 Alvin，OutReval 的智能体智能负责人。现在大家都听说过“刷 Token”了。公司里有人专门建了排行榜，比谁用的 AI Token 最多。在 Meta，他们甚至还给这事起了个头衔，叫“Token 传奇”，作为一种荣誉徽章，颁给消耗算力最多的人。这是个挺有意思的梗，但如果你真的在交付产品，那么 Token 用量其实是个很奇怪的成功衡量标准。Token 不是工作本身，而是你为完成工作所付出的成本。在 OutRrival，我们的数字员工通过电话、短信和电子邮件处理复杂任务。在应用层面，我们每天处理数十亿个 Token。而当你以这样的规模运行时，每一次多余的模型调用都会层层累积，造成巨大的延迟和数百万美元的算力成本。来看看这段提示词。Hey everyone, I'm Alvin, the head of agent intelligence at OutReval. By now, everyone's heard of token maxing. People have built leaderboards inside companies for who's using the most AI tokens. At Meta, they even had a title for this, tooken legend, a badge of honor for whoever's burning the most compute. It's a funny meme, but if you're actually shipping a product, token usage is a bizarre measurement of success. Tokens aren't the work. They're what you pay to get the work done. At OutRrival, our digital workers handle complex tasks over phone, text, and email. At the application layer, we're processing billions of tokens every single day. And when you operate at that scale, every redundant model call compounds into massive latency and millions of dollars in compute costs. Take a look at this prompt.

我构建了一个合成代理，用来模拟管道中的结构性臃肿是什么样子。假设我们正在为一家名为 HarborBite 的虚构公司对客户意图进行分类。这样的东西我们都见过：包含 20 个边缘案例、定义过分吹毛求疵，还有一整段话几乎是在恳求模型不要产生幻觉。每个请求都要发送这么多文本，而且一天发送数百万次，这就不再只是提示词了，而是一笔巨大的、自己强加给自己的算力税。如果你想让智能体具备更多能力，就需要在架构上留出足够的空间。今天，我们要重新定义 token 最大化。重点不该是消耗最多的 token，而是尽可能拓展……的覆盖范围。I've built a synthetic proxy that mirrors what structural bloat looks like in a pipeline. Let's pretend we're classifying customer intents for a fictional company called HarborBite. We've all seen one of these. It's got 20 edge cases, overly pedantic definitions, and a paragraph practically begging the model not to hallucinate. When you're sending this much text on every single request, millions of times a day, it isn't just a prompt anymore. It's a massive self-inflicted compute tax. If you want to give your agents more capabilities, you need the architectural headroom to do it. So today, we're reclaiming token maxing. It shouldn't be about burning the most tokens, but about extracting the maximum surface area of

观察令牌、延迟和提示词版本Observe tokens, latency and prompt versions

以绝对最少的 token 负载获取智能。所以，下面来看看如何真正实现 token 最大化。我们开始讲实战手册。首先，观察。无法修复无法衡量的东西。为共享模型客户端添加监测，让每次尝试都成为一条追踪记录，将单个任务中的模型调用、工具和重试串联起来。记录提示词版本、模型设置、token 用量、延迟和成本。OpenTelemetry 是一个非常适合追踪这些数据的框架。有了这些遥测数据，你就可以建立基准，了解生产环境中的实际表现。但大多数优化工作都会在这里碰壁。让我来告诉你通常会怎么做：你拿出一个庞大的提示词，把它交给一个更小、更便宜的模型，然后运行它intelligence from the absolute minimum token payload. So here's how to actually token max. Let's get into the playbook. First, observe. You can't fix what you can't measure. Instrument your shared model client so each attempt becomes part of a trace connecting the model calls, tools, and retries for a single task. Record the prompt version, model settings, token usage, latency, and cost. Open telemetry is a great framework for tracking this. Now that you have that telemetry, you can actually build a benchmark of how production is behaving. But here's where most optimization efforts hit a wall. Let me tell you how it usually goes. You take your massive prompt, you hand it to a smaller, cheaper model, and you run it

候选者分歧与生产方差Candidate disagreement and production variance

我们用几千条历史生产样本进行了测试。我们做了这件事。结果发现，我们成本更低的候选模型与昂贵的生产基线模型有约 18% 的时候意见不一致。所以我们理所当然地把它淘汰了。它还不够好。但这 18% 实际上代表什么？由于生产环境中的语言模型要处理本质上含糊不清的现实输入，这个 18% 实际上是一个综合指标。它引入了一个关键的隐藏变量：模型自身的基线方差。当输入杂乱时，你的生产提示词不可避免地会与模型过去的决策不一致。为了剥离这个隐藏变量，并弄清楚重写究竟改变了什么，我们提出了一个特别简单的问题：生产环境的模型自己和自己意见一致吗？我们进行了一次 AA 测试。我们称之为天花板测试。我们让庞大而昂贵的生产模型对相同输入分别运行三次，然后比较答案。against a few thousand of those historical production examples. We did this. Our cheaper candidate model disagreed with our expensive production baseline about 18% of the time. So naturally, we threw it out. It wasn't good enough. But what does that 18% actually represent? Because language models in production are dealing with inherently ambiguous real life inputs, that 18% is actually a composite metric. It introduces a critical hidden variable, the baseline variance of the model itself. When the input is messy, your production prompt will inevitably disagree with its own past decisions. So to isolate that hidden variable and understand what your rewrite actually changed, we asked a really dumb question. Does prod even agree with prod? We ran an AA test. We call this the ceiling test. We took our massive expensive production model and ran it three separate times on the same inputs and compared the answers.

我们发现，它大约有 7% 的时候会自相矛盾。生产环境中的系统自身一致率也只有约 93%。所以，我们要求廉价候选方案达到完美标准，而现有的昂贵系统甚至都达不到这个标准。一旦你确认这一点We discovered that it disagreed with itself roughly 7% of the time. Production's own agreement was only about 93%. So we were demanding a standard of perfection from the cheap candidate that our existing expensive system couldn't even hit. Once you establish that

使用 DSPy 和 GEPA 进行前缀缓存与指令优化Prefix caching and optimizing instructions with DSPy and GEPA

完成上限测试基准后，你终于有了一个经过测量的基线。现在可以开始降低这项工作的成本。我喜欢使用三个技术手段，而且它们有先后顺序。第一个是缓存。具体来说，调整提示词的结构以最大化前缀缓存率，是一种非常简单且有效的方式，可以降低运行特定任务的成本。第二个手段是缩减提示词本身的体量。我们再来看看 Harborbite 的提示词示例。这个提示词将近 5,000 个 token。为什么会这么长？因为开发者日积月累地往里添加了好几段规则。比如，“不要把热情误认为授权”就是其中一条。呃，还有大量硬编码的对话记录，充当演示示例。没人愿意当那个删掉一行代码、结果把生产环境弄坏的人。ceiling test benchmark, you finally have a measured baseline. Now you can start making the work cheaper. There are three technical levers I like to pull and they run in a sequence. The very first is caching. Specifically, structuring your prompts to maximize your prefix cache rate is a really easy and really effective way to bring down the cost to run given tasks. The second lever is to actually shrink the footprint of the prompt itself. So, let's bring that Harborbite prompt example back up. This prompt is almost 5,000 tokens long. And why? Because over time, developers kept bolting on paragraphs of rules. For example, do not confuse enthusiasm with authorization is one such example. Uh, plus massive hard-coded transcripts that were acting as worked examples. Nobody wants to be the one who deletes a line and breaks prod.

那么该怎么做？自动化。把提示词编写转变为一个搜索问题。借助 DSPI 这样的框架，你实际上可以不再手动编写指令。你需要定义精确的接口契约、输入、必需的输出模式以及评分函数，然后把这些交给像 Jeepa 这样的优化器。优化器会把你的提示词视为一个搜索空间。它会针对你的基准进行测试，尝试匹配那些历史标签，并利用失败反馈来修改指令。它删掉了我们几乎所有冗长的示例，换成了简洁的规则。在应用契约保持不变的情况下，指令 token 减少了 51%。现在我们可以开始测试质量是否能保持不变。这就是……So what do you do? You automate it. You turn prompting into a search problem. With a framework like DSPI, you can actually stop manually writing instructions. You define the exact interface contract, the inputs, the required output schema, and the scoring function. And then you hand it off to an optimizer like Jeepa. The optimizer treats your prompt as a search space. It runs against your benchmark attempting to match those historical labels. And it uses feedback on failures to revise the instructions. It stripped out almost all of our long worked examples and replaced them with compact rules. A 51% reduction in instruction tokens with the same application contract. Now we can move to test whether quality holds. That is

小型模型与基准测试过拟合Smaller models and benchmark overfitting

即时利润空间。第三个杠杆是真正缩小模型规模。现在你已经有了精炼的提示词和可衡量的搜索循环，就不必总是使用庞大的前沿模型。你可以大幅削减计算开销。这里有两种值得关注的策略。第一，你可以把一个庞大的提示词拆分成更小的组件。使用更便宜的模型处理初步路由或数据提取之类的任务，只在需要进行复杂推理时才调用昂贵的前沿模型。第二，你可以使用我们刚刚构建的同一个 DSPI 搜索循环，只不过接入一个更小的模型。让这个小模型在自己的提示词空间中搜索，使其输出与大型生产模型生成的基准答案相匹配。本质上，你是在用 Jeepa 教会一个小模型，让它针对你的特定数据集达到与大型模型相当的任务表现。instant margin. The third lever is to actually downsize the model. Now that you have a tight prompt and a measurable search loop, you don't always need a massive frontier model. You can start aggressively cutting compute. There are two interesting strategies here. First, you can break a massive prompt into smaller components. you use a cheaper model to handle something like initial routing or data extraction and only wake up your expensive frontier models for heavy reasoning. Second, you can use that exact same DSPI search loop we just built, but plug a smaller model into it. You let the smaller model search its own prompt space to match the benchmark answers generated by your large production model. You're essentially using Jeepa to teach a tiny model to match the task performance of a massive one for your specific data set.

但对于这些小模型来说，优化的实际情况并没有那么简单。嗯，如果你试图把大型模型的行为压缩到一个更小的模型中，就会发现这会引入新的失效点和过拟合。嗯，在我们许多缩小模型规模的实验中，呃，我们发现这些提示词还不够完善，无法发布，这就引出了最后一个难关。我们已经把 Harbor 的提示词缩短了一半，但现在还得证明我们能……But the reality of optimization is not quite so easy when it comes to these small models. Um if you're trying to compress the behavior of a large model into a smaller one, you'll notice that you introduce new points of failure and overfitting. Um, and in many of our downsizing experiments, uh, we found that the prompts weren't good enough to release, which brings us to the final hurdle. We've cut Harbor's prompt in half, but now we have to prove that we

影子评估中的三项比较Three comparisons in a shadow evaluation

实际上并没有破坏产品。记住，这些模型是概率性的。生产环境有时连自己都无法达成一致。同样，我们这个新的低成本候选方案也会有自己的内部差异。因此，你要用完全相同的输入进行大规模影子部署，并从三个角度衡量系统：生产环境与生产环境、候选方案与候选方案，以及候选方案与生产环境。这里有一段很有用的数学计算：先计算两个版本之间的分歧，再减去它们各自内部平均分歧。剩下的结果就能衡量它们决策概率有多大差异。这正是上限测试发挥作用的地方。我们不是在猜测。即使什么都没变，我们也有一个数学基准，可以衡量这些答案会有多大波动。actually didn't break the product. Remember, these models are probabilistic. Production doesn't always agree with itself. And likewise, our new cheaper candidate is also going to have its own internal variance. So you run a massive shadow deployment on the exact same inputs and measure the system from three angles. Production against production, candidate against candidate, and candidate against production. There's a useful bit of math here. You take the disagreement between the two versions and subtract the average disagreement inside each one. What's left gives you a measure of how different their decision probabilities are. This is where that ceiling test pays off. We aren't guessing. We have a mathematical baseline for how much these answers move around when we haven't changed a thing.

这就是这种偏移背后的秘密：不同并不意味着错。如果你的优化器完成了任务，那么这些分布理应有所不同，因为成本更低的提示词实际上表现得更好。我们不一定要打造一个与生产系统完全一致的克隆，而那个系统本身可能已经带有一些固有缺陷。And here's the secret about that drift. Different doesn't automatically mean wrong. If your optimizer did his job, the distributions should be different because the cheaper prompt is actually doing a better job. We aren't necessarily trying to build a perfect clone of a production system that may already have some flaws baked into it.

人工审核与配对发布标准Human review and paired release criteria

所以，你不再为原始差异惊慌，提取差异内容，并手动审查其中一部分。候选版本更好吗？生产版本更好吗？还是说，对于一个含糊的输入，它们只是两种不同但同样可接受的处理方式？以 Harbor Bite 为例，假设审查完成后确认了 443 个回退，但也发现了 2,463 个明显改进。也就是说，提示词长度只有一半，却几乎每出现一次回退，就有六次改进。但你不能只凭点估计就发布。你必须考虑抽样误差。我们在这批特定请求上测得了巨大的提升，但还需要准确了解这一结果有多大的统计置信度。因此，我们围绕净提升计算置信区间。你要严格保持配对比较，让两个版本都回答完全相同的请求，并将同一组对话的重复运行归在一起。So, you stop panicking about the raw differences, you pull the diffs, and you manually review some of them. Was the candidate better? Was production better? or were they just two different equally acceptable ways to handle an ambiguous input? For Harbor Bite, let's say the completed review confirmed 443 regressions, but it also found 2,463 outright improvements. That is almost six improvements for every one regression from a prompt that is half the size. But you do not ship on a point estimate. You have to account for sampling noise. We measured a massive gain on this specific batch of requests, but we need to know exactly how much statistical confidence we can put behind it. So, we calculate a confidence interval around that net gain. You keep the comparison strictly paired, both versions answering the exact same request, and you keep the repeated runs of the same conversations grouped together.

要说候选方案更好，我们希望置信区间的下限高于零。如果区间跨过零，就需要更多证据才能做出这个判断。最后还要做一项合理性检查：必须分别对关键类别进行这项计算。整体平均表现再好，也可能悄悄掩盖某个重要客户意图的表现正在急剧下滑。如果关键类别的表现都站得住脚，而且置信区间下限高于零，答案就很明确了。发布吧，并持续衡量效果。 这套流程能让你用明确的发布规则，取代提示工程中的猜测。把显而易见的内容缓存起来。让优化器精简你的指令。选对模型，并用上限测试来了解差异。然后测试质量能否保持稳定。 那么，这是你周一早上的作业。To say the candidate is better, we want the lower confidence bound above zero. If the interval crosses zero, we need more evidence before making that claim. And one final sanity check is that you have to run this math on your critical categories separately. A great global average can actually uh silently hide that one vital customer intent is tanking. If your critical categories hold up and your lower confidence bound is above zero, you have your answer. Ship it and keep measuring. So this process is how you can replace the guesswork of prompt engineering with an explicit release rule. Cache the obvious stuff. Let an optimizer shrink your instructions. Rightize your model and use the ceiling test to understand the variation. Then test whether quality holds. So here's your homework for Monday morning.

去挑一个你最昂贵的任务。测量它的基线方差。把它放进搜索循环，准确找出你为冗余付出了多少代价。我们得重新把 token 利用率拉满。让行业里的其他人去争排行榜高分吧。真正的 token 传奇，是那些让每一个 token 都物尽其用的人。Go pick your most expensive task. Measure your baseline variance. put it through a search loop and find out exactly how much bloat you're paying for. We need to reclaim token maxing. Let the rest of the industry fight for the high score on the leaderboard. The real token legends are the ones making every single token count.