从零构建智能体评测排行榜：Wolfram Ravenwolf 的实战方法

简要描述Brief Description

From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf， Weights & Biases

目录Table of Contents

• 研讨会设置
  Workshop setup
• 认识沃尔夫拉姆·雷文沃尔夫
  Meet Wolfram Ravenwolf
• 评估思维与方法论
  Evaluation mindset and methodology
• 为什么一个平均值会掩盖可靠性
  Why one average hides reliability
• WolfBench 的五项指标
  The five WolfBench metrics
• 在统一条件下重复运行
  Repeat runs under uniform conditions
• 评估配置，而不仅仅是模型
  Evaluate a configuration, not just a model
• 评估流水线的六个层级
  Six layers of an evaluation pipeline
• 选择并固定基准测试
  Choose and pin the benchmark
• Harbor 和评估运行器版本
  Harbor and evaluation runner versions
• 隔离沙箱
  Isolated sandboxes
• 比较智能体运行框架
  Compare agent harnesses
• 会改变结果的设置
  Settings that change the result
• 提示、工具和资源限制
  Prompts, tools and resource limits
• 超时、并发与重试
  Timeouts, concurrency and retries
• 修复推理提供方路由
  Fix inference provider routing
• 启动 Git 恢复任务
  Launch the git recovery task
• 保留分数之外的证据
  Preserve evidence beyond the score
• WolfBench 数据集收集工具包
  The WolfBench collection toolkit
• 在 Marimo 仪表板中检查运行记录
  Inspect runs in the Marimo dashboard
• 筛选不完整和无效的运行记录
  Filter incomplete and invalid runs
• 导出结果并生成图表
  Export results and generate charts
• 在一项任务中获得满分
  A perfect score on one task
• 阅读互动排行榜
  Read the interactive leaderboard
• 一致性、平均值和上限
  Consistency, average and ceiling
• 令牌用量和总成本
  Token usage and total cost
• 当更多推理反而降低得分时
  When more reasoning lowers the score
• 检查底层的 Weave 跟踪记录
  Inspect the underlying Weave traces
• 查看配置和失败原因
  Review configurations and failure causes

研讨会设置Workshop setup

在我们开始之前，如果你想跟着一起操作——虽然这是一个实践工作坊，所以我看到现场有不少笔记本电脑，太好了。嗯，如果你想完全按照我的步骤来操作，你应该有一个 W&B 账户，也就是 Weights & Biases 账户。如果你还没有，可以注册一个免费的账户。我们还会提供推理额度，这样你就可以用这种方式进行评估，也包含沙盒访问权限。嗯，如果你想用别的东西，比如 OpenRouter，也可以。或者用其他服务提供商，随你选择。我会在这次工作坊里用这个作为示例。如果你有一个 AI 智能体，可以直接告诉它改点东西并让它……Before we get started, if you want to follow along, though this is a practical workshop, so I see a lot of notebooks out there, great. Um if you if you want to follow along exactly as I'm going, you should have a wand B account or weights and biases account. If you don't have one, you can register and get a free one. And we also give you inference credits, so you can do an evaluation that way, and sandbox access is also included. And um if you want to use something else like like open router, you can do that as well. Or even a different provider, whatever you want to use. I will just use this as the example in the workshop. And if you have an AI agent, you can just tell it change something and make

认识沃尔夫拉姆·雷文沃尔夫Meet Wolfram Ravenwolf

它和我的配置兼容。那么，我们开始吧。简单介绍一下我自己。我叫 Wolfram Reimann。我的口号是：早餐时评测模型，晚上构建智能体，整天宣讲 AI。我在 Corvus 和 Weights & Biases 担任 AI 布道师，做的就是这些。我还是播客《Thursday AI》的联合主持人。Alex 也在这里。他是主要负责人。如果你在研讨会期间遇到技术问题，可以联系他。他会帮你解决，这样我们就不会被打断。有问题找他就行。谢谢 Alex 的帮助。你也可以在 X 或 LinkedIn 上关注我，还可以访问 Wolf Bench：wolfbench.ai。很好记。 不过，这次研讨会不只是关于 Wolf Bench，而是讲我为什么以及如何构建它，这样你就能从零开始搭建自己的模型和智能体评测项目，一路冲上排行榜。简单说说我的经历吧：我从小就喜欢 AI。it work with my setup. So, let's go. Quick about me. I'm Wolfram Reimann Oh. And my slogan is I have evaluate models for breakfast, build agents at night, and preach AI all day long. I do that as an AI evangelist at Corvus and weights and biases. And um I'm also a co-host on Thursday AI, the podcast. Alex is also here. He's a the main guy. And if you have technical issues during the workshop, you can contact him. He will help you out so we don't get interrupted. But he's your man. Thanks, Alex, for the help. And yeah, you can follow me on X or LinkedIn, and Wolf Benches wolfbench.ai. Easy to remember. Um but this workshop isn't just about Wolf Bench. It's why and how I built it, so you can stand up your own model and agent evil from zero to leaderboard. Um yeah, my backstory basically I started as a little child loved AI from the beginning.

后来 ChatGPT 出现了，我就决定把职业重心放在这方面。嗯，如果有人还记得 LocalLLaMA 的美好旧时光，那时候 TheBloke 在量化模型，我当时就在测试这些模型，或者说，我就是这么入行的。很多人找我帮忙解决推理之类的问题。于是我就成了顾问。然后，去年我全身心投入了一家德国 AI 初创公司，今年则加入了 Qorvo。我上这个播客已经很久了，对吧，Alex？不过今天的重点是你们。这个工作坊是关于你们是谁，以及你们为什么来这里。我觉得大家可以分成不同类型，我很好奇你们觉得自己属于哪一类。比如，智能体构建者。这里有没有人在构建智能体，并且想评估自己的智能体？这边有一位，那边也有一位，还有几位。还有新手。有没有人想做有意义的评估，但还不知道该怎么做，想来学习一下？Then when ChatGPT happened, I decided I want to focus my career on this. Um if anyone knows the old good old times in local llama when the block was quantizing models, I was testing the models or I started that way. Was asked a lot about helping with inference and stuff. So, I became a consultant. And yeah, last year I went full in in a German AI startup and this year I joined Qorvo. And I've been on the podcast for a long time, right Alex? So, but this is about you. So, the workshop, who you are and why you are here. I think there are different kinds of people and I'm interested in which category you think you belong to. So, for instance, the agent builders. Are people building agents here that want to evaluate their agents? Someone here, someone here, a couple. And evil newcomers. Any people who want to do meaningful evaluations but have no idea what they are doing yet and want to learn this?

好，这些人更多。还有模型评估者。谁关心该信任哪个模型和智能体？我该用哪一个？有人特别关注这个吗？那边，还有那边。很好。平台和评估负责人。想要进行可比较、可复现评估的人。这里有一位。这里有几位。很好。轨迹调试人员。这些人会查看实际轨迹，试图找出哪里出了问题，以及如何改进。那边。很好。还有怀疑者。谁不相信排行榜？我是说，不，所有人都应该举手，对吧？Okay, those are more. And model evaluators. Who is interested in which model and agent should I trust? Which is the one I want to use? Anyone interested in that specifically? Over there and over there. Great. Platform and evil leads. The people who want to do comparable and reproducible evaluations. Someone here. A couple here. Great. Trace debuggers. Those are the people who are looking at the actual traces and trying to find out what went wrong and how to improve this. Over here. Great. And skeptics. Who does not trust the leaderboards? I mean No, every hand should go up, right?

主持人：但这是为什么？我们该如何改变这种状况？这正是我们正在探讨的一部分。Host: But why is that and how can we change that? And that is part of what we are

评估思维与方法论Evaluation mindset and methodology

我们正在讨论的内容，以及你如何构建自己的。但首先，我的指导原则是：思维方式高于技能，而技能高于工具。因此，工具就是我们正在使用的东西，而在 AI 领域，工具几乎每周、每个月都会变化。总会有新东西出现，而且一切都在不断变化。技能是指我们如何使用现有的工具。这一点适用于不同的工具。所以，如果你技能出色，呃，知道如何使用这些工具，那么你也可以转而使用其他工具。但思维方式才是最重要的，因为它最难改变或提升。具体到评估，这就是我们为什么要进行基准测试，以及需要注意什么。技能就是如何使用这些工具。而工具】【。talking about and how you can build your own. But first, my my guiding principles are mindset is greater than skill set greater than tool set. So, the tool set is what we are using and that changes in AI changes almost every week, every month. Something new comes out and it changes all the time. So, skill set is how we are using tools that we have. That applies to different tools. So, if you have a great skill, uh, you know how to use the tools, you can switch to other tools. But, the mindset is the most important thing because that is the hardest to change or to improve. And for evaluation specifically, that is why we benchmark and what to watch out for. And skill set is just how to use the tools. And tool

为什么一个平均值会掩盖可靠性Why one average hides reliability

我会向你展示几套工具集，你也可以像我用 Wolf Bench 那样，自己搭建一套。它的前提是：一个分数是不够的，因为性能是一种分布，而不是一个点。我不喜欢排行榜和我见过的许多评测的一点是，你会看到一些漂亮的柱状图和一些数字，但那只是平均值、汇总数据，并不能告诉你太多信息。这就是为什么人们总是（或者经常）会说：“我不相信这个结果，我的体验不一样。”数字背后还藏着一些我们可以找出来、仔细查看的东西。比如，它真的是个好模型吗？它只是碰巧表现不错吗？有哪些隐藏的取舍？诸如此类。这就是我构建 Wolf Bench 的原因，因为我想知道哪些模型可以搭配哪些智能体使用。sets, I will show you a couple of tool sets and you can build your own that way like I did with Wolf Bench. And the premise is one score is not enough because performance is a distribution, not a point. The thing I disliked about the leaderboards and a lot of evaluations I have seen is you see some nice bars and you see some numbers and it's an average, an aggregate, and it doesn't really tell you very much. And that's why people always say or often say, "I don't trust this and I have a different experience." And there are things hidden behind the numbers that we can find out and look at. Because is it a really a good model? Did it have a lucky run? What are the hidden trade-offs? Stuff like that. And that's why I built Wolf Bench because I wanted to know which models can I use with which agents.

这个智能体随着时间推移发生了变化。一月份是 Open Claw，二月份是 Hermes Agent。嗯，我现在用 Codex 用得很多。之前用的是 Cloud Code。我仍然想知道哪些模型搭配哪些设置效果最好。是啊，就像我说的，大多数评测结果最终都浓缩成一个分数。大多数排行榜也只给模型排名。你甚至看不到背后的智能体系统。或者说，不同的版本、设置——即使能看到，也未必知道他们用了什么设置来进行评测。而这些设置差别很大，所以你会看到一些分数，也会看到其他人报告同一个基准测试的结果，但他们使用的设置却不同。那么，它们真的可以比较吗？有没有办法查看那些人得到的结果？所以，Wolf Bench 是一个框架。它有图表、仪表板和排行榜。顺便说一下，它是开源的，而且还有一个……The agent changed over time. In January, it was Open Claw. In February, it was Hermes Agent. And uh, I'm using a lot of Codex now. It was Cloud Code before. And I still want to know which models work the best with which settings. And yeah, like I said, most evaluation results collapse into just one score. And most leaderboards rank the models. You don't even see the agentic systems behind it. Or yeah, different versions, settings, even if you even see what uh, settings they have used for this. And they vary a lot, so you see some scores and somebody else reports the same benchmark, but they use different settings. So, can you really compare them? And is there a way to inspect the results to those people got? So, Wolf Bench is a framework. It has charts, dashboards, leaderboard. It's open source, by the way, and there's also

WolfBench 的五项指标The five WolfBench metrics

方法论，而这正是我最想谈的。所以，你可以将这种方法论用于自己的工具。对于 WolfBench，有一点适用于所有基准测试：你必须决定自己关注哪些指标。你想看平均分吗？还是想看最高分和最低分？你想看它究竟能解决多少百分比的基准测试吗？它的表现有多稳定？成本又是多少？这也是并非所有人都会考虑的因素。它消耗了多少 token？花了多少钱？有时，一个更便宜、更快的模型，最终可能比某个更大型的模型耗时更长、花费更多，因为它消耗了太多 token。methodology, and that is what I want to talk about most. So, you can use that methodology with your own tools. And for WolfBench, one thing that is that is applicable to all the benchmarks is you have to decide which metrics are interested to you. Do you want to look at an average? Do you want to look at the worst and best scores? Do you want to look at how many percent of the benchmarks it can solve at all? And how much does it consistently and also cost. That is also something that not everybody takes into account. How many tokens has it burned? How much did it cost? And it can happen that a cheaper and faster model takes longer and costs more than one of the bigger models because it just used so many tokens.

在统一条件下重复运行Repeat runs under uniform conditions

统一条件也非常重要。也就是说，每次都要使用相同的设置，因为如果你只改变其中一个变量——我们会看到很多这样的变量——那基本上就成了一个新的基准测试。还有多智能体：如果你想使用不同的智能体，该考察哪些智能体？并且一定要多次运行。单次运行就像基准测试抽奖，结果带有随机性，只能算个大致数值。所以你必须多次运行。我用 WolfBench 跑五次，但如果你有足够的资金，当然也可以跑更多次。这始终是一种权衡。对我来说，透明度也极其重要，因为我会公布所有结果，供大家查看。所以，我不会只展示一个分数，然后说这就是为什么这个模型是我的第一名，另一个模型是第二名。Also very important are uniform conditions. So, you have the same setup every time because if you change just the variable, one of the many that are there, and we will look at a bunch of those, if you change it, it is basically a new benchmark now. And also multi-agent, if you want to do different agents, which ones do you look at? And always do multiple runs. A single run is benchmark lottery. It is a random result, and it is just a ballpark number. So, you have to do multiple runs. I do five with WolfBench, but you could do more if you have the money, of course. It's always a trade-off. Uh transparency is also super important to me because I publish all the results, and people can look at that. So, I don't just show a score and say this is why this is my number one model, number two model.

我把所有东西都上传，你就可以检查查看，也可以让你的智能体检查一下。考虑到我们收集的所有数据，这很有道理。所以，基本上就是把思路理顺，I upload everything and you can inspect it and check it out, and have your agent check it out, which makes a lot of sense with all the data we've collected. So, basically get the thinking right,

评估配置，而不只是模型Evaluate a configuration, not just a model

这样一来，这些工具就可以互换使用了。我认为人们最需要意识到的一点是：你不是在给模型排名。如果你在对模型进行基准测试，测试对象是这个模型，但实际上，你测的是某个特定配置。你使用一套数据集，如果更换数据集，那就是一个新的基准测试。你有模型，而模型背后还有推理引擎，以及端点上正在发生的各种事情——这些通常都不是你能控制的。但如果他们更新了推理软件，结果就可能再次发生变化，而你实际上无从察觉。除非你是在自己的系统上运行自己的推理。Agent Orange。另外，Agent Orange 也是如此：如果我测试 blood code 或 code X，下周更新后再测，测到的就会是不同的版本。所以这一点也很重要，要记在心里。and then the tools become interchangeable. And the most important thing I I think that people need to realize is you're not ranking a model. If you're benchmarking a model, it's this model, but you are actually benchmarking a specific configuration. So, you use a data set, and if you change the data set, it's a new benchmark. You have the model, and behind the model you have the inference engine and whatever is happening on the endpoint that you often can't control. But if they do an update for the inference software, it can change the results again and you you have no way to actually see that. Unless it's your own inference on your own system. Agent orange. Also the agent orange, if I test blood code or code X, it is a different version if I test it next week and I do the update. So that is also important to keep in mind.

沙箱也是一样。即使你使用的是某个沙箱服务提供商，如果他们更改了后端的某些内容，你可能根本不会注意到，但这可能会产生影响，也会影响你自己选择的所有设置。所以，只要改动任何一部分，就会变成一次不同的实验。你必须牢记这一点。嗯，好，下面是一个术语表。你们会拿到演示文稿，所以我就不逐一讲解了。你们大多数人都很熟悉这些内容，需要的时候我会做相应说明。我们现在要看的，是一套特定的工具，因为这是一个实践工作坊，而这只是搭建工具集的一种方式，并非唯一的方式。我们会向你们展示不同的方法，你们可以选择自己喜欢的。因此，要跟着一起操作，你应该准备好笔记本电脑和终端；建议安装 Python、UV 和 Git。我们有一些文件可供下载，所以如果你不打算跟着操作，也可以下载结果文件并导入，然后照样使用这套工具。And same thing with the sandbox. Even if you are using a sandbox provider and they change something in their back end, you may not even notice and it could have an effect and all the settings you choose yourself. So change any part and it's a different experiment. You have to keep that in mind. Um okay, here's a glossary. You will get the presentation, so I'm not going through all of this. Most of you are very familiar with this and I will explain whatever is needed at the moment. So we are looking at a specific tool set because it's a practical workshop and this is one way to build it, not the only way and will show you the different ways and you can pick your own stuff. So to follow along, you should have your laptop and your terminal, Python, UV and Git are recommended. We have files to download, so if you are not following, you can download the results and import them and use the tool set anyway.

所以你不必全程跟着做，不过如果你想的话，也可以照着做，并用自己的结果。嗯，对，你应该需要 Wi-Fi，API 密钥则是可选的。我一开始演示过了，待会儿会再演示一次。这样你就能拿到这个。再看这里，截个屏，扫描二维码。通过这里可以进入一个表单，在 AI 工程师大会期间申请免费的推理额度。要申请的话，你需要一个免费的 One B 账号。如果你还没有账号，可以在这里注册。已经有人有账号了吗？有人在用 One B 吗？有不少人。太好了。好，我用的沙箱也是你们产品中的 CoreWeave 无服务器沙箱，每项任务都有一个隔离的环境。这一点非常重要，否则每次评测运行都会改动你自己的系统，结果就会变得更加随机。So you don't have to follow along everything, but if you want to, you can do it and use your own results that way. Um yeah, you should have the Wi-Fi and an API key is optional. I showed it in the beginning. I'll show it in a minute again. So you can get this. So here again, take a screenshot. Scan the QR code. This is where you can get you get to a form where you can get free inference credits for the duration of the AI engineer conference and to do that you need a free One B account. There you can sign up if you don't have already. If you don't already have one. Who has an account already? Does anyone here use One B? Bunch of people. Great. Okay, and also the sandboxes, I'm using the core weave serverless sandboxes in your offering where you have an isolated box for each task, which is super important because otherwise you would change your own system with every evaluation run and then it would be even more random.

评估流水线的六个层级Six layers of an evaluation pipeline

好了，我们现在来看六个可替换的层，逐个选择，你也可以照着做。这就是具体的技能，以及背后的思路。现在，我们来为正在构建的工具包选择不同的工具。首先需要一个基准，也就是一个数据集；还需要一个评估框架，用来运行数据集和进行评估；此外，还需要负责协调这一切并在沙箱中运行它的工具，这样它就不会在你的系统上运行。我们还需要一个智能体框架。我们想用哪个智能体？还有模型和服务。当然，这通常是大家最感兴趣的部分：这个模型表现如何？问题在于，你使用的不只是一个模型，而始终是一个端点。它总是由某个服务提供商托管；如果是 OpenRouter，他们会将请求路由到其他服务提供商。Okay, we are looking at six swappable layers and we'll choose each one and you can do the same. So this is the skill set how to do it and the mindset in the background and now we choose the different tools for this toolkit that we are building. So we need a benchmark, which is a data set. We need an evaluation harness to run the data set and the evaluation and yeah, to orchestrate all of this and to run it in sandboxes so it doesn't run on your system. And also we need an agent harness. What which agent do we want to use? And the model and service. That is of course usually the thing that people are most interested in. How does this model do? And the thing is you are not just using a model, you're always using an endpoint. It's always on a provider if it's open router, they are routing to other providers.

它们数量很多，而且无论是 OpenAI、Google、xAI 还是其他什么，它们都有各自的配置，你也会在某种程度上对这些进行评估。还有所有可以选择的设置。They are a lot and if it's open AI or Google or XAI or whatever it is, they have their own setup and you are also evaluating that in a way. And all the settings you can choose.

选择并固定基准测试。Choose and pin the benchmark

那么，首先要考虑的是：我们用哪个基准测试？你可以从基准测试入手。我之所以这么做，是因为我想用不同的 harness 测试智能体式用法，所以我决定找一个能做到这一点的基准测试。于是我查看了前沿实验室使用过的所有基准测试，发现几乎每家都用 Terminal-Bench。它是一个智能体基准测试，不是编程基准测试。它关注的是智能体任务，比如修复 Git 仓库、破解密码、修复数据库之类的事情——这些都是你在日常使用中可能会让智能体做的任务。这就是我选择 Terminal-Bench 的原因。不过，如果你对多语言方面感兴趣，MMLU 是一个选择；如果你对多模态方面感兴趣，MMU Pro 是一个选择；如果你对 GUI 智能体的使用感兴趣，那就应该选择另一个基准测试。So the first thing is which benchmark do we use? You can start with the benchmark. I did it that way because I wanted to test agentic use with different harnesses so I decided I want a benchmark that does that. So I looked at all the benchmarks that have been used by the frontier labs and I saw terminal bench was on pretty much everyone. And it's an agentic benchmark. It's not a coding benchmark, not just it's a agentic benchmark where there are tasks like like fix the git repository, crack a password, repair a database, stuff you would ask your agent in normal use as an example. And that's why I picked terminal bench. But you could if you are interested in multilingual stuff, MMLU is one thing. or if you are interested in multimodal stuff, MMU Pro is something. If you are used interested in GUI agent use, then you would book pick a different benchmark.

如果你对数学、编程或其他任何领域感兴趣，都有相应的基准测试。关键是找到一个你觉得最有意思、而且具有可比性的基准。当然，你也可以自己制作基准，创建自己的数据集。至于 Terminal Bench，它是一个智能体基准测试，所以我选择用它来评测智能体。目前有 2.0 和 2.1 版本，3.0 也正在开发中。我选择 2.0，是因为我开始时 2.1 还不可用。如果你现在开始，应该选择最新的 2.1 版本；但我用的是 2.0，因为改用新版本就相当于采用了一个新的基准测试，分数也就无法再比较了。实际上，2.1 只修复了 89 项测试中的 28 项，但对所有模型来说，它仍然是一个不同的基准测试。我就得把之前测过的所有内容都重新测试一遍。If you are interested in mathematics, coding, whatever it is, there's a benchmark for it and the thing is find the benchmark that is yeah, most interesting to you and that is comparable. You could also make your own, of course. You can make your own data set. And regarding Terminal Bench, that is a gentic, so that's why I chose it to benchmark agents. There are versions 2.0 and 2.1 and 3.0 is in the making right now. I chose 2.0 because 2.1 wasn't available when I started. If you are starting now, choose the latest version 2.1, but I'm using 2.0 because choosing a new version would be a new benchmark and I could not compare the scores anymore. It actually fixes only 28 of the 89 tests, but it would be a different benchmark for all the models. I would have to retest everything I already did.

我做了很多，但永远都觉得不够。不过我确实做了很多，所以现在不想重新制作一个版本的 Wolf Bench。也许以后某个时候我会做。And I did a lot and never enough, but I did a lot, so I don't want to make a new version of Wolf Bench right now. I will do that at a later point, maybe

Harbor 和评估运行器版本Harbor and evaluation runner versions

到 3.0，或者如果能力有了重大飞跃，而我们需要换一个基准测试。评测框架。也就是说，你已经选好了要使用的基准测试和数据集。现在，你需要一个真正运行评测的工具，而 Harbor 是 Terminal Bench 的默认框架。还有其他选择。Inspector 是英国人工智能安全研究所（UK AI Security Institute）推出的一款优秀框架，提供了许多不同的基准测试供你使用，也非常受欢迎。LM Evaluation harness 我多年前用过，所以它已经存在很久了；OpenAI 也有自己的评测工具。你可以自己构建运行器，也可以选择其他框架。由你决定。有些人也会先选运行器，再从它支持的基准测试中挑选一个。这也是一种可行的方式。现在，进入第一个实践步骤：我们来安装 Harbor。with 3.0 or if a big new jump in capabilities happens and we need a different benchmark. Evaluation harness. This is you have chosen which benchmark you want to use, which data set to use. Now you need something to actually run the evaluation and Harbor is a default framework that is for Terminal Bench. There are others. Inspector is a great framework by the UK AI Security Institute, which has a lot of different benchmarks you can use. It's also very popular. The LM Evaluation harness, I used that years ago, so it's been around forever and OpenAI has their own eval stuff. You can build your own runner or one of the others. You pick what you want. Some people can also Some people start with picking their runner and then they pick one of the benchmarks it supports. That is also a valid way. And now for the first practical step, let's install Harbor.

在这种情况下，我用的是当前版本。昨天它还是最新版本。我不确定现在是不是最新版本。它变化很频繁，但我会把它固定在某个版本上，并且强烈建议大家坚持使用同一个版本，因为每当你更改某个变量时，如果它会影响评估结果——而很多变量确实会——那么你就已经改变了一个变量，比较的可比性也会受到影响。即使它有 bug，至少你测试所有模型时用的都是同一个 bug。不过，这当然是一种直观的想法，因为有时你必须更新：如果新版评估框架支持了不同的 agent 或服务提供商，你可能就不得不更新。我建议真正重视这件事的人 fork 这个代码仓库，维护自己的 fork，然后只把你需要的内容反向移植过来。In this case, I'm using the current version. It was current yesterday. I'm not sure if it's current right now. It changes a lot, but I pin it to the to the version, and I highly recommend people stick to one version because anytime you change one of the variables, if it has an effect on the evaluation, and many have, then, yeah, you are changing already a variable, and the comparison comparability is suffering. Even if it has a bug, you are at least testing all the models with the same bug. But, it's a visual thinking, of course, because sometimes you have to update if a new version of the harness of the evaluation harness supports a different agent or provider, then you may be forced to update. What I recommend for the people who are really serious about this, fork the repository, maintain your own fork, and just backport whatever you need.

如果你安装它，这里缺少一个换行符。它是 UV tool install Harbor。所以它是在 UV 环境中安装的，不会影响你的系统，而且之后可以干净地卸载。然后，你应该会得到这个版本。And if you install it, this is There's a line break missing. It's UV tool install Harbor. So, it's installing in a UV environment, so it doesn't affect your system, and you can cleanly uninstall it again. Then, you should have this version.

隔离沙箱Isolated sandboxes

下一步，智能体需要一个沙箱，因为如果你只是在自己的系统上本地运行它，那么第二个智能体运行时，看到的系统就已经被更改过了，而且还会干扰你的系统，所以我不建议这么做。你可以使用 Docker，这很常见，但根据你使用的系统不同，要实现并行运行会非常困难，因此你需要大量资源。这就是为什么会有云服务提供商。我还要特别感谢 Daytona。在 Qualif 推出自己的沙箱之前，他们从一开始就支持我，而我当时也想从这个方向入手。他们确实给了我很大的支持，我用 Daytona 做了很多评估。现在我们有了自己的产品，所以我也在亲自使用它，因此这里有两个我推荐的选项。The next step, agents need a sandbox because if you just run it locally on your own system, then second agent running, it would find a system that has already been changed, and it would interfere with your system, so I would not recommend that. You can use Docker. That is very common, but it's very hard to parallelize that depending on what system you are using. So, you need a lot of resources. So, that's why there are cloud providers, and I have to give a big shout out to Daytona. They supported me in the beginning before Qualif had their own sandboxes, and I wanted to start with this. So, they really supported me a lot, and I have done a lot of the evaluations I have with Daytona. Now, we have our own offering, so I'm dogfooding it, and um that's why there are two options here that I recommend.

Qualif 沙箱适用于使用 Kubernetes 服务的用户，而这里的王牌是 Qualif 无服务器沙箱。你只需要一个 Vent B 账户。The Qualif sandboxes are for those using the Kubernetes service, and the the golden one here is the Qualif serverless sandboxes. You just need your Vent B account.

嘉宾：关于那个……好吧。我们还有一个，嗯，我……你能……呃，我把 Slack 关掉了，所以这里没共享。如果有人想马上拿到 PDF，直接找 Alex 就行。这大概是目前最简单的办法。嗯，好。那么，安装好之后，呃，Harbor 就装好了。现在，我们需要一个 Harbor 自带的沙箱客户端。这里涉及几件事。这是演示文稿的二维码。对。很好。嗯，`uv tool install harbor` 会安装用于使用沙箱的提供程序，你不需要做任何事。安装完成后，你只需要有自己的 API 密钥。然后在 YAML 文件里配置它，接下来就是这一步。你不用照抄这些内容。后面几张幻灯片里有一个链接，还有一个简单的网址，你可以直接输入，其中包含完整的配置。Guest: What about the Ah, okay. We also have a Um, I Can you uh I I turned off Slack, so it's not shared here. If anyone wants to get the PDF right away, just go to Alex. This is probably the easiest way right now. Um, okay. So, after you've installed this, uh Harbor is installed. Now, we need a sandbox client that is part of Harbor. And in this case, there's a part of a couple of things. Here's the QR code for the presentation. Yeah. Good. Um, UV tool install Harbor in square brackets won't be installs the provider to use the sandboxes, and you don't have to do anything. Once it's installed, you just have to have your API key. And you configure this in a YAML file, and here's just the next step. You don't have to copy this. There is a link and an easy URL you can just enter with the full configuration a couple of slides ahead.

所以，只是为了让你……这只是为了So, just so you This is just so

比较智能体框架Compare agent harnesses

你看到了。不过，如果你正在跟着操作，只需输入 UV 工具安装命令来安装 Harbor。如果你使用这个，就不会是 provider。我们继续。下一步是选择 agent。这是我最喜欢的部分，因为 agent 有很多，我想测试这个 agent 搭配这个模型的效果如何？下面列出了一些 agent。这些就是所有的 agent，而且 Harbor 框架还支持更多。我们有 Cursor、Claude、Klein、Codex、Open Claude Hermes agent，还有 Terminus 2，它是 Terminal Bench 和 Harbor 框架的默认 agent。不过，这并不是真正的 agent。你不能用它来编写代码，或者说你不应该这么做。它更像是在为实际模型模拟 CLI 环境，并发送按键操作，由 agent 执行这些操作。you see it. But, if you are following along, you should just type the UV tool install command to install the Harbor won't be provider if you are using this. Let's continue. Next step, agent. This is the most fun part for me because there are so many agents, and I want to test how does this agent do with this model? And here's a list of some of the agents. These are all the agents, and there are even more that are supported by the Harbor framework. So, we have Cursor, we have Claude, we have Klein, we have Codex, Open Claude Hermes agent, um, and Terminus 2 is the default agent for the terminal bench and Harbor framework. And this is not a real agent. You can't use it for your coding stuff, or you shouldn't. It's more like simulating CLI environment to the actual model, and it sends keystrokes, and the agent executes them.

这是实现这一点的一种方式。比如，Simple Agent 是默认选项；如果你在某处看到 Terminal Bench 的分数，而没人说测试时用的是其他 agent，那他们用的就是这个 agent。接下来，你想测试哪个模型？你必须选择一个模型和一个提供商。如果你使用 GPT 5.5，就会使用 OpenAI。如果你使用 Claude、Fable（如果你能用的话），就会使用 Anthropic。Google 也提供一些 Anthropic 的模型。以上只是其中一部分。OpenRouter 几乎提供所有模型，甚至可能是全部，而且即使启用了 DDR，也不会向模型供应商发送任何数据。我们还有自己的推理核心，提供无服务器推理，你也可以从中选择开源模型。It's one way to do this. So, in simple agent, which is a default if you are seeing Terminal Bench scores somewhere, and nobody said it's tested with a different agent, that is the agent they used. And then, which model do you want to test? You have to pick a model, and you have to pick a provider. If you are just using GPT 5.5, you will be using OpenAI. If you are using Claude, Fable, lucky if you are to use it, then you will be using Anthropic. Also, Google also provides some of the Anthropic models. And these are the just some of those. OpenRouter provides almost all of them, probably all of them, even with DDR enabled, so you are not sending any data to the model vendors. And we also have our own inference core, with serverless inference, where you also get selection of open-source models.

在这个例子中，我想测试的是 GLM 5.2，它是我测试过的最新模型，也是最好的开源模型、开放权重模型。所以我就选它来演示。当然，你也可以选择其他模型。现在要看的是设置。这里……And in this case, what I wanted to test is GLM 5.2, which was the latest model I tested, and also the best open-source model, open-weights model. So, I just picked this for the demonstration. You could choose any other, of course. And now the thing is the settings. There

会改变结果的设置Settings that change the result

有很多设置。有些对大家来说很熟悉，有些可能没那么熟悉。上下文窗口大小也非常重要，因为你总会看到这个模型的上下文窗口大小，但每家服务提供商都不一样。有些服务提供商会选择提供不同的、更小的窗口，以提升性能。比如我们自己的推理服务也不提供 GLM 5.2 的完整 100 万上下文窗口，只提供 256K；这样我们就能更快地提供服务，而且模型质量在达到一定 token 数量后会下降。所以，基本上，使用一半的上下文长度已经能明显感受到差别了。问题是，如果你总是用到 100 万 token，然后再压缩上下文，成本会很高。所以我个人认为 256K 比较合适，这也是我建议推理团队在这种情况下使用这个长度的原因。我在 Codex 里也用同样的设置。are a lot of settings. Some are very familiar to people, some maybe not as much. So, the context window size is also very important, because you always see this model has this context window size, but every provider has the Yeah, some providers choose to deliver a different size, a smaller size, for performance performance reasons. Like our own inference also doesn't offer the full 1 million of G GLM 5.2, only 256K, but that way we can serve it faster, and the quality of the model dedegrades after a certain amount of tokens. So, basically, half context is already very um very much noticeable. And the thing is it costs a lot if you always go to the million and then you compact. So, I personally think 256K, that's why I recommended it to our inference team to use this in that situation. Um I also use the same with Codex.

嗯，基本上，目前它在质量、性价比和速度方面都表现最佳。思考和推理投入。这是一个重要因素，我用不同的思考模式做过不同的评估，它们对模型的影响确实很大，有时还会产生非常出人意料的效果：提高思考投入后，表现反而会下降。不过，等我们讲到那张幻灯片时，我会再谈这个问题。接下来是系统提示词。你并不总是能修改系统提示词，或者……Um basically, right now that gives you the best quality at the best price performance and speed. Thinking and reasoning efforts. This is a big factor and I have done different evaluations with different thinking modes and they really have a big effect on the model and sometimes a very surprising effect where if you raise the thinking, it goes down, but I will tell about this when we get to the slide. So, system prompt. You don't always have the ability to change the system prompt or

提示词、工具和资源限制Prompts, tools and resource limits

甚至检查它，但这是一个非常重要的因素。如果你想想像 Open Law Hermes agent 这样的智能体，你有一个唯一的 MD 文件，里面放了很多指令。现在想象一下，你用它来跑基准测试。如果你给它的指令是“你是一个乐于助人的助手”，和告诉它“你是一个会自主执行任务的智能体”，结果会大不相同。我见过这样的情况：模型或基准测试得出了糟糕的结果，因为模型试图询问用户，但实际上并没有用户，而且它也不知道这一点。所以，如果我修改系统提示，告诉它不能询问用户，它的表现就会不一样，得分也会更高，我很确定。但那就完全改变了我所测试的东西。这里的变量实在太多了。工具配置也是如此。如果你给智能体开放网页访问权限，我见过模型直接去查找基准测试的答案，因为 Terminal Bench 是一个公开的基准测试。even inspect it, but it is a very important factor. And if you think about an agent like Open Law Hermes agent, you have the sole MD where and put a lot of instructions. And now imagine you are using this with the benchmark. And if you give the instruction you are helpful assistant, it will be very different if you give it an instruction it is an autonomous agent doing stuff. I have seen results where the model or the benchmark delivered a bad result because the model tried to ask the user, but there was no user and it didn't know. So, if I change the system prompt and tell it you can't ask the user, it would behave differently and it would get a better score, I'm pretty sure. And that would be completely changing what I'm testing. So, there are so many variables here. Also, tool config. If you give the agent web access, I've seen where a model went out and looked up the solution for the benchmark because Terminal Bench is a public benchmark.

你得查看跟踪记录，看看这里发生了什么。它……这其实不算什么坏事。如果你想要一个智能体，它就应该聪明。如果这个问题已经有现成的解决方案，那它为什么还要自己想办法呢？所以，这取决于你真正想衡量什么，以及你是否想授予它访问权限。并不是所有智能体都会这么做，也不是每次都会。沙箱资源，所有这些东西。如果你给它的 CPU 或内存较少，它可能会耗尽资源，或者运行得更慢，这样完成某件事就需要更长时间，甚至可能崩溃，然后你就会得到糟糕的结果。这不是模型的错，而是沙箱配置的问题。任务超时，每个任务……And you have to look in the traces to see what happened here. It It's not really It's not even a bad thing if you want an agent, it should be smart. And if there's a solution for the problem, why should it try to think of it if it there is a published solution? So, it depends what you really want to measure and if you want to give it access or not. And not all of those do that. Or not every time. Sandbox resources, all the thing. If you give it less CPU or memory, it may run out or it be maybe slower, so it takes longer to do something or even crash and then you get a bad result. And it's not the model's fault, it's because of the sandbox configuration. Task timeout, every task

超时、并发与重试Timeouts, concurrency and retries

会设置超时时间。默认情况下，我觉得时长从 5 分钟、15 分钟到 3 个多小时不等。有一个是 3 个多小时，另一个大约是 2 个小时。这意味着你得决定愿意投入多少时间。我刚开始使用 Wolf Bench 时，给每项任务都固定设置了 2 个小时，因为我不想让开放权重模型处于劣势——它们运行时使用的推理端点通常没有大型顶尖前沿实验室模型那么强大。比如，如果你在本地系统上运行 DeepSeek，它通常没有 GPT 在 OpenAI 系统上运行得快。如果不给它无限的时间，得分就会受到影响。但我发现，设置 2 个小时后，如果模型无法完成任务，有时就会原地打转，不停地尝试、失败、再尝试、再失败。因此成本很高，耗时也很长：如果每项任务花 2 个小时，而总共有 89 项任务，再根据我使用的并发度来并行运行（这就是接下来要讲的一点），整个过程就会特别漫长。gets a timeout. By default, they range from I think 5 minutes 15 minutes to over 3 hours. One is 3 hours something and another one is 2 hours or so. And that means you have to decide how much time you want to invest. When I started with Wolf Bench, I gave every task a fixed 2 hours because I didn't want to basically handicap open weights models which are often running on less powerful inference endpoints than the big SOTA frontier lab models. So, if you are running DeepSeek on a local system, it's usually not as fast as GPT on OpenAI systems. And that would skew the score if you don't give it unlimited time. But I noticed when I was going with 2 hours that if the model can't do the task, it sometimes just goes in circles and tries and tries and tries and fails and fails and fails. So, it was very expensive and took a long time because if I do 2 hours per task and there are 89 tasks and I parallelize them depending on how much concurrency I'm using, which is the next point, it takes so long.

这会花很长时间，也要花很多钱，而且你得不到结果，因为它就是无法完成这项任务。所以我注意到，1 小时是个不错的基准。如果你使用默认的 Terminal Bench，测试时长就会增加。我指的是并发。你肯定不想等 98 89 个小时，甚至等上两倍的时间。你想要的是并发，也就是让多个试验同时进行。但接下来你得决定并发数可以设多高？如果如果我把并发数设为 89，就可以一次性运行 Terminal Bench 的全部 89 项任务。但如果我的推理端点速度不够快，导致请求受到速率限制，最后我就会得到很差的分数，甚至没有分数。沙盒也是一样。如果我有沙盒提供商，就能获得资源；如果每个沙盒使用 8 GB，资源消耗就会很快累积起来，可能会超过允许的限额。It takes so long and it costs so much and you don't get a result because it just can't do the task. So, I noticed 1 hour is a good mark. If you use the default terminal bench the test duration will rise. Concurrency, that is what I meant. You don't want to wait 98 89 hours or even twice the amount of time. You want to do concurrency, so you have multiple trials happening at the same time. But then you have to decide how high can I go? If If I go concurrency 89, I can do all the 89 tasks of terminal bench at once. But, if my inference endpoint is not fast enough and I get rate limited, then I will get bad scores or even no scores at the the end. The same with the sandboxes. If I have sandbox provider, I get resources and if I use 8 GB per sandbox, yeah, it sums up quickly and maybe you get over the limit you are allowed.

重试策略，也就是某件事要重试多少次。如果我不重试某个任务，而它因为沙箱崩溃而失败，那模型就会得低分。但这并不是模型的错。所以，有些错误应该重试，有些则不应该。比如，如果模型自己做了蠢事导致崩溃，那确实是你想要衡量的一种失败。温度参数之类的常规设置。嗯，这对基准测试来说是个难题，因为该用什么温度呢？有些模型提供商会给出他们模型的推荐温度，有些不会。有些智能体有默认值，有些则会根据模型来决定。组合实在太多了，你得决定要不要逐个查找每个模型的推荐设置？Retry policy, how often to retry something. That is also something if I don't retry a task and it fails because a sandbox crashed, I get a bad score for the model. But, it wasn't the model's fault. So, if there are errors that you should retry and there are errors you should not retry. Like if the model causes a crash on its own by doing something stupid, that is actually a fail you want to measure. Temperature, the usual stuff. Um that is a difficult thing for a benchmark because which temperature do you use? Some model providers they give you the recommended temperature to use with their model. Some don't. And some agents have a default. Some agents decide based on the model. And there are so many combinations that you have to decide, do you want to look for every model what is the recommended one?

如果没有推荐值，你会用哪个？你怎么确保做同一项基准测试的人，都使用你选定的相同设置？这很难。top P 也是如此。你用 0.95、1，还是任意一个值？这只是一个设置，还有很多其他设置。还有 max new tokens，也就是允许模型生成多少个 token？你必须对此做出决定，因为服务提供商会给出建议，托管服务商也会有限制。你得选一个值，而且通常还得进行配置。如果可以的话，你也可以设置一个 seed，以减少随机性。我——我刚开始做基准测试时，把 temperature 设为 0，这样随机性就会小一些，但随机性从来没有完全消失过。我还见过很小的改动，比如更新流量驱动器，就会改变我得到的实际结果。If there is none, which do you use? How do you make sure that people who are doing the same benchmark are using the same setting you chose? It's difficult. And top P as well. Do you use 0.95 or do you use one or do you any value? It's just one setting and there are so many others. And max new tokens, how many tokens is the model allowed to generate? You have to decide on this because there are recommendations by the providers, there are limitations by the by the the hosters. You have to choose something and usually you have to configure it. And if you can, you can also set a seed so the randomness is reduced. We I when I started benchmark, I used temperature zero so there was less randomness. There was never no randomness. And I've seen things as small as a traffic driver update to change the actual results I got.

即使一切都是确定性的，或者看起来是这样。因为存在浮点舍入误差，也有性能优化，所以即使把温度设为零，仍有很多因素会改变结果。如今，如果模型推荐使用温度 1，那你当然不应该把温度设为零，否则就违背了它的建议。在这种情况下，你可以设置一个随机种子；但即使你选择像 42 这样的种子——它非常受欢迎，你知道为什么——问题在于，有些模型用这个种子可能会比其他模型更走运。Even if everything was uh deterministic or looked like that. Because we have floating point rounding errors, we have performance optimization, so many effects that change the result even if you use a temperature zero. And nowadays, when the model is recommending temperature one, then you should not set zero, of course. Otherwise, you would be going against the recommendation. In that case, you could set a seed, but even if you choose a seed like 42, it's very popular. You know why. And the thing is, some model may be more have more luck with that seed than others have less luck with that seed.

修复推理服务提供商路由Fix inference provider routing

这些都是你需要做出的选择。现在，针对这个选择——我们要使用哪种推理方式——左边是常规推理核心，使用无服务器推理。如果你想跟着操作，并且想使用 OpenRouter（因为你没有按这里的步骤获取自己的密钥），也可以使用 OpenRouter。无论哪种情况，你只需使用一个 .env 文件，并在下面填入变量。只要把变量和密钥放进 .env 文件，我们就会在代理中使用它们。嗯，如果你使用 OpenRouter，就必须设置一些特殊选项，才能使用我们的提供商——One API 提供商。如果你使用 OpenRouter 并对模型进行基准测试，这一点非常重要。提供商有很多，默认情况下，系统会选用其中一个。它们会决定当前哪个提供商最合适。It's all choices you have to make. Now, for this choice, which inference we want to use, I have on the left side the normal inference core with serverless inference. If you want to follow along and you want to use open router because you don't didn't go through the getting your own key here, you could use open router as well. In either case, you just use an end file and put in the variable at the bottom here. You just put the variable, the key in the end file, and then we will use that with the agent. Um if you use open router, you have to set some special settings to use our provider, the One API provider. This is very important if you are using open router and benchmarking a model. There are so many providers, and by default, you get one of these. They decide which is the best at the moment.

所以，如果你运行一次包含 89 项任务的基准测试，可能一半任务会使用这个提供方，另一半任务会使用另一个提供方。因此，你肯定得设置好自己要使用哪一个。完成这一切之后，你已经为我们讨论过的六个层分别选定了一个部分。我们选择了 Terminal Bench 作为评估基准，选择了 Harbor 及其一个特定版本，选择了我们的沙箱，并选择 Terminus 2 作为智能体。我们还选择了 GLM 5.2，并设置了特定参数。如果你要进行评估，也需要做同样的事。无论你使用什么框架、什么模型，都必须做出选择，并且应该明确说明自己使用的是什么。发布结果时，这一点很重要，这样其他人才能知道你实际使用和测量的是什么。好了，这里还有另一个二维码。So, if you are running a single benchmark run 89 tasks, you may get half the task with this provider and half the task with the other provider. So, you definitely have to set which one you want to use. And after all this, you have chosen one part for each of the six layers we went through. We have chosen Terminal Bench for our evaluation, we have chosen Harbor and a specific version of that, we have chosen our sandboxes, we have chosen Terminus 2 as the agent. GLM 5.2 and we have specific settings. If you are doing evaluation, you will do the same. Whatever framework you use, whatever model you use, you have to make choices and you should be very clear what you are using. This is important when you are publishing your results, so others know what you have actually used and measured. So, here is another QR code.

这样你就能获得完整的配置，不用从幻灯片上复制内容。这里的格式有点乱，请见谅。嗯，如果你下载这个配置，我会在下一张幻灯片中演示如何运行它。在这种情况下，实际获得结果大约需要 5 分钟。我也会做同样的演示。所以，如果你想下载这个配置，这里有二维码。This gives you the full configuration, so you don't have to copy stuff from the slides and here's a formatting that has been messed up a little. Excuse this. Um If you download this configuration, I will show you in the next slide how to run it and it will take about 5 minutes to actually get the results in this case. And I will also do the same. So, if you want to download the config, here's the QR code.

启动 git 恢复任务Launch the git recovery task

接下来我讲下一个。所以，我不只是给大家看，还要实际运行一下。请看这里。我们要做的是，加载 .env 文件，获取前两页提到的 API 密钥，然后运行 `UV run harbor run config` 命令，并提供你下载或创建的配置，这样就能得到结果。我现在就在 shell 里操作。首先，安装 Harbor 框架。执行命令。我把画面放大一点。如果我的 Wi-Fi 按预期正常工作，速度应该是满的。好了，我们让它运行一会儿，我过一分钟再回来。接着讲演示内容。我们正在运行这个程序，它只会执行一个测试。我不想在这里跑 89 个测试，不然我们就得超时一个小时了。And I will go to the next one. So, let me just not just show it, but actually run it. So, take a look at this. What we are doing, we are sourcing the end file to get the API key we find two slides back and then we are running it with the UV run harbor run config command and giving it the configuration you downloaded or created and then we can actually get the results. I will do this right now in the shell. So, the first thing, installing the harbor framework. Executing the command. I will zoom in a bit. So, if my Wi-Fi is working the way it should, it's full. Okay. We are letting this run. I will go back in a minute. So, continuing with the presentation. So, we are running the thing and it will do just one test. I didn't want to use 89 tests for this, otherwise we would be sitting an hour overtime.

而这只是我给每项测试分配的时间，之后验证器还得运行并检查结果是否正确。这也需要一些时间，所以总耗时会更长。不过，Harbor 会从配置中获取任务和 Terminal Bench 2.0 数据集。这里就是配置中指定 Terminal Bench 2.0 版本的部分。然后，它会使用我们正在使用的提供程序启动沙箱。接着，它会为每项任务创建一个沙箱，具体创建多少个取决于并发数。然后运行代理。它实际上会进入沙箱、安装代理并执行任务。给它，给它一个系统用户提示词，基本上就是说：“这就是你要做的事。”代理就会照做。之后，在沙箱被销毁之前，它会收集结果。它还会获取执行轨迹，看看代理到底在做什么。And that is only the time I give every test, so afterwards the verifier has to run and check if it's correct. That takes also some time, so it takes even longer. But, um Harbor pulls the tasks, the Terminal Bench 2.0 data set, it uses that from the configuration. So, this is the part where it has Terminal Bench version 2.0 in the config. Then, it starts the sandboxes using a provider we are using. So, it creates a sandbox per task, depending on the concurrency, how many it uses. Then, it runs the agent. It actually goes in the sandbox, installs the agent, and executes the task. Give it Give it a system user prompt, basically, saying here, "That is what you have to do." The agent does it. And afterwards, it collects the results from the sandbox before it is destroyed. And it gets the trajectories, really, what is the agent doing?

这样你就能获得很多信息。这个小任务是最简单的任务之一：一个 Git 仓库里有个东西丢失了，需要在 Git 树中找到它并恢复。这大约需要 3 到 5 分钟。我们来看看现在进行到哪一步了。现在已经安装好了。接下来，我们安装 One B 客户端。然后，在这个例子中，我要从 1Password 获取 API 密钥。接着，我在这里编写配置。这个配置你可以下载。我用 cat 命令把它写入一个文件，然后加载 env 文件。现在，API 密钥已经放在我的环境变量中了，我接下来……这是运行命令。运行评估。现在它开始运行了，启动沙盒、安装智能体，然后运行。一切都在这里进行。You get a lot of information that way. And this small task, it's one of the smallest, it's just a Git repository where something has been lost, and it should be find it in the Git tree and restore it. This takes about 3 to 5 minutes. Let's see how far we are here. It's installed now. Then, we install our One B client. And the next step, in this case, I'm just getting my API key from the One Password. And then, I'm writing the configuration here. This is the one you can download. I'm just catting it into a file. And loading the end file. Now, the API key is in my environment variables, and I will just This is the run command. Run the evaluation. So, now it's starting to go out, starts the sandboxes, install the agent, run it. And this is all happening here.

保留评分之外的证据Preserve evidence beyond the score

在它运行的同时，我会回到演示文稿。下一部分：作为证据运行，而不只是得到一个分数。这里的格式完全乱了。我会修好演示文稿，然后上传。这只是评估文件夹中内容的目录。等它完成后，我们就可以现场创建这个目录。当然，可能需要大约 5 分钟。现在它正在启动环境，也就是创建沙箱。我们就让它继续运行吧。哦，现在还太早。等我们有了这些之后，会得到一个任务，并对它进行评分。结果要么是 1 分，要么是 0 分。它成功了吗，还是失败了？如果对 89 个任务都这样做，就能知道它完成了多少个任务；然后，比如说再运行 5 次，就能得出一个百分比，从而知道这个模型在 Terminal Bench 中拿到了多少分。And while it's doing that, I will go back to the presentation. Next part. Run as evidence, not only a score. The formatting is totally messed up here. I will fix the presentation and we will upload it. This is just the directory of what is in the in the evaluation folder. We can create this live as soon as it's done. Of course, it will take about 5 minutes should it take. Right now, it's starting the environment, which is creating the sandboxes. So, just letting it run. Oh. This is too early. And when we have that, we get one task and it is scored. It either It gets a score of one or zero. Did it succeed or did it fail? If you do that for the 89 tasks, you get an amount how many tasks it did and then you do that over five runs, for instance, and then you get a percentage and you know how many percent of Terminal Bench did the model get.

然后，这通常就是实验室报告的得分。大多数人只报告五次运行的平均值。这就是 Wolf 的安装过程。And then this is usually the score that the labs report. Most people just report the average across five runs. And this is the installation of the Wolf

WolfBench 集合工具包The WolfBench collection toolkit

Bench Toolkit。之前我们做的所有工作，都是基于 Harbor 框架和 Terminal Bench；这是我编写、创建的东西。Wolf Bench Toolkit 是一种处理这些数据的方式，因为否则你现在手头会有几个文件和一个分数，但你要怎么分析它呢？所以，它是一个收集器，这就是 Wolf Bench 的作用。等我打开程序时，最好给大家演示一下。那我先回到这里。在这期间，我会在一个新的 shell 里启动它。我们只要克隆下来，然后进入目录就行。它是开源的，采用 Apache 2 许可证。你可以把它用于自己的项目。当然，如果你用的不是 Terminal Bench，就得做些适配，不过现在适配起来很容易。只要告诉你的智能体就行。问个问题：有多少人会用智能体来做这类事情？不是所有人都会吗？Bench Toolkit. What we have done before, it was all the Harbor framework and Terminal Bench and this is what I programmed, what I created. The Wolf Bench Toolkit is a way to work with this data because otherwise you now have a couple of files and you have some score, but how do you analyze it? So, this is a collector, what Wolf Bench does and I will It's best if I show it when I open the program. So, let me go back here. While this is happening, I will just start it in a new shell. And we just clone it and go in there. So, it's open source, Apache 2 license. You can use it for your own projects. Of course, you have to adapt it if you are using something else than Terminal Bench, but it's easy to adapt nowadays. Just tell your agent. Quick question. Who's using agents for stuff like that? Not everybody?

哦，我很惊讶。顺便说一句，我还是看到很多笔记本电脑。因为现在有些人直接掏出手机，截个图，然后说：“好吧，智能体，开始吧。把他在舞台上展示的内容都做一遍。”明年。好，现在它正在下载，我也在下载目前为止所有运行产生的结果。先剧透一下，我现在总共跑了呃 430 次。那个……在哪儿来着？呃，是 430 次。不过，总之，这是我目前测试过的智能体：Claude Code、Hermes Agent、OpenClaw、Cursor、Codex。Klein 还没发布，它在我的待办清单上，还有其他一些智能体。如果你正在开发智能体，想让它也出现在这里的话，可以联系我。我还测试了一些模型，其中很多是在 W&B 上测试的，那些是用我们自己的推理服务跑的。所以，最终费用很遗憾没有显示出来的那么高，这里显示的差不多是 4 万。Oh, I'm surprised. I still I still see a lot of notebooks, by the way. Because some people now just pull out their phone, take a screenshot, and say, "Okay, agent, go. Do whatever he is showing on the stage." Next year. Okay, so it's downloading and I'm also downloading all the results I did with all the runs I did so far. So, quick spoiler, I did uh 430 runs right now across um where is it? Uh it was 430 runs. But anyway, here are the agents I tested so far. Cloud code, Hermes agent, open clock cursor, Codex, Klein is not published yet. That is on my list and some other agents. If you are making an agent, contact me if you want to appear as well. And some models I tested, many on those W&B, that was on our own inference. So, the the final cost was unfortunately not as high as it is displayed, almost 40K here.

所以，这里面有些是我们自己推理得出的，不过我只是用了已经公布的价格。好，我们来看看现在的情况。已经下载好了。这 13 兆字节就是结果。如果你想查看所有轨迹，它们都已经上传到 Weave，你可以直接检查。你呃，嗯，不用下载好几个 GB 的数据。现在，如果你想使用 Wolf Bench，可以设置一些变量，它也有上传功能。不过我给你演示一下，直接启动它会是什么样。So, this is some of this is on our own inference, but I just use the prices that have been published. So, let's see where we are here. It's downloaded. So, the 13 megabytes, those are the results. If you want to see all the trajectories, those have been uploaded to weave and you can just inspect them. You uh um don't have to um download gigabytes of data. Now, if you want to use the Wolf Bench, you can set some variables, so it is also an uploader, but I will show you if I just start it.

在 Marimo 控制面板中检查运行记录Inspect runs in the Marimo dashboard

它使用一个 Remeo notebook 来提供仪表板，让你可以轻松处理所有评估数据。好了，马上就出来了。看，这就是一个仪表板。你可以输入相关信息，指定要上传 traces 的位置（如果你有 We 相关内容的话）。然后，如果你使用 XE dev 或 Hetzner，就可以发现虚拟机：它会找到那里的所有虚拟机，你还可以筛选出哪些虚拟机名称对应评估用虚拟机。因为我通常不会在自己的笔记本电脑上运行 Harbor Framework，而是在虚拟机里运行，这样我就可以关掉笔记本电脑，或者避免 Wi-Fi 断开，导致与沙盒的连接失败，因为这也可能会影响……And this uses a Remeo notebook to um provide a dashboard, an easy way to handle all the evaluation data. So, any moment now. There it is. So, this is a dashboard. You can enter your information where you want to upload the traces if you have the We stuff. Then you can discover VMs if you are using XE dev or Hetzner, it will find all your VMs on there and you can filter which VM names are evaluation VMs. Because I don't run the Harbor Framework on my own laptop usually. I run it in a VM so I can turn off my laptop or my Wi-Fi doesn't go off and the connection to the sandboxes fails because that could also affect

筛除不完整和无效的运行记录Filter incomplete and invalid runs

某些东西。嗯，在这种情况下，只要我下载了这些文件（我已经下载了），所有数据就都在本地了。所以我只要扫描本地目录，就能找到我已经运行过的所有运行数据。这里有 443 个不重复的运行。这里还显示了一个筛选器：只有得分的运行才会被纳入测量。它必须包含所有测试。如果有些测试尚未运行，那这次运行就无效，只是一次测试运行。错误或超时也不能太多。然后我会得到一份所有这些数据的列表，可以选择想用哪些模型来制作漂亮的图表。在这个例子中，我会全部使用。之后你也可以更改模型的名称，有时还得更改思考模式，因为你设置的参数可能会被代理修改。比如你使用“思考：超高”，或者想使用“思考：最大”模式，但你的代理不支持，可能它采用了另一种名称。something. Um in this case, everything is local as soon as I download the stuff which I did. So, I can just scan my local directory and it will find all the run data I already did. So, here are the 443 unique runs. And it shows you here a filter so I every run that has a score is is measured. It has to have all the tests. If some tests have not been run, it's not a valid res a run. It was just a test run. It shouldn't have too many errors or timeouts. And then I get a list of all of this stuff and I can choose which models I want to create um uh nice chart for. In this case, I will just use all of them. Then you can also just change the names of the models and sometimes you also have to change the thinking because the parameter you give can be changed by the agent. Like if you are using a thinking extra high or if you want to use a thinking max mode and your agent doesn't support it, maybe it is called a different way.

然后你得对它进行修改，让它真正反映实际使用的内容。现在，所有这些都完成之后，我就可以直接创建结果文件了。你可以……And then you have to change it so it's really reflecting what has been used actually. Now, after all of this is done, I can just create my results file. You can

导出结果并生成图表Export results and generate charts

如果你有密钥，就可以上传到 We view。我还没有设置密钥，所以现在没有这个按钮。现在，我创建一个保存文件，这是一个包含所有结果的 JSON 文件。你可以把它分享给其他人。最后，我生成一张图表，这是我生成的一个产物。现在，根据我得到的所有结果，我生成了一张新图表。接下来也是一样，我还有一个上传器。图表生成后，我只要把它上传到 wolfbenchmark.ai，网站上的内容就会更新。如果我启用这个选项来替换 index.html，它就会立即上线。那么，回到现在……哦，我们来……upload to We view if you have the key. I do not have set the key so I right now don't have the button. Now, I create a save file and this is one JSON file that contains all the results. You can share that with other people. And in the end, I generate a chart which is a artifact I generate. So, now from all the results I did, I get a new chart. And this is the same that I then in the next step, I have an uploader as well. After the chart is generated, I just upload it to wolfbenchmark.ai and it's updated. If I activate this option to replace the index.html and it is online immediately. So, going back to the present Oh, let's

某项任务取得满分A perfect score on one task

先看看基准测试运行到哪儿了。哦，已经完成了。看，已经完成了。试验次数为 1，平均得分 1.0。也就是说，它有一个任务，并且成功完成了这个任务。所以它运行成功了。它已完成运行，并修复了 Git 仓库。现在这个任务就在我的目录里，我可以导入它，但这不是一次完整运行，所以默认不会显示。等我们完成这一步并上传之后……just check how far the benchmark run. Oh, it's done. See, it's done. Trials one, mean score 1.0. So, it had one task, it solved the task. So, it worked. It finished and it fixed the Git repository. And this task is now in my directory and I could import it, but it's not a full run, so it is not displayed by default. And once we have done that and uploaded

阅读交互式排行榜Read the interactive leaderboard

我们可以看看 Wolf benchmark 提供了什么。这又是一个思维方式的问题：我们想关注什么？在这个例子中，我们刚才做的就是这个。我给你们看了仪表板。现在我们要看看排行榜以及它意味着什么，因为排行榜本身，无论生成什么内容，都不只是一个答案，它还能帮助你找到更好的问题。比如说，你在看什么？你关心的是模型的能力，还是模型的可靠性？我会访问 wolfbenchmark.ai，向你们展示这意味着什么。接下来，我把它缩小一点。所有内容从左到右排列，相关信息都在这里。你看过的大多数基准测试，都只显示终端的两个分数，而且只显示平均值。这是一个交互式页面，不仅用于展示结果，还可以用来创建图表，然后复制粘贴到其他地方。this, we can look at what Wolf benchmark provides. This is also the mindset thing again, what do we want to look at? And in this case, this is what we just did. I showed you the dashboard. And now we have to take a look at the leaderboard and what it means, because the leaderboard itself, whatever it generates, it is not just an answer, it helps you to find a better question. Like, what are you looking at? Do you care about the capability of a model or the reliability of a model? And I will show you what that means by going to wolfbenchmark.ai. And the thing is, I will just make it a little smaller. So, they are sorted from left to right and I have all of the information here. Most benchmarks you look at, they just show the terminal's two score and they only show the average. So, this is an interactive page, which is used not only to display results, but to create charts that you can then copy and paste and put somewhere else.

这里有一个截图图标，比如我可以把内容展示在 X 上。如果我想展示这个模型的表现，只要把它标出来就行。我可以移除一些条形，也可以重新排列它们，所以它是交互式的。我可以这样生成报告。它还会把这些内容保存在 URL 里，所以只要把 URL 发给别人，他们看到的内容就和你屏幕上显示的一样。现在我切换到这里。有时候看起来有点让人眼花缭乱，因为信息太多了，同一个条形上有非常多的信息。就像我说的，你可以筛选不同的信息，让条形图变得简单一些，但我喜欢这样看，因为底部颜色较深的部分，There's a screenshot symbol so I can show stuff on X for instance. So if I want to show how did this model do, I can just mark it. I can remove some of the bars or reorder the bars. So it's interactive. So I can create my report that way. And what it also has it in the URL. So everybody can if you give the URL to this, they see the same thing you had on your screen. So I'm switching over here. And it looks a bit overwhelming sometimes because there are so many information so so many information on the same bar. Like I said, you can filter to different information to make it a simple bar, but I like to see this because the darker part at the bottom,

一致性、平均值和上限Consistency, average and ceiling

这个百分比表示，在我进行的五次运行中，每次运行里模型解决了整个基准测试的百分之多少。这就是我所说的一致性。这个模型和这个智能体每次都解决了整个基准测试中 60% 的任务。这意味着，如果这个分数很高，那么模型就非常可靠，因为只要它能解决某项任务，你就可以期待它每次都能解决这项任务。黄色的分数则是平均分，也就是通常会报告的那个分数。它周围的向上和向下箭头，分别表示五次运行中的最高分和最低分。如果这两个分数彼此很接近，那么方差就不大，因此你可以期待得到相同的分数。如果它们相差很大，就像这里从 79% 到 60% 这样，那就说明方差很大。这表明模型的表现存在很多随机性。that percentage is how many percent of the whole benchmark has a model solved in every run of the five runs runs I did. So that is what I call consistency. This model and this agent, they have solved the task 60% of the whole benchmark every time. And this means if that is a very high score, the model is very reliable because if it solves the task, you can expect it to solve the task all the time. While the yellow score is the average, the one score that usually is reported. And the up and down arrows around it, those are the best and the worst scores in one of the five runs. So if those are very close to each other, then there's not a high variance. So you can expect the same score. If they are very far apart like we see here from 79% to 60%, that means there's a lot of variance. And that is a sign that the model there's a lot of randomness involved here.

你其实不能完全相信它会执行任务。即使得分很高，它能不能做到也有点随机。最高分是星标，那就是上限。它至少成功完成过一次的基准测试任务占百分之多少？任务数量能告诉你……是啊，理想情况下，如果模型完美无缺、每次都能做到，那这个分数就应该是它始终能拿到的分数。不过，这也说明它可能是针对这个基准测试专门优化过的。嗯，但它能让你看到模型的能力上限。所以，你想要的是非常非常高的上限、非常稳定的平均水平，以及非常高且扎实的基础表现。如果两个模型的分数相同、表现也几乎一样，你还是能看出它们的基础表现有所不同。这种情况下，我会选基础表现更好的那个。You can't really trust it to execute the the task. Even if it has a high score, it's a bit random if it does it. And the highest mark is the star, that is the ceiling. How many percent of the benchmark has it solved at all, at least once. The amount of task that tells you how Yeah, in an ideal case, that would be the score the model should get all the time if it's perfect and always doing it. Although, that would also be an indicator that it is probably benchmarked for this. Um but it shows you how far it can go. So, what you want is a very very high ceiling, a very stable average, and a very high solid base. And if a model has the same score and they are almost the same, you can still see that the solid base is different. And then in that case, I would pick one that has a higher solid base.

就像这里，是 73% 对 74%，几乎一样。如果只看平均分，两者几乎完全相同。可能只是方差，单纯的方差。但这里的差距更大，有 6 个百分点，差异很明显。所以，你看，正如我们预期的那样，超高强度思考优于中等强度思考。不过，如果我们看……嗯，这也很有意思。我会在演示文稿中展示。让我切换到演示文稿。这些就是不同的……Like here, it's 73% to 74%. It's almost the same. If you just look at the average score, it's almost exactly the same thing. Could be a variance, just variance. But this is a bigger difference. It's 6% points, and that is a noticeable gap. So, you see the extra high thinking is better than the medium thinking, as we expected. But if we are looking at um This is also very interesting. I will show it in the presentation. Let me switch to the presentation. So, these were the different ways to

令牌用量和总成本Token usage and total cost

想象一下这个模型。啊，风险和成本承受能力。我得给你看看这个，因为这是……如果你看我评测中排名第二的模型，那就是 Gemini 3.5 Flash。所以，这个 Flash 模型的表现几乎和 GPT 5.5 一样好。那是怎么回事？如果我只看 Terminus，这样柱状图就不会太多；我也可以说，我只想比较这两个。没错，Gemini 3.5 Flash 是个很棒的模型。但如果我看看它是如何达到这个表现的，就会看到一个 3D 视图，显示它为此用了多少个 token。它用了数十亿个 token 才达到这个分数，而 GPT 5.5 用的少得多。我放大一下。GPT 5.5 用了 1.36 亿个 token，相比之下，Gemini 3.5 Flash 用了 16.5 亿个 token。而且，如果我们再看看价格——这是另一个 3D 视图——这就是成本。imagine the model. Ah, risk and cost tolerance. I have to show you this because this is something If you look at the second best model on my evaluation, it was Gemini 3.5 Flash. So, this model, a flash model, is almost as good as GPT 5.5. So, what gives? If I just look at terminus, so we don't have so many bars, and I can also say I just want to compare these two. And yeah, it's a great model, Gemini 3.5 Flash. But if I look at how did it achieve that, I have a 3D view, which shows you how many tokens it used to achieve that. And it has been billions of tokens to get that score, while GPT 5.5 used much less. Let me zoom in. So, 136 million tokens for GPT 5.5, compared to 1.65 billion tokens for Gemini 3.5 Flash. And the thing is, if we also look at the price, which is another 3D view, this is the cost.

我甚至可以同时展示这两项。现在成本显示为阴影。你可以看到成本就在这里。它们重叠了，因为它用了很多 token；但这说明 Gemini 3.5 Flash 与 GBD 5.5 非常接近，不过它用了太多 token，所以成本当然甚至比 GBD 5.5 还高。在这个例子中，它的成本是 620 美元，而 GBD 5.5 是 497 美元。所以，差不多是 500 美元到 600 多美元。也就是说，并不是说你……是的，你得考虑这一点。你想用一个又快又便宜的模型吗？但你得比较一下，它真的更快、更便宜吗？因为所有这些 token 也需要时间。这是一点。回到演示上来。这是一个决策面，不只是一张图表。所以，你加载快照，如果你也做了自己的快照并运行代码，就可以把它添加进来并看到结果。然后你可以像我刚才展示的那样，按不同的指标和模型进行筛选。And I can even show both at the same time. The cost is now a shadow. You see that the cost is over here. It's overlap because it used so many tokens, but this shows that Gemini 3.5 Flash is very close to GBD 5.5, but it used so many tokens that it cost even more than GBD 5.5 of course. So, it costs in this case $620 compared to $497. So, 500 to over 600. So, it's not that you Yeah, you have to think about this. Do you want to use a fast and cheap model? But you have to compare is it really faster and cheaper because all those tokens also take time. That is one thing. Going back to the presentation. And it's a decision surface. It's not just a chart. So, you load your snapshot and you can add your own if you did it and you execute the code, you would see them. And then you can filter like I showed you different metrics and models.

你可以重新排列这些柱形图，然后导出自己的版本。之后你可以展示出来，围绕它发一条社交媒体帖子、写篇文章，或者做任何你想做的事。嗯，你可以这么做。在这个例子中，我想看看哪个模型在这项基准测试中的表现最好。所以，目前排名第一的是 GBD 5.5 和 Codex。它甚至比 Fable 还好，因为如果你想知道 Fable 怎么了，那也很值得看看。如果我把 Fable 放进来，它有 13 次拒绝。也就是说，它拒绝作答了。再看看成本，花费高得惊人。它是我做过的最贵的一项基准测试。别告诉任何人，因为光是测试 Fable 就花了 11,000，而且它还失败了，因为它拒绝了太多测试。哎呀。当然，这也是你必须做出的判断。人们会说：“哦，我不相信这项基准测试，这个模型排在第一位。”You can reorder the bars and export your own. And then you can show this and make a social post about this or write an article, whatever you do. Um you can do this. In this case, I was looking at which is the best model on this benchmark. So, my number one right now is GBD 5.5 with Codex. It is even better than Fable because if you are asking what happened to Fable, that is also very interesting to take a look at. If I put Fable in here, it's had 13 refusals. So, it refused. And if you take a look at the cost, it goes so far. It was my most expensive benchmark. Don't tell anyone because it cost 11,000 just to test Fable and it failed because it refused so many tests. Whoops. And that is also something of course you have to decide. That is when people say oh I don't trust this benchmark this model is in the first place.

看看这里发生了什么。基准测试必须足够透明，这样你才能查看运行轨迹，了解发生了什么，然后一切就会豁然开朗。好吧，回过头来说，正如我所说，表现最好的模型是搭配 GPT 5.5 的 Codex，但你也看到了，它的费用相当高。我们放大看看，花费将近一千美元。而使用相同模型的 Cursor 只花了 282 美元。相比之下，这便宜多了，而且这里消耗的 token 也少一些。不过，你可以说这是最佳组合，但从价格和性能的角度来看，另一个可能更好。Terminus 的表现上限也最高，但如果各项分数彼此非常接近，那结果可能只是随机波动。要想非常确定，你得运行 10 次，甚至更多次，但有时候你必须做出选择，选一个然后继续用下去。Look at what happened here and the benchmark has to be transparent so you can look at the traces and see what has happened then it makes sense suddenly. Okay. Going back and yeah like I said the best model was Codex with GPT 5.5 but as you can see it was pretty very expensive. Let's zoom in. It cost almost a thousand bucks. While Cursor with the same model cost only 282. So that was very cheap compared to the other one and it used a little less tokens here. But yeah that's you could say okay this is the best combination but maybe the other one is from a price and performance perspective the better one. And Terminus also had the highest ceiling but if the scores are very close to each other that could be random. You have would have to do 10 runs or even more to be very sure but you have to make a choice sometime and pick one and go with it.

这也是我刚才展示的内容：Gemini 3.5 Flash 和 GPT 5.5 相比，分数更低，账单却更高，所以在这方面……This was also the thing that I just showed Gemini 3.5 Flash compared to GPT 5.5 lower score bigger bill so in that

当更多推理反而降低得分时When more reasoning lowers the score

在这种情况下，使用另一个模型会更合理。当时我测试 Opus 4.6 时还发现了一件特别有意思、让我很惊讶的事：我把思考等级调到最高后，分数反而下降了。它的表现确实变差了。我在 Claude Code 里也做了同样的测试，因为我也用了 Titanus，在那里看到了同样的现象。对于 GPT，如果我把思考等级从关闭调到超高，或者从中等调到超高，随着思考等级提高，模型会变得更聪明，这正如你所预期的那样。那么，为什么 Opus 的表现反而变差了呢？这让我很意外。如果我看执行轨迹，区别就在于你会发现模型想得太多了，过度思考了。这意味着我得给它 1 小时的时间。如果模型想了一两分钟，然后决定要执行某个命令，但机器上还没有这个命令，它就会再想两三分钟。case it would make more sense to use the other model. And this was also super interesting and surprised me a lot when I tested Opus 4.6 at the time I raised the thinking to max and the score decreased. It really got worse and I did the same with Claude Code because I used Titanus as well and there I saw the same effect. When I do that with GPT if I increase from thinking off to extra high or from medium to extra high if I raise the thinking level the model gets smarter as you would expect. So why did it get worse with Opus? That was surprising to me and the difference if If look at the traces is that you see the model it thinks too much. It's overthinking. And that means I give it 1 hour. And if the model thinks for a minute or two and then decides I want to execute this command and the command doesn't exist on the machine yet, then it thinks another two or three minutes.

那我该怎么安装这个？而一个根本不思考的智能体，只是在做事。所以我最先想到的就是运行这个命令，收到一条错误消息；再运行这个命令，又收到一条错误消息。我安装它，收到一条错误消息，然后修复问题。这样它就能进行更多轮交互，也就能做更多事情。这就是花一个小时思考却什么也不做，与不断尝试、反复尝试、通过试错来解决问题之间的区别。有意思的是，GPT 模型默认情况下看起来没有 Opus 那么聪明。Opus 开箱即用就非常强大。即使关闭思考功能，它的得分也已经很高了。相比之下，它在这里关闭思考功能时的表现，已经超过了 GPT 5.5 在这里开启中等思考时的表现。而且它的基础能力也更强。所以这些都是你需要关注的方面。And how do I install this? While an agent that is not even thinking, it's just doing stuff. So the first impulse is I run this command, get an error message. I run this command, get an error message. I install it, get an error message, fix it. And it gets more turns. So it can do more. That is the difference between thinking for an hour and not doing anything or just trying, trying, trying, trial and error. And the interesting stuff is that the GPT models, they are by default they appeared not as smart as Opus. Opus is very capable out of the box. Even with thinking off, it already got a great score. If you compare that here, it is already with off better than medium here on GPT 5.5. And even the solid base was higher. So it did more tasks in every run. Yeah. So that is stuff you have to look at.

如果结果令人惊讶，就看看执行轨迹，弄清楚这里发生了什么。模型想得太多了。顺便说一下，我在 Fable 上也看到过同样的情况：模型非常聪明，而且似乎知道自己很聪明。于是它决定朝这个方向做。即使基准测试表明应该那样做，它还是说：“不，我懂得更多，我要按自己的方式来。”最后，验证器运行并进行测试，结果失败了。所以在某些情况下，模型聪明反被聪明误。这也是一个我必须向你展示的重要特性。这个指标是一项主张，And if it's surprising, take a look at the traces to find out what is happening here. Model overthinking. I also saw the same with Fable, by the way, where the model was very smart and it seemed to know it was so smart. So what it did is it decided to go this direction. And even if the benchmark said it should be done that way, said, "No, I know better. I will do it my way." And then the verifier at the end, it ran, tested, and failed. So the model was too smart for its own good in some cases. And this is also an important feature I have to show you. The bar is a claim,

检查底层的 Weave 跟踪记录Inspect the underlying Weave traces

轨迹就是证据。我一直在讲轨迹。所以我们有这个条形图，也就是我们在 Wolf Bench 上看到的。你能看到这些漂亮的条形。关键是，你可以点击任意一个条形，随便哪个都行。点一下就会跳转到我们的 One 页面，我们在这里有 weave，它可以让你检查所有轨迹。在这个例子中，我刚才点了一个条形，现在看到的是构成这个得分的全部五次运行。现在我可以选择任意一次运行，点进去，然后，呃，这个我应该删掉。我稍后再展示。呃，消失吧。所以，我们有 89 个和弦，也就是我一直在执行的 89 个任务，你可以在这里看到所有轨迹。数量很多，所以加载得有点慢。然后你可以点进其中任意一个，查看任务名称、奖励、任务是否成功，以及所有步骤。the trace is evidence. I'm talking about traces all the time. So we have the bar, which is what we see on Wolf Bench. You have the nice bars. And the thing is you can click any of the bars, any of these. You just click it. And you get taken to our One page, where we have weave, which allows you to inspect all the traces. In this case, I just clicked one bar, and I'm seeing all the five runs that composed that score. And I can now pick any of the runs, go in here, and uh this I should remove. I'll show this later. Uh go away. So, we have 89 chords. That are the 89 tasks that I have been doing, and you can see all of the traces here. It's a lot, so it's loading a bit. And then you can go into any of these, and you see the task name, and you see the reward, did it succeed or not, and you see all the steps.

你会看到最初的请求，也就是用户发来的那条消息：“你是一个助手。你的任务是处理这件事。”接着，你会看到 LLM 发出的第一条命令、它进行了哪些分析、它打算做什么，以及它随后发送的命令，等等。你可以仔细检查并查看这一切发生的过程。或者，你也可以使用你的智能体来完成这件事。我们还有一项新功能：平台内置了一个智能体。因此，你可以在 weave 里使用智能体，直接告诉它去做这件事。我的同事 Tim Sweeney 将于 7 月 1 日就此发表演讲，届时你可以去看看。我会使用智能体。You see the initial request, the user message that says, "You are an assistant. You are tasked to do with this." Then you see what the LLM did as a first command, what analysis it did, and what it wants to do, and then the command it sends, and so on. And you can really inspect all of this what has happened, and look at it. Or use your agent, and have this done. And we have a new feature, which is also an agent built into the platform. So, you have an agent inside of weave, and you can just tell it to do this. There's a talk about this by my colleague, Tim Sweeney is doing it on July 1. There you can check it out. I use agents

检查配置和故障原因Review configurations and failure causes

很多，毕竟阅读所有追踪记录要花很长时间。如果智能体给你一个 Weave 链接，而你发现它所说的确实属实，那也很好。总而言之，影响基准测试的变量非常多：基准测试版本（OneBench 2.0、2.1）、智能体版本、模型服务和路由，甚至推理端点的更新，都可能改变结果和设置。还有沙箱资源，正如我所说，如果智能体耗尽资源，就会失败，导致模型得分很低，尽管这并不是模型的错。还有采样、温度，以及我们讨论过的所有事项。思考默认设置也很重要，例如，如果思考时间过长，就不提高最大值；规模更小、响应更快的智能体可能效果更好。还有超时策略和重试策略。如果不重试，而此时发生了基础设施错误，就会得到很低的分数。a lot, because reading all the traces takes a long time. And if the agent gives you a link to weave, and you see that it's true what it says, that is also very good. So, to recap, there are so many variables relevant to benchmarks. You have the benchmark version, the OneBench 2.0, 2.1, the agent version, the model service and route, even updates to the inferinference endpoint can change the results and the settings. Sandbox resources, like I said, if the if the agent runs out of resources, it will fail, and then you get a bad score for the model, although it wasn't the model's fault. Sampling, temperature, all things we discussed. Thinking defaults, like maximum not being higher if it thinks too long and a smaller, faster agent reaction could work better. Time out policy, retry policy. If you don't retry and there has been infrastructure error, you would get a bad score.

因此，每次运行结束后，我都会逐一检查所有错误和失误，看看是否存在推理问题。如果出现重复字符，或者提示模板中的令牌泄漏到了实际响应中，就说明可能是分词器或推理服务器出了问题。尤其是对于新模型，这一点很重要。如果你在模型刚发布时就对它们进行基准测试，它们还很新，推理引擎可能尚未提供最佳支持。所以，我曾做过一些评估，当时新模型表现不太好。一周后，我用完全相同的设置再次评估，结果模型表现变好了，因为推理引擎得到了改进。这也是整个过程的一部分。所以，记住这个要点：所有组件都由你自己选择。你已经有了这些构建模块，可以看看演示文稿，想想自己想做什么、要用哪种评估、哪个智能体，以及要测试哪个模型。So, after every run I go through all of the errors and mistakes and check if there have been inference issues. You can find that out if there are repeated characters or if prompt template um tokens leak into the actual response, then you know there's a tokenizer issue, an inference server issue. And especially with new models, that is a big thing. If you are benchmarking models as they come out, they are so new that the inference engine maybe has not the best support yet. So, I have done evaluations where the new model wasn't that great. A week later I did the same, everything was the same, and it was better because the inference engine has been improved. That is also part of this. So, use the message, bring your own everything. You have the building blocks, check out the presentation, think about what you want to do, which evaluation to use, which agent to use, which model to test.

现在，你已经具备了相应的思维方式，知道该关注什么，并能将其应用于制作更可靠的排行榜、图表和评估，而不是随便跑个测试、发布一个分数。所以，请运行你自己的、可复现的基准测试配置，并分享结果。如果你想和 Wolf Bench 合作，欢迎联系我。如果你想运行一些模型，那就太好了。我们可以合作并发布结果，这样我就不用把所有事情都做一遍了。如果你想基于它做一个分支项目，也很棒。用其他框架也行，随你喜欢。如果你正在构建模型或智能体，欢迎联系我，我可以对它们进行基准测试。我已经在和一些智能体开发者合作，很快就会发布结果。好了，这基本上就是演讲的全部内容。希望你能从中有所收获。You have the mindset now to know what to look out for and apply this to make uh leaderboards and charts and evaluations that are more reliable than if you just run something and publish a score. So, run your own reproducible benchmark configurations. Share the results. If you want to work with Wolf Bench, contact me. If you want to run some models, that would be great. We could collaborate and publish the results so I don't have to run everything. And uh if you want to make an off-spin, that's also great. Use another framework, whatever you want. And if you are building models or agents, feel free to contact me so I can benchmark them. I work with some of the agent uh creators already and will publish results soon. Yeah, that's basically the end of the presentation. I hope you could get something out of it.

如果你们还有什么问题，现在可以提问。否则，可以到展位来找我。还有，非常感谢 Alex 的帮助。播客《Thursday I》每周四都会播出，时间是早上——我所在的时区，哦，是太平洋时间早上 8:30。每周都是如此，因为要跟上正在发生的事情实在太难了，对我们来说也是如此，所以我们不得不这么做，好把这些情况报告给你们。好了，谢谢。And if you have some questions, now is the time. Otherwise, you can catch me at the booth and uh big shout out to Alex for the help. The podcast Thursday I is every Thursday and in the morning the time zone I'm from another time zone. 8:30 um Pacific time. Every week because it's so hard to keep track of what's happening and it's hard for us as well so we are forced to do it so we can report it to you. Okay, thank you.