别再和表格聊天：设计值得信任的财务分析智能体 | Francesco Galletta 与 Felippe Felisola Caso

简要描述Brief Description

Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso， Monzo

目录Table of Contents

• Intro: Francesco & Felippe
• 利用 AI 创造影响，而非一味堆砌 token
  Using AI for impact, not token-maxing
• 速度不是瓶颈，信任才是。
  Speed isn't the bottleneck. Trust is.
• 组件与设计原则
  Components vs. design principles
• 信任的五项原则
  The 5 principles of trust
• 使用场景：为婚礼做预算
  The use case: budgeting a wedding
• 这些原则如何层层递进
  How the principles build on each other
• 原则 1：共享领域
  Principle 1: Shared domain
• 实时演示：使用代理进行规划
  Live demo: planning with the agent
• 从计划到决策树
  From plan to decision tree
• 为什么冗长的线性聊天会变得混乱
  Why long linear chats fall apart
• 原则 2：一致性（数学逻辑应写在代码中）
  Principle 2: Consistency (math belongs in code)
• 原则 3：证据（拿出凭据）
  Principle 3: Evidence (show the receipts)
• “主文件”恐怖故事
  The "master file" horror story
• 原则 4：问责制
  Principle 4: Accountability
• 原则 5：验证
  Principle 5: Verification
• 黄金示例与外部参考点
  Golden examples & outside reference points
• 问答：你应该构建多少个 harness？
  Q&A: How many harnesses should you build?
• 别把斧头磨得太锋利。
  Don't over-sharpen the axe
• 问答：技能与专用工具框架的区别
  Q&A: Skills vs. a dedicated harness
• 要点：有意识地以建立信任为目标进行设计
  Takeaway: design for trust on purpose

Intro: Francesco & Felippe

早上好。感谢大家来到这里，也欢迎大家参加 AIE。我想这大概是大家的第一场会议，对我们来说也是。Morning. Thanks everyone for being for being here and welcome to AIE. I guess first first session for everyone and for us as well.

主持人：我们想先做个自我介绍，告诉大家接下来要讨论什么。我们觉得，不能再靠电子表格来沟通了。所以，今天要聊的是如何搭建一个值得信赖的财务分析师工作框架。我们是 Franchesco 和 Felipe。先简单介绍一下我自己。我来自委内瑞拉，在英国住了大约七年，最近搬到了马德里。我在 Data 工作，也和 Felipe 一起在一家银行工作。呃，我喜欢在每个去过的城市寻找最棒的餐馆。如果你们有纽约的餐馆推荐，我很乐意听听，也想在城里到处逛逛。我这些爱好听起来这么普通，是因为我大部分时间都在做东西、沉迷于各种技术，牺牲了自己的私人时间。所以我的爱好才会列成这样一张清单。Host: So h we wanted to introduce ourselves and tell you about what we're going to discuss. We think we need to stop talking with spreadsheets. So, this is all going to be about building a financial analyst harness that you can actually trust. We are Franchesco Filipe. A little bit of a quick introduction about me. I'm originally from Venezuela. I lived in the UK for about seven years and recently moved to Madrid. H I work at data and I work at a bank as well with Felipe. Erh. I like just finding the best eeries in any city that I visit. So if you have any recommendations for New York, I'm more than happy to take them and just travel around. And the reason my hobbies sound so vanilla is because I just spend most of my time building stuff and nerding out at the expense of my personal time. So that's how you end up with a list like that.

嗯，然后挺不错的。好。嗯，很高兴认识大家。嗯，简单自我介绍一下。我叫 Philipe，原籍巴西。嗯，我也搬到了英国，到现在一年半了。这段旅程很有意思。嗯，生活几乎彻底颠覆了。不过，是的，能在那里生活还是挺好的。嗯，我常讲一个故事：在决定要不要在大学里学这个之前，我每天弹钢琴，简直能弹七个小时。嗯，所以当时自然而然就决定去学电气工程了，毕竟谁不会这么选呢，对吧？嗯，后来有一次，我开始接触数据，因为有人给我看了一个当时叫 Google Data Studio 的东西。现在它叫 Looker Studio，不过他们后来又改回去了。我也不太清楚到底是怎么回事。嗯，我做的第一个仪表盘竖得离谱，简直像一卷莎草纸。Uh and then nice. All right. Um happy to meet everyone. Um also quick intro. I'm Philipe. I'm originally Brazilian. Um, I also moved to the UK. It's been a year and a half now. So, it's been a fun journey. Um, life turned pretty much upside down. But, yeah, it's good to be it's good to be there. Um, I like to tell a story that before going to to decide whether I wanted to do it in university, I used to play piano for like literally seven hours a day. Um, and the natural decision there was to then study electrical engineering because who wouldn't, right? Um, I ended up at some point landing in in data because someone showed me something that at the time called uh Google Data Studio. Now it's Looker Studio, but now they went back. I don't I don't even know what's going on. Um, and then my first dashboard looked like a papyrus of how vertical it was.

你得一直往下滚，滚啊滚啊滚啊滚，才能看到点东西。嗯，不过是的，剩下的就是历史了。所以我在很多初创公司、小团队和规模化成长型公司工作过。嗯，后来最终去了 New Bank 这样的大公司。也许你们有些人知道。嗯，之后又回到了 Monzo。我会说它处于规模化成长型公司和大型企业之间，不过目前发展得非常快。嗯，是的，我现在大部分时间都在 Monzo 工作，主要是为数据团队构建由 AI 驱动的内部解决方案，以提高内部工作效率。所以，我们基本上是想通过运用 AI 让数据团队变得更强。嗯，是的，我不跟机器人和智能体聊天的时候，你可能会在某个地方找到我在攀岩，嗯，可能是在户外，也可能是在健身房。嗯，还有一些刚开始像……You had to scroll scroll scroll scroll scroll down to see something. Um, but yeah, the the rest is history. So I worked at a lot of um startups and small teams and scaleups. Um and then eventually went to a big company at New Bank. Maybe some of you do know um and then before going back to to Monzo which is I would say a middle turn between a scale up and a big enterprise company but it's growing uh so fast for now. Um and yeah most of my time currently has been working at Monzo with building internal AIdriven solutions for internal productivity primarily for the data team. So, we basically want to get our data team better by um building with AI. Um and yeah, when I'm not talking to robots and agents, uh you can probably find me climbing somewhere um either in nature or in a gym. Um and some starting like

利用 AI 创造影响，而非一味堆砌 tokenUsing AI for impact, not token-maxing

我们今天为什么要把这个话题带给大家？我想，Franchesco 和我都曾在金融服务领域的数据团队工作过，比如 Monzo、New Bank 以及这个行业里的其他几家公司。我们现在正在推动 AI 在这个领域发挥作用。我们在 Monzo 的团队一直在从影响力的角度积极探索如何采用 AI。我们不想一味追求 token 使用量，也不想花很多钱却无法带来实实在在的业务价值。因此，我们采用 AI 的方式非常审慎，也已经取得了很好的成果。 另一方面，我们也一直在做另一件事，尤其是 Franchesco：运用 AI 让生活更轻松。我们并不是想说“我要搭建一个开放的客户端实例”——这本身没什么不好——然后把它连接到所有东西上。我们真正想做的是找出生活中对自己重要的事情，再借助 AI 的帮助，让这些事情变得更轻松。points of why are we bringing this for you today? I think um both Franchesco and I worked in data teams in financial services. So again, Monzo, New Bank and a few other companies in the in the sector. Um, we're now pushing what AI can do there. Our team at Monzo has been um, working a lot with with adopting AI from an impact perspective. We don't want to be like token maxing. We don't want to be spending a lot of money if that doesn't bring us like concrete business value. Um, so the adoption is very very deliberate and we've been seeing very good results on that. And on the flip side, this is something also something we've been we've been doing, Franchesco particularly is applying AI to make our lives easier. Um and again not through the approach of I'm going to set up an open client instance nothing against it um and just give it connections to everything but actually what are the things that I care about in my life that I can um get the help of AI to to u make easier.

所以这就是为什么我们说，在这个例子中我们并不是在代表 Monzo。我们会运用从 Monzo 带来的经验，但会和你们一起构建一个示例，让我们能在实践中真正看到它的效果，同时不用讨论太多领域知识。如果你们对我们在 Monzo 正在构建的东西感兴趣，明天我们有一场分组会议，会讨论 Agent Chip，也就是我们内部开发的远程代理能力。好了，Franchesco，交给你。So this is why we say we're not representing Monzo um in this example because we're going to use the knowledge we bring from Monzo but uh build with you an example where we can um actually see that in practice without having to discuss a lot of domain knowledge. Um but if you are curious about the things we are building at Monzo we have a breakout session tomorrow um where we're going to talk about agent chip which is our inhouse remote agent capability. Um so with that Franchesco back to you.

速度不是瓶颈，信任才是。Speed isn't the bottleneck. Trust is.

嘉宾：很好。提醒一下，今天的目标是建立信任。这听起来可能有点违背直觉，因为感觉有些空泛；而且这是一个工作坊，你们本来应该埋头写代码两个小时。不过你们会看到，我们会稍微换个角度来看这件事。因为我觉得，我们已经不必再花好几个小时敲键盘、反复折腾，才能做出值得信任的东西了。 那么，我们先从这个领域说起。金融是因人而异的。你有自己的数据，也有自己的策略。有时候，你其实并没有什么策略。你对金融、个人情况等方面有所了解，通常也会存在一些误解。 谈到金融分析师时，这个职位有非常明确的工作描述。我们对这个工作描述做了稍微宽泛的解释。Guest: Cool. So, as a reminder, today the goal is trust and it may seem a little bit counterintuitive because it feels very wishy-washy and this is a workshop and you're supposed to be coding really hard for two hours. H but you're going to see we're going to give it a little bit of a spin. Uh because I think we're past the point of having to spend hours like grinding through your keyboard uh to get something that you can trust. So, let's start with the domain. Finance is tailored. You have your numbers. You have your strategy. Sometimes you don't really have a strategy. And there is the knowledge that you have on finance, on your personal situation and so on or generally misconceptions. H when we talk about a financial analyst, there is a very specific job description. We are being a little bit generous with that job description.

所以，它涵盖的范围非常广，从为投资、投资公司、银行等做研究，一直到个人用途。但金融就是金融，对吧？所以现在，产出成果、掌握自己的财务状况、进行分析，真正的瓶颈并不是速度。我相当确定，你们当中有些人可能已经在用 Claude 或 ChatGPT 跑后台任务，处理一些事情。所以这并不是瓶颈。根据我们过去几个月的经验，我们认为现在的瓶颈在于它如何与我们建立联系。嗯，信任这件事不能想当然，尤其是我们具体谈到金钱问题时，必须坦诚以待。这是风险最高的议题之一，所以我们需要投入精力，确保系统内建了信任机制。而且坦白说，如果金钱对你来说还不算高风险，或者你能一次性解决自己的金钱问题、预算、财务安排等等，那研讨会结束后请来找我，我会带你入门，聊聊一些投资点子。那我们之后再聊。嗯，一切都从目标开始，而这个目标就是赢得信任。So it's going all the way from doing research for investments, investment firms, banks and so on all the way to like personal use cases as well. But finance is finance, right? So right now the gap on producing something, getting hold of your finances, doing an analysis is not really speed. H I'm pretty sure some of you may have some background tasks running with like claude or chpt getting some stuff done. So that's not the bottleneck. H we believe that based on the experience we had over the last few months, the bottleneck now is how it connects with us. Um so trust is the thing we cannot take for granted and transparently because we're talking about money specifically. Erh this is one of the highest stakes topics uh we need to invest in making sure that there is trust built into the system and transparently if money is not high stakes enough for you and you can like oneshot your money problems your budgeting your financing everything else h find me after the workshop I will buy you in into some investment ideas so yeah let's chat later please um so where everything starts is from the goal which is earning the trust.

这样你就能真正把枯燥的工作委派给多个智能体、你的智能体群、你的工厂，诸如此类。但你怎么才能相信得到的结果呢？呃，我要随口提几件事，因为说实话，它们都不是什么新东西。你可能已经在琢磨了：嗯，可以通过设置追踪、运行循环，或者我们掌握的其他那些关键词来建立信任。所以，用智能体做单元测试和规划，我们知道这很有用。呃，做评估当然也行，还有追踪、可观测性、循环和图表——我可以一直列举那些你认为能开始建立信任的组成部分，但它们基本上都只是一些组成部分。呃，我们想要的是……So then you can truly delegate the boring work to multiple agents, your swarms, your factories, whatever. But how you do trust the results that you're getting. Erh, I'm going to namerop a few things because honestly none of them are new. You're probably already grinding thinking well you buy trust by h setting up traces or doing loops or any of the other keywords we have. So unit testing that gets old planning with your agent we know that that's good. Uh doing evals for sure traces and observability loops and graphs like I can keep naming components of how you think you can start building trust but they are basically components. Erh what we want

组件与设计原则Components vs. design principles

今天我们实际要讨论的是如何通过设计原则来建立这种信任。要构建一个真正能处理金钱事务的智能体框架，需要极大的信任。所以我们不会讨论具体技术，也不会给大家念某个产品的 README；我们会假设你们已经了解前面提到的一些概念。嗯，这次工作坊的形式会稍有不同，因为我们希望你们亲身感受这些原则在现实中如何发挥作用。今天我们围绕一个非常简单的例子搭建了一个智能体框架，希望你们试试看。但别太关注“如果我输入这个，就能更快得到响应”，或者“界面有多流畅”之类的——先剧透一下，它不会有那么流畅。呃，但更重要的是，我们希望你们感受一下：对于一个基于特定智能体框架构建的智能体，我们可能会如何建立信任。to discuss today is actually the design principles on how to achieve that trust. So building a hardness that actually h works for money matters h requires an in im immense amount of trust. So we're not going to be discussing specific technologies. We're not going to read you the readme of a particular product and we will assume you know some of the concepts that have been listed so far. Um and mainly this is going to be a slightly different workshop because we want you to feel how the principles work in reality. So we have built a hardness today around a very simple example and we want you to try it out but don't focus on oh if I type this then I get the response so much quicker and the UI is so slick. It spoiler it won't be. Erh but more importantly we want you to feel how it may feel like having trust in an agent that has been built with a very specific hardness.

呃，所以再说一次，这只是一个非常简单的实现示例。呃，如果你觉得这太高层次了，之后可以尽情吐槽我，不过我们可以晚点再讨论这个。所以，信任……我可能很快就会因为一直说“信任”而疯掉，不过我们认为你……Uh so again this is just one very simple implementation an example h you can roast me later if you think this is too top level but we can discuss that later. So trust and I'm probably going to run out of times I can say trust without going insane but we think you

信任的五项原则The 5 principles of trust

需要从头开始构建，对吧？所以你们有一个共享的领域，也就是围绕计划的整个体系，确保你和模型使用相同的术语。这听起来比实际做起来要简单得多。然后是保持一致性，我很确定房间里的每个人都会达成共识，但关键是要确保这些数字来源正确。因此，与其让语言模型处理大量算术，不如让它调用一些工具。它会为我们编写一些脚本，并尝试找到一种确保结果一致的方法。我们还需要证据，基本上就是拿出凭据，对吧？你可以追溯并梳理决策，以及这些决策是如何得出的。问责制可能是个相当敏感的话题，也可能是其中最抽象的一项，但归根结底，即使你在运行 100 个智能体，它们所做的事情产生的结果仍然要由你负责。need to build it from the ground up right. So you have a shared domain, the whole thing about a plan, making sure that you and the model are talking in the same terms. And it sounds more trivial than it actually is. Then consistency and pretty sure everyone in the room will be on the same page, but it's making sure that the numbers come from a correct place. So rather than a language model trying to process a lot of arithmetic, it will do some tool calling. It will write us some scripts. It will try to come up with a way that the results are uh consistent. We also want evidence where basically show the receipts, right? You can follow back and trace back the decisions and what led to them. Accountability is probably a very spicy one and maybe the mo most abstract of them, but ultimately even if you're running a 100 agents, the results of what they do, you're still accountable for them.

所以，这是我们在工作中非常重视的一点。我们认为，这是一种思维方式的转变：一旦人们完成了这种转变，就会开始构建能够与他们共同演进的系统，而不是把问题一股脑扔进黑箱里，指望能碰巧奏效。最后是验证。我们要确保有一个独立的观点，也就是另一个视角。我得承认，我大概每年会有三次觉得自己想出了世界上最棒的点子，结果菲利佩却会告诉我：“别把那个付诸实施。求求你，千万别。”So that's something we take very seriously at work. And we think it's a mindset shift that when people make it, they start building systems that evolve with them and not black boxes that they just throw problems at. hoping that something sticks. And finally, verification. So, we want to make sure that there is a separate point of view, another perspective. H I would say I'm guilty of thinking I have the best idea in the world probably like three times a year and then only to have Filipe tell me, please don't implement that. Please, please, please.

主持人：我，我——哎呀，等一下。我想就这一点补充两点。第一，我们有很充裕的时间可以一起合作。所以，在我们逐步构建这些原则的过程中，能向你们学习经验会很有帮助。如果你们做过的任何工作与此产生共鸣，或者在使用智能体时遇到过什么特别的挑战，请尽管提出来，然后我们可以一起完善。嗯，再强调一下弗朗切斯科刚才说的一点：我们并不是说这就是唯一的模式，也不是要走上大舞台说“忘掉你们知道的一切，这才是做事的方法”。这只是我们在一些应用场景中行之有效的一种方法。稍后我会分享几个 Monzo 的例子，弗朗西斯科也有其他一些例子。不过，再说一遍，没有什么万能灵药能解决每一个问题。我们也欢迎你们提出质疑。Host: And I I Oops, one second. And I would just compliment on on this bit that I think two things. The first one is we have a very generous window here to to work together. So um it will be very nice to learn from your folks's experience as well as we build through this principles. So if this resonates with any work you've done or any particularly any challenges you've faced um when working with agents feel absolutely free to to raise that and then we can we can build it uh together. Um and just dunning uh doubling down on something that Franchesco said this we're not saying this is the model you won't say us going to to the big stage and saying forget everything you know this is how to do it this is a model a way to approach it uh that has been working for us in a number of use cases I'll share a few examples at Monzo later and Francisco has a few other examples as well um but yeah again there's not a single silver bullet to to solve every single problem we And also feel feel free to challenge us.

如果你觉得信任其实被高估了，那就往一个问题上扔100个智能体，问题总会解决的。很乐意听听成功案例。不过，基于反复实践、经验，以及工作内外的许多不同用例，我们确实认为，你需要构建一个系统，让你有信心事情正朝着正确的方向发展，而不只是看起来没问题。所以，是的，信任。皆大欢喜。嗯，说到信任，我得坦白，今天这场工作坊我另有目的。呃，所以我马上就要辜负你们的信任了。是什么呢？不到五分钟。你们是来帮我的，Francesco，If you think actually trust overrated just throw a 100 agents at a problem eventually will get solved. Happy to hear the success stories. H but from repetition from experience and a lot of different use cases at work and outside. We do think you need to build a system that gives you the confidence that things are going in the right direction and not just that things look correct. So yeah, trust. Happy days. Um, speaking of trust, I'm here to confess that I have an ulterior motive with today's workshop. Uh, so breaking your trust immediately. What is it? Less than five minutes. You are here to help me, Francesco,

使用场景：婚礼预算规划The use case: budgeting a wedding

来规划我的婚礼。所以大家都不是特别兴奋，不过你们会兴奋起来的。呃，具体来说，就是想搞清楚一些婚礼费用方面的事。嗯，既然这是个会议，我想把气氛弄得尽可能尴尬。所以，举手示意一下。呃，有谁办过婚礼？谁结过婚？好，太棒了。宾客人数少于50位的？to plan my wedding. So nobody's super excited, but you will get excited. uh specifically just to figure out some some stuff around the finances of paying for a wedding. Um and because it's a conference, I want to make it as cringey as possible. So, show of hands. Uh who has had a wedding? Who's gotten married? Okay, awesome. Number of guests under 50.

嘉宾：是啊。好吧。不，还是举手吧。还是挺尴尬的。好，行。Guest: Yeah. Okay. No, still show of hands. It's still like cringey mode. Okay, cool.

主持人：大约100。是的，不错。150怎么样？Host: About 100. Yeah. Nice. How about 150?

嘉宾：好的。Guest: Okay.

主持人：不错。200。Host: Nice. 200.

客人：再来点。好吧。Guest: More. All right.

主持人：是啊，是啊，盛大的婚礼。不错。这大概会引起很多人的共鸣，不过听起来这似乎只是个非常简单的计算、非常简单的流程，婚礼也会顺顺利利地办完。但结过婚的人大概都知道，事情并非如此。嗯，甚至要把账算清楚，弄明白费用怎么会这么快就涨得这么高，也不是件容易的事。这算是个小故事吧。希望我女朋友——未婚妻——不会看到这段，不过我们当时在算账，她说：“不对，嗯，但每顿晚餐的费用是这么多。”我就说：“没错，而且除此之外还有另外两项浮动费用。所以总数不是那个数。”总之，要给这场婚礼做预算实在很难，甚至可以说有点低估了难度。我不建议这么做，不过嘛，我已经上了这条船。Host: Yeah. Yeah. Big wedding. Nice. This is probably going to resonate, but it sounds like it's going to be a very simple calculation, very simple process, and the wedding is going to go by nicely and smoothly. And for the people who got married, probably you know that is not the case. Um, and even running the calculations and figuring out how things got so expensive so quickly. It's not trivial. This is a little bit of storytelling. And hopefully my girlfriend doesn't fiance doesn't see this, but we were doing some calculations and she was like, "No, yeah, but the cost per like dinner is this much." And I'm like, "Yeah, and you have another two variable costs on top of that. So that's not the number." And anyway, budgeting for this has been a challenge to underelling it a little bit. I wouldn't recommend doing it, but you know, I'm already on the train.

呃，所以今天你要帮我做婚礼上的无偿劳动，或者你也可以按自己的计划来。嗯，我们把几样东西放进了一个你能访问的仓库里。所以如果你想选择自己的冒险，那就去做吧。我不建议这么做，因为我确实需要帮忙，不过随你选。Erh, so today you're going to give me a hand with the wedding unpaid labor or you can do your own plan. H So we put a few things in a repo that you're going to get access to. So you if you want to pick your own adventure, do that. wouldn't recommend because I do need help, but whatever you choose.

嘉宾：那么，我们给你几分钟来设置一下。你可以扫描二维码，也可以访问我的代码仓库，里面会有设置说明。我们没打算在这段时间放背景音乐，比如电梯里的那种音乐。Guest: So, we'll give you a few minutes uh for you to set this up. Just either scan the QR code or go to my uh repo and there will be some instructions on how to set it up. We did not plan a background music like an elevator music for this moment.

主持人：好的，如果你在设置时遇到任何问题，请告诉我们。Host: Yeah, let us know if you have any any trouble setting up.

嘉宾：另外，如果有人唱歌的话，现在正是……的好时候。Guest: Also, if anyone sings, it would be a good time to

主持人：卡拉OK直播。Host: karaoke live.

嘉宾：顺便说一下，你现在还不需要执行任何操作。只要确保代码仓库已按你偏好的方式配置好，我们之后会指导你完成执行。Guest: By the way, you don't need to execute anything just yet. Just make sure you have the repo in your preferred um setup and then we'll guide you through the the execution.

主持人：不错。呃，所以你刚才克隆了一个病毒。开玩笑的。不过里面有很多引导内容。这个结构确实说得通。我们准备了一份指南，介绍今天的工作坊该如何开始。嗯，还有今天分享中最重要的示例，也就是帮我筹划婚礼。嗯，还有工具框架本身——我们搭建它，是为了向你展示这些原则如何应用。最后，还有一个文件夹，用来存放你做的所有偏离我们当前内容的东西。所以再说一次，这有点像“选择你自己的冒险”，只是多了一点风险：如果你卡住了，我们可能帮不了你。不过你可以自己尝试，而且今天做的所有东西，你都可以带回家，看看自己做了什么、喜欢什么、不喜欢什么。尤其是工具框架本身——这也呼应了我们刚才开的玩笑：它目前就做到这个程度。Host: Cool. Uh, so you just cloned a virus. No, just kidding. But a lot of vioded stuff in there. The structure does make sense. So we have a guide on how we start for today's workshop. H, we have the example most important thing of the talk today. So helping me figure out my wedding. H, also the harness itself, the thing we build to show you how the principles are applied. And then finally, there is a folder where everything you build that drifts away from what we're doing is going to get stored in. So again, it's a little bit of a pick your own adventure uh with the added risk that we may not be able to help you if you get stuck, but you can um and everything you do today, you can take it home and you can check what you did and what you like, what you didn't. And particularly I was going to say particularly the harness itself talking to to the jokes we were just making about like this is as far as this was built.

嗯，再说一下，我们在这里分享的这些原则、这里构建的这些原则，正如我们昨天才讨论过的，这有点像是我们给你们看一张桌子，教你们怎么造桌子，如果你们明白我的意思。嗯，但我们确实、确实认为，嗯，构建本身并不难。难的是知道我们想用什么方式来解决问题。所以我们才会用这种方式来讲。不过我们有时也会稍微揭开一点幕布，带你们看看内部是怎么运作的，展示一下我们所应用的这些原则背后究竟发生了什么。Um again the the principles we're sharing here the principles that are built here we were discussing just yesterday is like uh this can feel like we are showing you a table to teach you how to build a table if that makes sense. Uh but we really do we really do believe that the again building is not the hard thing. The hard thing is knowing how we want to approach the problem. So that's why we're bringing it like this. But we're gonna open the open the look under the hood a bit um at times just to show you what's going on um behind the the principles we're applying.

嘉宾：基本上就像菲利佩说的那样，这感觉就像是在说：“嘿，Claude，请帮我打造一家价值 100 万美元的公司。”如果你不清楚自己为了实现目标所遵循的原则和做出的决策，那么不管尝试多少次，你很可能都只是在碰运气，指望好事偶然发生。Guest: And pretty much like Filipe was saying, it can feel akin to saying like, "Hey Claude, please build me a $1 million company." If you don't know exactly what are the principles and the decisions you're making in order to make something happen, doesn't matter how many times do you try, you're likely going to be just leaving yourself up for the randomness of something good happening by accident.

这些原则如何层层递进搭建而成How the principles build on each other

所以，我们想再次确立这些原则，因为它们是层层递进的，而且我们认为它们的顺序安排得不错，逻辑上也说得通。首先，建立一种共享的领域规划模式，叫什么都行。然后，确保一致性，保证凡是能够或应该可靠运行的东西都能可靠运行。接着，要有证据支撑决策的依据、信息的流向，以及是否在进行测试，等等，并把这些摆在最显眼的位置。然后是问责：你该如何充分掌握机器在做什么？你不想完全进入黑箱模式，因为那样一来，问责就无从谈起了。我敢肯定，在座的很多人都遇到过这种情况。但另一种做法也不太现实，比如说：“你每做一次改动，就把差异发给我。”So we want to anchor the principles again because they build on top of each other and we think there is a nice sequence that kind of makes logical sense. So building that shared domain plan mode however you want to call it. Then setting up consistency making sure that whatever can run or should run reliably does then having the evidence to back up where the decisions are coming from what information is coming through if things are being tested and so on and putting that front and center. Then the accountability how do you keep a good enough grasp of what the machine is doing? You don't want to go full blackbox mode because then your accountability is gone. And that has happened to many of us in this room, I'm sure. But the other side is also not very realistic where oh for every change you make send me the diff.

我肯定会逐行检查。以前从来没人这么做过。呃，最后是验证。所以在数据和金融领域，你往往需要一个独立的参照点，一个可以拿来对照的反事实。忽略等式中的这一部分，只会让你按照自己的偏见行事，所以结果总会有所偏差。这其中也有一些目光短浅的因素。因此，这些原则只是为了引导我们的努力，让我们把精力用在最有用的地方，最终得到真正值得信赖的东西。嗯，再提醒一下，今天肯定会有东西出问题，尽管联系我们。我们会尽量和你一起修好。有些东西也设计得过于复杂了。这不是一场关于优秀 UI/UX 设计的讲座。好了，这只是想让你看看它可能是什么感觉，展示一下它的潜力。I'm going to review it line by line for sure. Nobody has done that ever. Uh and then finally verification. So in data and in finance uh you often need to have a separate point of reference a counterfactual that you can check against h and ignoring that part of the equation just leads to you running things on your own biases. So your results are always going to be skewed. There's some shortsidedness that plays into that. So these principles are just to try to funnel our efforts into where is most useful to get something that we can actually trust. Um heads up again something will break today surely just call us to you. We will try to fix it with you. H some of it is also overengineered. This is not a session on good UI UX design. Okay. So this is just to show you how it could feel, show you the potential.

呃，我们绝不是在做一个产品。嗯，有问题就举手。哪里不太顺利，或者有点困惑，就举手。我们会在教室里走动看看。嗯，你可以检查一下能不能正常运行。试着运行那个命令。对。我想现在我们可以到大家那里看看，帮忙解决遇到的问题，也给大家五分钟开始做这一部分。如果设置顺利，你应该会看到 UI 打开，终端里也会显示成功。很好。然后第二条命令会打开一个设计得很不错的 UI，供你进行一些操作。下面是我们要演示的示例，因为这东西有点像分层结构，你知道，就像洋葱或者食人魔一样。我们会演示一个示例测试框架如何在一个示例用例上运行。Uh by no means we're building a product. Um and yeah, just raise your hand. Something's not working out, you're a little bit confused, raise your hand. We're going to float around the room. Um so you may want to check if this works. Try and run that command. Yeah. And I think at this point we can just go around if things are not working for anyone and give everyone five minutes to get going with this bit. You should see the UI opening if everything goes successful with the setup works success in the terminal as well. Cool. And that second command there will open a very well-designed UI for you to do some stuff. Here is the example we're going to show you because this is kind of like layered, you know, like an onion or an ogre. We're going to show you an example hardness being run on an example use case.

如果你想运行最下面那个命令，就会看到一个更完整一些的示例。但如果你只运行第二个东西、只使用 UI，可能会觉得我让你用 UI 来看看硬度是如何表现的很奇怪，但这正是重点。重点是让你感受一下它在现实世界中应该是什么样的。嗯，我要不要先把它留在这里一会儿？If you want to run that command at the bottom, you will see kind of like a more fleshed out example. But if you stay on the just running the second thing and just use the UI, it may feel weird that I'm asking you to use an UI to see how a hardness behaves, but that's the point. The point is to show you how it feels in should feel in the real world. Um, should I leave it here for a moment?

主持人：嗯，我想是的。Host: Yeah, I think so.

嘉宾：是的。好。好。我听到打字声少了。我就当你已经输入了想输入的命令。呃，那么我们来看看实际的……Guest: Yeah. Okay. Okay. I hear less typing. I'm going to assume you've typed whichever command you wanted to type. Erh, so let's go through the actual

原则 1：共享领域Principle 1: Shared domain

步骤以及硬度和共享域的原则是第一项。能不能请打开过 UI、但还没做过其他操作的人举一下手？steps and the principles of the hardness and the share domain is the first one. Can I see little show of hands of who opened the UI and hasn't done anything else with it?

主持人：好，好，好，好，好。行，长话短说，你右手边有一个聊天窗口，里面基本上什么都没有。呃，如果你想通过一个例子来聊聊财务方面的顾虑，或者做一些你想做的分析，那就可以从那里开始。基本上，我们想从希望实现的目标出发，尽力给出解释。你可能每天都在和你的智能体做这件事，但要注意理解一点：你很可能和智能体掌握的背景信息并不一样。嗯，等等，我能不能前后翻一下，以防万一？好。对。对。演示。嗯，我想解释一件事，而且我觉得大多数人都没意识到，模型所知道的东西不一定就是你以为它知道的。所以，除非你特别有钱，还专门训练了很多模型，否则，呃，你对某些事物的定义很可能和大多数人所采用的定义不完全一样。Host: Cool, cool, cool, cool, cool. All right, so um long story short, you have a chat window on your right right hand side and it has nothing essentially. Erh, if you want to follow through an example of a financial worry, some analysis you want to do, that's the place to start. And basically, we want to walk from the goal we want to achieve and just try our best to give an explanation. You're probably doing this daily with your agents anyway, but focus on understanding that you're likely not going to have the same context as the agent. Um, wait, can I go back and forward just in case? Okay. Yeah. Yeah. Demo. Um, so I wanted to explain something and I think most people don't realize the fact that what the model knows is not necessarily what you think. So unless you're very rich and you have a lot of models trained for you, h you're like likely your definition of certain things are not going to be the exact as what the majority of the population defines.

所以，如果你在查找某个特定术语，智能体——模型可能足够聪明，知道它属于这个领域，但它未必就是你想找的东西。所以，和你的模型来回交流确实很有帮助。而这种难度是为了鼓励你在自己尝试时深入探究，挑战一些自己的假设，并问自己一些你可能还没意识到需要问的问题。所以，如果你想做……对。那么我来做和你们一样的事情，不过我要做一个 harness 示例。我回到这里一下。确认我都准备好了。有人完全跟不上吗？还是大家都还好？好吧，也许你能帮他一下。So if you're looking up for a certain term, the agent may the model may be smart enough to know well this sits in this space but it's not necessarily the thing that you meant. So going back and forth with your uh model really makes a difference. And the hardness is supposed to encourage you if you're trying it out yourself to dig deeper to challenge some of your assumptions to ask you questions you may not knew you needed to to ask yourself. So if you want to do Yeah. So let me do the same thing that you folks are doing, but I'm going to do harness example. Let me go back here for a second. Just make sure I have everything Is anyone completely lost or are good? Okay, maybe could you give him a hand

嘉宾：我运行所有内容的时候……对于后来到场的各位，我也会把二维码给你们，这样如果有人弄丢了，就可以再拿一次。好吧。嗯，在我们让大家继续操作的时候，我先给你们看两样东西。这就是你们使用婚礼示例运行时应该看到的内容，对吧？嗯，我们一会儿会解释这里发生了什么。如果你选择……我也给你们看看，如果你选择自己确定方向，你会看到类似这样的空白界面。然后，Francesca 演示婚礼示例时，你也可以说说自己想用 harness 来做什么。比如，嗯，我也不知道，策划自己的婚礼、规划旅行、为你的公司或个人生活规划一项新的投资，都可以。Guest: while I I I run everything. And for the folks that arrived later, I will give you the QR code as well for the so if anyone lost that you can you can pick it up again. Okay. Um let me show you two things while I while we we let you go with it. So this is what you should be seeing when you run with a wedding example, right? Um we're we're going to explain in a bit um what's going on here. If you are choosing just to show you as well, if you are choosing to uh pick your own pick your own direction, um you're going to see something like this where it's empty. And then what you can do is as Francesca is going to run through the wedding example, you can say something you want to to use a harness for as well. So things like I don't know plan your own wedding, plan a trip, um plan a new investment in your company or in your personal life, doesn't matter.

嗯，任何与财务生活相关的东西，我们都可以用同样的原则来实现。再说，如果你想稍微看看底层是怎么回事，如果需要在两者之间切换，比如婚礼那个例子和你自己的例子，你只要删除 my 和 var 文件夹里的那个数据库就行，这个数据库基本上就是用来存储这些内容的。所以直接删掉就可以了。哎呀。然后用之前的命令再次打开界面，你就会看到示例。Um anything that relates to financial the financial life that we can um implement the same principles. If you ever need uh again just opening the hood a bit. If you ever need to switch between one and another like the the wedding example or your own um you can just delete the in the my and var folder you're going to have a database that's basically what's storing this. So you can just delete it. Oops. And then once you open the the UI again with the previous command, you're going to get the example.

主持人：你想打开干净的那个吗？Host: Do you want to open the clean one?

嘉宾：可以，我能。没问题。Guest: Yes, I can. Sure.

主持人：是的。所以，要认识到你拿到的真的是一张空白画布。我们想稍微明确一下。如果你打开婚礼示例，那是一个包含所有要素的完整示例。但假设我们从这里开始，对吧？我遇到了一个问题。你那里有听写功能吗？Host: Yeah. So, appreciating that you're literally you're literally being given uh a blank canvas. We want to disambiguate a little bit. So, if you open the wedding example, that's a work through example that has all the elements. But let's say this is where we're starting, right? And I have a problem. Do you have a dictation thing there?

嘉宾：嗯，我有。嗯，Guest: Um, I do. Um,

主持人：好的。Host: okay.

嘉宾：我们来看看这能不能正常运行。等一下。也就是说，在右侧那个聊天窗口里，你看到的是一个熟悉的界面，对吧？那么，如果你想针对某个具体问题向 AI 智能体寻求帮助，该怎么做呢？只要开始聊天就行。你可以先提出一个问题，然后描述你想达成什么目标。所以，Guest: Let's see if this is working. Wait a second. So, basically on that like chat window on the uh right hand side, you are it's a familiar interface, right? So, how would you go about getting some help from an AI agent for whatever specific problem? Just start with a chat. You start with a a problem and you start to describe what you want to achieve. So,

实时演示：与智能体一起规划Live demo: planning with the agent

如果让我来做，我会说我有一场婚礼，需要分三次付款。一次付 6,000 美元，另一次金额还不确定，呃，还有一次付剩下的钱。另外，呃，我的一些开销是固定的。我知道 DJ 要花多少钱，也知道装饰要花多少钱吗？不，我不知道装饰要花多少钱，而且我也不知道会有多少人来参加婚礼。那就是我要为每个人、每顿饭、每个座位支付的金额。所以，我甚至不知道该怎么着手，才能确保我能按时付款。按时，按时按时。嗯，我就说让我先把前面那部分删掉。好。所以，如果你想用一个你自己更在意的问题来试试，你就从那里开始，而我所做的就是和智能体一起规划。如今，大多数智能体框架都有某种机制，可以判断你是在为一个问题做规划，并会尝试提出合理的思路，比如一系列能让问题更清晰的问题。if I were to do it, I'll say I have a wedding and I need to make three payments. One for $6,000, another one for an unknown amount, uh, and another one for the rest of the money. And also, uh, some of my costs are fixed. I know what I'm going to spend on a DJ and I don't know decorations, but I don't know how many people are going to attend the wedding. And that's the amount of money I'm going to pay per h per dinner per seat. So, I don't even know how to go about making sure that I can pay in time in time. On time, on time on time. Um, I'll say let me just cut the first bit. Cool. So if you want to run it for a problem that you actually care a little bit more about uh you will start there and all I'm doing is planning with the agent. Most of the harnesses today come with some notion that triages that you're trying to plan for a problem and it will try to come up with a reasonable idea some set of questions to give some more clarity.

这个框架本身会引导你思考应该提出什么问题，以及在动手实现之前，计划需要严谨到什么程度。它的目的不是立刻进入一味附和的模式，直接照着你提出的那些很不明确、甚至有些疯狂的想法去实现，而是帮助你层层深入，弄清楚它究竟需要做什么。这就是建立共同理解的原则。The harness itself is what guides the sort of questions that you will get asked and the sort of rigor that the plan needs to have before implementing anything. The idea is that it doesn't go immediately into a yes man mode and just implements whatever crazy idea you came up with that is very unclear but that helps you drill down and dig into what it needs to do. So that's the principle of getting a shared context.

主持人：你继续之前先问个小问题。坐在房间后面的人看得清吗？我是说文字大小。可以。好。不错，谢谢。嗯，不好意思，我得照着念，因为这像是在直播。好，就按我手头这份来。好。所以我也许可以继续在这里补充一些澄清信息。我先往后说，不过比如说：我没有存款。这笔钱得从我自己的口袋里出。我妈妈可能会给我大概五千。我知道参加婚礼的人均费用大约是150，花我还要花500买。我得买西装。我就是故意加入一堆复杂细节，还故意弄得乱七八糟，因为这个智能体的意义就在于，它不会放过你含糊其辞、没有提前提供信息这一点。它的全部目的就是给你设一道护栏，迫使你真正逐项梳理它制定计划所需要的每一条数据。Host: Quick one before you you continue. Is the size okay for the back of the room? The size of the text. Yes. Okay. Cool. Thank you. Yeah. Sorry, I'm reading because this is like live. Okay. From the one I have. Okay. So maybe I can keep adding clarifying information here. And I'm going to skip ahead, but something like I have no savings. It will come from my pocket. My mom may give me like 5k. Uh, I know the price per a person going to the wedding is around 150 and I'm paying 500 for the for the flowers. I need to buy my suit. I'm just adding a bunch of like complexity and being disorganized on purpose because the point of the agent is that it doesn't let go the fact that you're being vague um in not providing the information ahead of time. The whole point of this is that it's really guardrail for you to for to forcing yourself to actually go through um every single data point that it needs to come up with a plan.

我觉得接下来我们就把遇到的事情都做了，然后尽量继续往前推进。I think for the next one we just build whatever comes up to try and move on.

嘉宾：提出计划。现在，Guest: Propose the plan. Now,

主持人：有人正在用它解决你自己遇到的问题吗？Host: is anyone trying it out with a problem you have personally?

主持人：是的。好的，太棒了。你愿意给我们简单介绍一下吗？Host: Yes. Okay, cool. Do you want to tell us a little bit about it?

嘉宾：哦，有道理，有道理，有道理。不错。好吧，它问了很多问题，而我……Guest: Oh, fair, fair, fair. Nice. Okay. It's asking a lot of questions and I'm

主持人：不。对。我、我接受了这个计划。对。Host: No. Yeah. I I accepted the plan. Yeah.

主持人：那么，你能回到那个坐着的例子吗？Host: So, can you go back to the seated example?

来宾：是的，我可以。嗯， 久久精品Guest: Yes, I can. Um,

主持人：为了省去这里大量来回沟通的过程——这些沟通对你们任何人来说都没什么太大价值——有几点值得记住。第一，要有这种自我认知，并且强迫自己牢记：你的领域知识和智能体掌握的知识不会完全相同。你要尽可能找到两者像维恩图那样的重叠部分。不过，这需要大量来回沟通，而这往往也是最耗时的地方。如果你设计了一个很好的框架，它就能一步步引导你制定出很好的计划、把事情实施到位，并了解哪些组件重要、哪些不重要。要是你的框架基本上就是“我说什么你都答应，然后尽力去实现”，那结果就会参差不齐。Host: so to save you a lot of the back and forth back and forth here because it's not very high signal for any of you. There's a couple of takeaways. One is having that self-awareness and forcing the hardness to remember that your domain knowledge and whatever the agent knows is not going to be the same. You want to find that kind of like vin diagram superposition as much as you can. H but it takes a lot of back and forth and that's often where most of the time is spent. If you design a really good harness, it will guide you by the hand on coming up with a really good plan, implementing things well, uh, and knowing what components are important and which ones aren't. Uh, if your harness is basically like, yeah, say yes to everything I say and do your best to implement it, then you're going to get variable results.

所以，这是坐着的……So, this is the seated

从计划到决策树From plan to decision tree

例如，顶部的“计划已接受”基本上意味着，我已经来回沟通了大约十分钟什么的。呃，智能体觉得它已经足够了解我的问题，可以把问题分解成几个步骤。所以，你在那边的图表中看到的，就是它认为需要做出决策的所有步骤——可以说，这些步骤构成了一棵决策树，帮助它真正做出决策。example. The plan accepted thing at the top means essentially that I already went through the back and forth for around 10 minutes or whatever. Uh and the agent feel it knows enough about my problem to break it down in steps. So what you have in the diagram over there is the all the steps that it considers as decision points in a decision tree if you may to actually make a decision

嘉宾：如果可以的话，我想说，正如我们一开始所说的，稍微了解一下底层机制会很有帮助，这样能让你大致有个概念。当然，你也可以自己探索，或者借助你喜欢的智能体来探索。不过，我们谈到弗朗切斯科描述的这些内容时，也会涉及我们在 harness 中设置的指令，对吧？这里包含了负责进行访谈的智能体所遵循的指令，以及它最终必须产出的内容。所以，这就是一种原则的具体体现——这种原则不是构建在 harness 之外，而是构建在 harness 内部。我的意思是，它是内置于 harness 中的。Guest: and if I if I may um I think it's it's as we said in the beginning it's nice to look under the hood a bit just to give you an idea and you can of course explore this by yourself or with your preferred agents help. Um but when we are when we are talking about everything that Franchesco is describing um this also comes from the instructions that we have inside the harness right so this is where we have the instructions of the agent performing the interview and what it must end up with so this is the materialization of the principle built beyond the harness right like built inside the harness I mean

主持人：如果你想回到那个界面的话……希望随着我们继续看这个例子，这一切会变得更容易理解。我们还加了一些提升使用体验的小功能：如果你想知道这个设计精美的整个界面中，哪些元素对应我们正在讨论的原则，只需把鼠标悬停在底部，就会看到详细拆解。你看，我们进行规划、分享领域背景时，它得出了一个计划，并在右上方高亮显示；然后它还列出了所有节点，以及与其他决策之间的连接，还有我们接下来需要界定范围的内容。那我们继续？好。还有人正在用聊天功能进行规划吗？Host: if you want to go back to um to the UI. So this is hopefully going to start making more sense as we as we progress in the example. But we added a couple of quality of life things where if you want to know what elements of this whole UI that is so beautifully designed h makes sense for whatever uh principle we're discussing. You just need to hover down at the bottom and it will be broken down. So you see when we did the planning the planning the sharing the domain context it came up with a plan is highlighted at the top on the right and then it came up with all the nodes and the connections in between the other decisions and what we need to start scoping from here. So should we move on? Yes. Is anyone still doing any planning with the chat?

嘉宾：酷。Guest: Cool.

主持人：好的。嗯，我们这里有几件事。比如说，如果你想自己试着操作并浏览一下这个测试框架，可以在界面中进入第三步，就像左侧小图示里那样。Host: Okay. Yeah. So, we had a few things here. If you wanted to uh play around and peruse the uh harness yourself, you could for instance go on the step three in the UI like in the little diagram on the left hand side.

客人：哎呀，抱歉。我本来想展示……Guest: Oops, sorry. I was going to show

主持人：标签页太多。Host: too many tabs.

访客：你可以点击，比如步骤编号……Guest: You could click on like step no

主持人：是的，第三步。它会稍微说明一下信息的来源，以及模型为什么认为这是你需要做出决策的节点，还会说明它是否根据你在与它规划时使用的语言做出了任何假设。比如说，智能体——你是这么说的吗？我可以用这个来指示。Host: step three in the Yeah. And it will give you a little bit of a breakdown of where the information came from where why the model thinks this is the node you need to make a decision of uh including if it's making any assumptions based on the language you were using during your planning with it. So for instance the agent did you say this I could point with this

嘉宾：呃，如果你按住那个，对。Guest: uh if you press and hold that yeah

主持人：太棒了Host: awesome

嘉宾：不是人工智能Guest: not AI

主持人：所以，智能体整理出的计划里有不少内容是它自己提出来的，你可以开始稍微拆解一下，弄清楚哪些其实是我提供的，哪些是我认为自己知道的事实，哪些是智能体根据我的措辞推断出来的。你往上稍微滚一点，就在同一页上。对，再往上，再往上，再往上。对，当然。嗯，这里有过一些来回沟通，因为我用的措辞比较含糊，它就根据我在这里用了这个关键词，假设我指的是这个。再说一次，这是个相当明显的例子，但你可以点进去看看，四处浏览一下。如果我们回到演示文稿，里面有一些注释，因为这再次说明，一切都是为了把这些原则落到实处，让人觉得它是为建立信任而设计的。Host: so there's there's quite a few things in the plan h that the agent put together that were proposed by it so you can start dissecting a little bit and understanding like actually this came from me this is a fact that I think I know or this is something the agent inferred based on my language. If you scroll up a little bit just on the same Yeah. more more more more there. Yeah, sure. Uh here there was some back and forth because I was being vague with the language I was using and it made an assumption that because I was using this keyword here, I meant this. Again, this is a pretty obvious example, but you can play around with clicking through, seeing things around. If we go back to the presentation, there's some annotation because again, this is all about grounding the principles into something that feels like it was designed for building trust.

所以你会看到一些标注。再说一次，这绝不是世界上最好的 UI 设计，但你可以知道哪些内容是你提出的，哪些是代理提出的，哪些仍然悬而未决、需要你做决定；你也能看出计划是如何制定出来的。你还可以尝试另一种做法：回到第二步，呃，实际上，如果我们点击那里的节点，对。然后你可以告诉代理——你已经……抱歉，你可以直接告诉代理，比如：“其实你对这一步做了一些不正确的假设”，然后再来回沟通几轮。呃，我还是会做举手调查，不过为了强化效果、提高参与度，呃，有谁参加过一次聊天会话，持续时间超过……So you have some notation. Again, not the best UI design in the world, but knowing if things came from you, from the agent, or is it still an open point and you need to make a decision on and you can tell how the plan has been built. You have another thing you can try. You can go back into step two uh and actually tell the agent if we click on the node there. Yeah. And you can actually tell the agent after you have sorry uh you can just tell the agent like well actually you're making some assumptions about this step that are not correct and then go a little bit back and forth. Erh, I will still do the show of hands thing, but because of reinforcement and engagement, uh, who has been in a chat session with over

为什么冗长的线性聊天会崩溃Why long linear chats fall apart

比如来回交流远不止10轮，至少来回10次，到某个时候你会说：“让我从头开始吧，这完全说不通。我们之前讨论过的一些内容，好像已经完全不见了。”有多少人经常有这种感觉？是的。这是我们如今使用的框架设计所带来的后果：它们是为线性讨论而设计的，并且会把所有内容都保留在上下文中，作为一场大对话。而我们这里的设计思路则是把内容拆分开来，这样你就可以讨论各个独立的元素，而不至于让整个事情变得更复杂。like well over 10 turns, so back and forth and back and forth at least 10 times where at some point you said, let let me start from the beginning like this is not making any sense. Some stuff we already discussed like it's well gone. Who has had that feeling on a regular basis? Yeah. So, that is a consequence of the design of the harnesses we have today where they're designed for a linear discussion and to keep everything in the context as one big conversation. Whereas here the way we designed it is supposed to break it down so you can discuss individual elements without necessarily making things you know more complicated in the grander scheme of things.

嘉宾：是的。Guest: Yeah.

主持人：嗯，好。那我先让你稍微玩一玩，不过我确实想再深入一点。所以我会……Host: Erh cool. So I will let you play around with it a little bit but I do want to maybe go a little bit further. So I will

原则 2：一致性（数学运算应放在代码中）Principle 2: Consistency (math belongs in code)

介绍一致性原则，这建立在共享领域之上。呃，因为你已经有了计划，而且智能体很擅长理解这些内容。你计划中的这些部分，只是一段代码，或者一张计算电子表格；它总会给出一致的结果，是你整个计划中一个确定性的元素。呃，基于这一点，你需要编写代码，实现某种每次都能给出相同结果的东西。呃，类似举手调查的环节：有人试过在聊天里做预算或进行财务计算吗？效果怎么样？你对结果有信心吗？你……嗯，有一点。introduce the consistency principle and this build on the shared domain erh because you already have a plan and the agent is very capable to understand well these pieces of your plan are just a piece of code or just a calculation spreadsheet like it will always give you a consistent result is a deterministic element of your whole plan. Erh and based on that you need to implement code. You need to implement something that will give you the same result every time. Erh similar show of hands thing. Has anyone tried to do like budgeting or any financial calculation in the chat and how did that go? Like are you confident on the results? You Yeah, a little bit.

客人：体温可能还算挺好吧，嗯，也不太好。挺糟的。Guest: Temperature maybe like very well um not too well. pretty bad.

主持人：好的。Host: Okay.

嘉宾：所以，好吧。我很想知道有没有人愿意分享自己的经历，说说他们看到了什么，以及为什么觉得会发生这种情况。不过有几次，我觉得自己已经提供了所有必要的信息。那个智能体给了我一个公式，甚至还把它排版成公式的样子。你知道，就是那种斜体公式，看起来特别漂亮。我当时还想，他看起来挺懂行的。结果它却说“我不是计算器”，而且我知道那个结果是错的。所以一开始我就想，算了，重来吧。还有人有其他类似的例子吗？Guest: So, so all right. So, I would love to know if anyone wants to volunteer their story, but what they saw and why they think it happened. But to me, a couple of times I thought I was giving all the necessary information. The the agent gave me back a formula. It was even formatted as a formula. You know, the kind of like italics formula thing that looks really nice. I was like knows what he's talking about. And then the result was like I'm not a calculator and I know that result is wrong. So from the get-go I was like nah start from scratch. H anyone has any other like examples

主持人：一致性，对吧？所以很多时候结果看起来不错、也合理，但你再运行一次相同的操作，结果就会有所不同。Host: consistency right? So a lot of times the results look good reasonable but you run the same thing again that's something.

主持人：是的。所以这基本上就是这个原理的灵感来源。抱歉，你想分享什么？Host: Yeah. So that's basically the inspiration on the principle. Sorry, what do you want to share?

嘉宾：是的。所以事情发展到每次它可能都会……Guest: Yeah. So it goes to the point that every time it might

主持人：是的，呃，这是一个很好的例子，说明了为什么我们要讨论构建一个 harness，而不只是把这些原则作为一套技能或指令，接入现有的 harness，比如 Codex、Claude Code，或者其他任何工具。因为当我们采用非确定性的方式，也就是给智能体一堆听起来正确的指令，却仍然依赖智能体自己去执行时，我们其实是在说：“呃，请别犯错，务必做对。”这就是关键所在。我们依赖着许多事情在后台发生，所以每次得到的结果基本上都是非确定性的。因此，我认为构建 harness 的价值就在于——再说一次，这里稍微深入到了底层，也涉及我们在 Monzo 应用的一些原则——把非确定性与我们希望加以控制的某些确定性行为结合起来。Host: Yeah. Uh and and that's a great a great um example to say part of the benefit why we are talking about building a harness and not just plugging these principles in a set of skills or instructions to an existing harness such as like codeex cloud code or anything else like when we when we do it the non-deterministic way which is to give the agent a bunch of instructions that um say the right things but still rely on the agent agent. Uh, make no mistakes, please do the right thing. Um, that's that's the point. We're relying on a bunch of things happening in the background and basically getting a nondeterministic result every time. So, I think the value here from building the harness again, this is looking a bit into the under the hood and like from principles we we applied at at Monzo as well is mixing the non-determinism with some deterministic behavior that we want to control.

呃，当……当相关时。um when when relevant.

嘉宾：我想你提到过，对于这个演示来说，这套框架的很多部分都相当……我理解得对吗？对于生产环境里的东西，比如说，你在聊天中自己做计算，和让模型用代码为你编写计算程序，这两者有什么区别？我有几个关于做预测的个人例子：你可以试着手动预测，或者做一个包含很多复杂计算的电子表格，这样你可能有几项能做对，也能得到一个还不错的预测。你也可以直接从可信的代码仓库中获取一个库，里面已经集成了所有功能。你还可以用更模块化的方式编写代码，并进行充分的单元测试和端到端测试，采用我们所熟悉的各种良好软件工程实践，确保那段代码能给你带来可靠的结果。Guest: I think you mentioned that for this demo a lot of this harness was quite I get this point right for production stuff like what's the difference between you making calculations in in chat versus you asking the model to code up calculations for you in code. So I have a couple of personal examples on doing forecast where you can try and do forecast yourself by hand or come up with a spreadsheet with a lot of fancy um calculations and you will likely get a few things right and get a decent forecast. You can also just get a library from a trusted uh repo and that has everything baked in. You can also write some code in a more modular fashion that is well unit tested, you're doing end to end testing, you're doing all the like good software um engineering practices that we know how to do to ensure that that piece of code is going to give you a good result.

嗯，所以我觉得不同之处……哦，抱歉。差别有一点在于，如果你只是把如何实现这些事情、什么时候该在代码里完成、什么时候该单独构建一些东西这样的决定都交给代理，那么结果就会很碰运气，时好时坏。Um, so I think the different Oh, sorry. The difference lies a little bit on the fact that if you're just leaving it to the agent to decide how to implement things and when they need to do it in code versus when it like you should just build something on the side, you're going to have a little bit of a big miss ratio like hit or miss.

主持人：是的，是的。我想我想问的是，这是否就是所谓的“没有采用良好的工程实践”的原因。这部分我明白。你们有没有经过测试的产品？Host: Yeah. Yeah. I I guess what I'm asking is if this is why quoted no good engineering practices were applied. I get this part. Do you have like a product that that was tested?

嘉宾：我、我再来做一下那个，呃，广告。呃，明天早上我们会有一场分组讨论，聊聊我们在 Monzo 构建的、应用了其中一些原则的东西。呃，不过，是的，我觉得我们说 harness 的编码方式时，背后指的更多是实现细节，比如抽象、方法以及实际代码；但无论如何，融合确定性与非确定性的原则都是一样的，不知道这样说你能不能明白。Guest: We we I'm going to do the the uh ad again. Uh tomorrow we're going to have a breakout session in the morning where we're going to talk about uh what we build at Monzo that applies some of these principles. uh but yeah I think the what is behind when we say like the harness being voded I think it's more about the implementation detail of like abstractions methods the actual code uh but the principles of mixing the determinism and the non-determinism are the same regardless if that makes sense

主持人：谢谢Host: thank you

嘉宾：呃，还有一件事，另一件事也说明这是一个值得考虑的好原则。过去大概一年左右，大多数智能体都开始更积极地调用工具，并构建自己的工具，以更确定的方式解决计算问题。说实话，智能体也在不断变得更好，对吧？所以我们完全可以等一等，到时候所有问题都会迎刃而解。我对这种想法有一个担忧：它会让智能体在运行过程中即时创建东西。所以，没错，你可能会在某个时点针对某个问题得到正确的代码，但也许等你运行到第 100 次时，它却又发明了一段本不需要重新发明的新代码。Guest: uh and another thing another thing that signals that this is a a good principle to consider on over the last what year or so most of the agents have started to do more aggressive tool calling and building their own tools to solve calculations in a more deterministic way and honestly agents are just getting better right so we could just wait and then all the problems are solved the one challenge I have with that mindset is that h it will create stuff on the on the go like in flight so Yes, you may get the right code for the right problem at an at a point in time, but maybe the time number 100, you run it, turns out is inventing a new piece of code that didn't need to be reinvented.

呃，我还能分享一些与工作无关的例子，比如做类似的事情。我每个月都会做一次预算结算，我不想说特别严格，但也算半严格吧。我会把账单上的每一笔都分类。一开始我把分类方式、汇总方式，以及退款、支出等其他所有事情都交给智能体处理。后来有一次我就说：“不，不，不，这得写成脚本。”因为每次得到的结果都略有不同，而且我花了越来越多的钱和时间，跟智能体来回折腾，只为让它给出我想要的结果——明明它已经有三次做对了，第四次却没做对。所以我不想再重复这个过程了。你得确保计算过程只编写一次，然后每次需要时都运行它。模型能够选择工具；你只需要确保你的运行框架足够严格，不让它做任何算术运算，也不让它仅仅通过解析 token 来尝试解决这些问题。uh and other examples non-workrelated that I can share err are around like doing exactly that like I have a I don't want to say strict but semi-strict close of budget every month and I categorize everything in my statements I left it to the agent to decide how to do the categorization and how to do the aggregates and everything else and the refunds and expenses and at some point I was like no no no this needs to be a script because I'm gettinglight slightly different results every time and I'm spending more money more money and time uh wrangling with the agent to get it to give me the results that I wanted that it got right three times already but didn't get it right in the fourth uh so I don't want to repeat it again but make sure that we run the calculations you write the calculation once and then you run it every time you need it um and the model is able to pick the tools is you just need to make sure that your hardness makes it very strict that it's not going to do any arithmetic or try to solve these problems by just parsing tokens.

那么，如果你不确保计算以及所有确定性的操作都在代码中运行，会发生什么呢？它们就会深深嵌在模型的内部。一旦开始新会话，这些内容就会消失；每个新问题或新会话都要重新生成代码，结果就会变得一团糟。同一个问题得到不同答案，这又回到了非确定性的问题：你是在要求一台随机性机器给出一致的输出，所以很可能得不到一致结果。 那么，这里有一些可以通过计算得出的例子，对吧？我敢肯定，如果你问十家不同的金融服务公司，它们处理这些事情的方式会略有不同，但总体公式和思路是一样的。因此，你完全可以把它整理成一个库，编写一次，之后再更新；这些都不是什么新东西。Um so what can happen if you don't make sure that your calculations and everything that is deterministic runs in code they're in the thickest in the model's head. So it's gone once you start new session the new code for every new question and or new session and it becomes a mess and same question different answer non-determination again you're asking the stoastic machine to produce a consistent output so you're likely not going to get it. Um, so here's some examples of things that have a way to be calculated, right? Like I'm sure if you ask 10 different financial services companies, they will have a slightly different way of doing these things, but the general formula, the general idea is the same. So you could realistically export a library, write it once, update it, and these things are not new.

他们看到的，比如软件方面的良好实践。那再演示一次，如果我们只是想转到那个“seated”示例……你能把这个高亮显示出来吗？好的。they saw from like software good practices. So demo again if we just want to go to the seated example h in the can you highlight this? Okay.

主持人：所以，再次说明，只用这个 h，你就能看到决策流程中的各个节点——嗯，其中有些需要用代码来实现。我认为，最简单的判断方法之一是：如果涉及计算，那很可能就得用代码实现，对吧？此外，还要进行一些测试，比如单元测试、黄金样例等。构建这样的测试框架时，你应该通过确保自己关心的、有关其实现的信息都清晰可见，来建立对智能体的信任。另一个现实中的例子是提供不同版本的代码。你的代码更有可能经过版本管理。如果代码版本有任何变化，而你又试图运行同样的流程，智能体应该能够识别出来，并把相关信息呈现给你，告诉你：“实际上，尽管函数没变，你可能不该相信这个结果，因为现在的计算方式 h 已经不同了。”Host: So again just using this h you can see how the nodes in your decision making process um some of them needed to be implemented as code and I think the one of the easiest way to check is if it's a calculation likely it has to be implemented as code right uh as well as some testing unit testing golden examples etc. the you should build trust with your agent when you're building a harness like this by making sure that the information you care about about its implementation it's in front of you. Another example of how this has looked in the wild is giving you versions of the code. Your your code is more likely versioned. If anything change with the version of the code and you're trying to run the same process, the agent should be able to pick it up and surface that information to you and say actually this code change you probably even though the function is the same you probably shouldn't trust uh what's coming out of this because the whatever calculation h is different now.

所以，当你使用这个 harness 并接受计划后，就会构建出这样一张很棒的图。你会在顶部看到类似“要我构建代码吗？”的提示，然后它就会替你完成。再说一次，整个过程都有引导。这就是 harness 的作用：引导各项工作。呃，有人在构建计划中的某个步骤吗？So as you're working with the with the harness and you accepted the plan, you have a nice graph like this being built, you will get something at the top saying, do you want me to build the code? H and it will do it for you. And again, it's all guided. That's the point of a hardness that it guides things. Erh, is anyone building any of the steps on their plan?

主持人：这样可以吗？Host: Is that okay?

客人：嗯。嗯。不错。Guest: Yeah. Yeah. Cool.

主持人：我切换到专注模式。Host: I switch to focus.

嘉宾：好的。嗯。Guest: Okay. Yeah.

主持人：有道理。Host: Fair enough.

嘉宾：有道理。有道理。Guest: Fair. Fair.

主持人：好。如果你沿用婚礼这个例子，可以尝试几种操作。点击第一步，然后点击“代码”，再点击“测试”，现在运行测试。测试就在这里。Host: Cool. So if you're following on the wedding example, you can try a few things. Click on step one, then code, and then the test and run the tests now. Tests here.

嘉宾：是的。Guest: Yeah.

主持人：95。Host: 95.

主持人：不好意思，现在几点了？Host: Sorry, what time is it?

嘉宾：9:54。好，很好。嗯，不过是的，这里的原则——回过头来把所有内容串起来——就是你不应该依赖自己记得去运行测试。你也不应该依赖自己记得，嘿，这听起来像是我每次都能创建一些代码来解决它。代理应该引导你完成整个过程。这也是我们在构建它时融入的原则之一。顺便说一下，我想我们告诉过你，仓库里有个额外的小文件，上面写着“构建你自己的”之类的内容。Guest: 9:54. Okay, good. Um, but yeah, the the the principle here and just going back to try and tie everything up together, you shouldn't rely on you remembering to run the test. You shouldn't rely on you remembering that, hey, this this sounds like I can h create some code to solve it every time. The agent should guide you through it. And that's one of the principles we baked into building this. Uh, as an aside, I think we were telling you, uh, there's a little extra file in the repo that says build yours or something.

主持人：嗯，它基本上按顺序内置了所有原则，并会指导你打造出类似的硬度。[嗤笑]Host: Um, it essentially has all the principles baked in in order and it will guide you on building a [snorts] similar hardness.

嘉宾：这是自己组装的。Guest: It's built your own.

主持人：好了。Host: There you go.

嘉宾：是的。所以，它会指导你构建一些东西，让它的感觉类似于我们向你展示的这个 harness：它优先考虑建立信任的机制，而不是只给你一个看起来正确的东西。所以你可以试试这个。[哼笑]测试应该能通过。你会发现，没有任何“思考”“浏览”之类的步骤——这些都是智能体在加载东西时会写出来的动词。对，因为这些只是需要在你的环境中运行的测试。别把它搞得过于复杂。我说这话还挺好笑的，不过，如果 harness 的所有组件都能像应该的那样协同工作，而不是寄希望于智能体靠什么梦想和魔法胶水把一切编排起来，那事情就简单多了。Guest: Yeah. So yeah, it will guide you on building something that feels similar to the harness we're showing you that prioritizes trust building mechanisms over just giving you something that looks correct. So you can try this. The [snorts] test should be passing. You will notice that there was no thinking, perusing, all the verbs that agents put when they're loading something. Yeah. because these are just tests that need to run in your environment. Don't overengineer it. Funny that I'm saying that, but it's easier if all the components of your harness just work together as they should and not based on like hopes and dreams and magic glue from an agent orchestrating things together.

主持人：我想我们有个问题。呃，所以如果我们……是的，我的意思是，我们可以这么做，但我不太想搞一场阅读会。基本上，这一切都关乎弄清楚你真正关心什么，以及一个正常运行的 harness 需要满足哪些功能要求。我会在谈话中穿插一些相关内容。再说一次，因为涉及知识产权，我们得稍微避免透露太多。不过，你要确保在构建组成 harness 行为的各个部分时，每个步骤的目标都有明确意图。其中一个最初组件的目的，是尽可能多地收集信息，以便制定计划。这一点也写在这里了。所以，这基本上就是关于如何编写指令的说明。就像宜家的组装指南。嗯，它应该能很好地引导你了解哪些事情值得关注，不过具体如何实现，你可以按自己的偏好来选择。Host: I think we had a question. Erh, so if we Yeah, I mean we can do that, but I'd rather not do a reading session. Basically, it it all goes about h figuring out what you actually care about and what are the functional requirements for a functioning harness. H I'm sprinkling some of that through the conversation. Again, need to avoid oversharing a little bit because of IP. Uh but you want to make sure that the objectives of each of the steps when you're building the parts that will make up the behavior of your hardness have an intent. The intent of one of the first components is gathering as much information as possible to come up with a plan. And that is written in here as well. So this is basically instructions for instructions. It's the IKEA guide. Um it is supposed to guide you very nicely on these are the things you should care about but but then you can pick uh your preference on how they get implemented.

嘉宾：问题。那么，这和规范驱动开发有什么不同？比如说，规范驱动开发是不是也像这样，是一个线性过程，从你提出所有问题开始，从想法一路推进到最后？它和这个有什么不同？这个和那个有什么区别？Guest: Question. So how is this different from let's say the specdriven development expected something like that which is also linear from starting from all the questions from you starting from idea all the way to the end and how is it different to this one is different to that

主持人：你介意我谈谈这一点吗？嗯，我想提两件我觉得和这里相关的事。嗯，我觉得第一件是，当我们针对某一类特定用例构建一个 harness 时，这和编码助手领域里的智能体有些不同，因为在编码辅助场景中，代码通常只是达成目的的手段，对吧？我们是在构建一个能够执行某种操作的东西。它可能是一个产品、一个应用，或者其他什么。嗯，在这个案例中，智能体是在帮助我们分析问题、解决问题，并一步步引导我们处理问题。所以，像是弗朗切斯科婚礼这样的庞大规格说明，内容会变化、会不断演变；如果我们回头查看这份不断变化的规格说明，就又要依赖智能体的非确定性来解读它，而这可能会非常、非常、非常稳健，并正确地完成这件事。Host: do you mind if I come to that um I I want to mention two things that I think are are relevant here um I think the first one is when we're building a harness for a specific type of use case it's a bit different from the coding assistant world of agents because with the coding assistance we usually the code is a means to an end right like we we're building something that performs an action. It's a product, is a an app, whatever that is. Um, in this case, the agent is helping us like analyze, solve a problem and work us through that. So the the gigantic spec for this like let's say for Franchesco's wedding things will change like it will evolve it will change and then if we go back to the everchanging spec we are relying again on the determin on the non-determinism of the agent to interpret that that can be very very very um robust and perform this the the correct thing.

所以，harness 基本上就是把规范拆解成一些带有明确主张的部分，如果这样说你能理解的话。嗯，我觉得第二点可以补充这一点，而且实际上也是我想结合我们在数据方面的经验来讨论的。我不知道你们有多少人从事数据工程、分析或类似工作。我觉得，我们从这套原则中带来的另一点，是这样一个想法：指标定义和维度定义总是在变化。就我们的经验而言，我们已经遇到过很多很多次这种情况了，我相信你们当中很多人也一样：大家会讨论“这个数字对吗？”我不知道。听起来好像是对的，但我们永远无法确定；如果我们永远无法确定，那这个数字到底对不对？嗯，我觉得这也来自同一种视角：我们信任这个智能体的程度，应该能达到我们信任仪表盘、指标或可视化图表的程度。So the harness essentially breaks down what would be the spec in opinionated sections if that makes sense. Um and I think the second thing to complement with that and is actually something I wanted to to like ground on our experience from data. I don't know how many of you work in data engineering analytics or anything like that. I think something else that we bring from this word in these principles is the idea that um metric definitions and dimension definitions always change like in our experience we've like [snorts] had many many many times and I'm sure many of you as well um that conversation were is this number correct? I don't know. It sounds like it is, but we're never sure and then if we're never sure, is the number correct? Um, I think this is also coming from that perspective is that to the point where we trust the agent as well as we trust a dashboard or a metric or a visualization.

嗯，我们已经与它达成了一致，而 harness 可以继续引导我们，让这种一致性随着时间推移持续保持。但前提会变，源数据会变，语境也会变。让 harness 能够适应这些变化，就意味着它能意识到变化是必然会发生的，而不必每次都重新提示一遍，再回到原来的流程。这样说得通吗？Um, we've reached alignment with it and the harness can continue guiding us into that alignment continuing to be true over time. But premises will change, source data will change, context will change. And then having the harness able to adapt to that is like recognizing that this change will be a thing um rather than having to every time pro prompt again and go back to the process. Does that make sense?

嘉宾：是的。Guest: Yep.

主持人：关于这一点的后续问题：有人对这种看法有异议吗？因为最可行的做法是，你在开发某样东西，尽管它会随着不同迭代而改变；AI 智能体也可以参与到这个开发流程中。Host: Follow question on that. any complnt this perspective because most most workable thing you are developing something even though it's subject to change through different iterations you are developing something with an AI agent can be part of this flow as well

嘉宾：绝对要用那个，呃，这部分，这个测试框架来做那个……它Guest: absolutely use that uh this part this harness to do that it

主持人：当然，而且从底层来看，本质上也是这么回事。所以，当我们创建……嗯，如果你从底层查看 my 文件夹，就会看到所有保存下来的内容。你可以把它看作是支撑这个 harness 的规范。所以，完全一样的原则也适用于这里：在规范驱动开发中，尤其是用于编码智能体时，通常会有一个东西作为正在构建的对象的定义来源。在这个案例中也是一样。区别在于，我们通过一些带有明确主张的工作流来呈现它，而不是用一份单独的……嗯，文档来呈现，不知道这样说你能不能明白。好，太好了。这是你可以尝试的另一件事。嗯，你可以试着改变计算方式。呃，而 harness 很可能会对此提出异议。Host: absolutely and this like under the hood this is essentially what's happening as well. So when we have when we're creating um the if you if you look under the hood in the my folder, you're going to have everything that is saved. So you can think of this as the spec that's behind the harness. So absolutely like the same principle applies here in the in the specdriven development for again coding agents is usually used. you have something that is the source of um definition for the thing being built. In this case, it's the same thing. The difference is that we surface that through opinionated like uh workflows rather than a single um looking document if that makes sense. All right, cool. So, that's another thing you can try out. um you can try and change how a calculation works. Uh and you're likely going to get some push back from the harness.

再重申一下，测试框架的作用是引导你，并迫使你做正确的事。这些原则则能帮助你每次都做出正确的决定。所以，再说一次，如果你试图把乘法改成聚合，可能只是快速修改一下，保存文件，成功——你破坏性地附加了分析，但当你……Just to reiterate, the point of the harness is to guide you and force you to do the right thing. And the principles are here to help you make the right decision every time. So again if you go try and change uh multiplication for a an aggregation you may just go make the quick edit save the file success you sabot attach the analysis but when you

原则 3：证据（拿出凭据）Principle 3: Evidence (show the receipts)

运行测试时，你无法真正得出自己想做的分析结论，因为硬度会根据它自身的定义和构建方式来阻止你。嗯，我们该接着谈证据了吗？呃，证据就是我之前说的那些凭据：要有凭据，并且能够证明，嗯，没错，这就是它的来源。之所以正确，是因为其中存在因果关系。它建立在第二条原则之上：既然你的代码经过了测试，其他一切也都经过了测试，嗯，你就有一条可以往回追溯的线索。这样，你就有办法为自己做出的决定提供佐证，也掌握了证据。这条原则之所以重要，是因为原则需要你去解读，对吧？我不想留给你一个非常模糊、空泛的想法，比如“哦，对，重视证据，太棒了”，但证据究竟有哪些方面是你需要注意的呢？回到我们数据专业人士的工作上，这件事每周都会给我们带来麻烦。run the test you won't be able to actually reach any conclusion of the analysis you wanted to do h because the hardness is going to stop you by its own like definitions and the way it was built. Um, should we move on evidence? Er, evidence is the whole thing I was saying before about the receipts, having the receipts and being able to show that well, yeah, this is where this came from. This is correct because of in doing that causality relationship. So it builds on the second principle as since you have code that is tested as well as everything else h you have a trail that you can follow backwards. So you have a way to substantiate the decisions you're making and you have the evidence and it's important in this principle because a principle is something you need to interpret right and I don't want to leave you with a very vague ambitional thing like oh yeah do evidence it's awesome but what is what about evidence is something that you need to take care of going back to being data professionals this is something that bites us on a weekly basis.

嗯，并不是所有信息都一样。如果你真的想展示一份分析，或者要为公司评估一项潜在投资并进行财务分析，就需要说明你的数字从何而来，并能够证明你的决策是基于你掌握的证据做出的。嗯，既然我们可以说明每件事的来源，就有一些错误需要避免，比如把所有东西都当成事实。比如说，你给我的这个数字——嗯，我想不起来了，还是拿我的婚礼来说吧：受邀人数是个事实。结果我的未婚伴侣又邀请了三个人，却没告诉我。现在我的整个模型都错了。嗯，我尽量把例子说得非常简单，不过我想你可以想象，财务分析中某个假设有误，而你又不知道它是从哪里来的，可能会造成多么严重的后果。Erh, not all information is the same. If you really want to present a piece of analysis or you went to do some financial analysis on a potential investment for your firm, you need to show where your numbers came from and be able to attribute your decision to the body of evidence you had. Erh so since we can show where everything came from uh there's some mistakes we want to avoid where everything is treated as a fact. So oh yeah this number you gave me for the number of um I don't know going back to my wedding the number of people invited is a fact. Turns out my fiance invited three more people didn't tell me. Now my whole model is wrong. uh and I'm trying to reduce it to a very simple example but I think you can imagine how having a wrong assumption somewhere and not knowing where it came from can be fatal for any financial analysis.

嗯，比如说，在构建智能体系统时，人们很容易想当然，或者把某些东西当作粗略估算。呃，模型无法分辨哪些是确凿事实，哪些只是估算。所以你需要通过设计你的运行框架来强制区分。呃，另一个很容易迅速出问题的例子是：想象一下，有个数据人员、分析师之类的人来找你，说：“不不不，结果就是这样。相信我，我算过了。”你会相信吗？嗯，我看到有人觉得不会。我想这里没人会想查看单个数字背后的所有日志和计算过程。但只要能追溯这项决策是怎么得出的，就能大大增强我们人类的信心。呃，这里还有一些不同的例子。Um so let's say in the world of a gentic building it's very easy to assume something or treat something as a gueststimate. erh the model can't tell what is a hard truth and what is an estimation. So you need to enforce that through the design of your hardness. Erh, and another example where this kind of like falls fall into pieces really quickly is imagine you have a data person, an analyst, something coming to you and being like, "No, no, no. Here's the result. Trust me, I run the numbers." Do you trust that as a Yeah, I see some like no. I don't think anyone here will want to see all the logs and the math behind a single number. But just having the trace to figure out, okay, I can follow back this decision increased increases confidence for us humans quite a bit. Erh, so we have some like different examples here.

呃，聊天中的非结构化想法需要标记为一种解读。智能体的推理，或者从记忆中提取的信息，都需要视为一种假设。而如果我提交一份银行对账单，那还有什么可讨论的呢？那就是事实：我花了多少钱，等等。再回到数据从业者的角度，数据标注以及确保你拥有经过整理的数据集——无论是用于训练某种模型，还是运行任何流程——都至关重要。基本上这也是同一回事：你只需要为收集到的证据赋予恰当的重要性级别，并确保尽可能多地收集证据。嗯，这一点挺有意思的，证据也体现在代码里。所以，问题不只是这个数字是从哪儿来的，还有你是怎么得出这个数字的？Erh, unstructured thoughts in a chat needs to be labeled as an interpretation. the agent reasoning or pulling information from memory needs to be treated as an assumption and then if I submit a bank statement what is there to discuss like that is a fact that's how much I spent etc if I take like go back again to being a data person data labeling and making sure that you have a curated data set for either training something or running any process is paramount and this is basically the same thing you just need to assign the right level of importance to the evidence you're gathering and make sure that you gather as much as as much of it as you can. Um, this is a really funny one, but evidence also goes into the code. So, it's not just where do you get this number from, but how did you get this number?

而且我有——我有一个轶事，有人当时在使用主文件。And I had I have an anecdote where someone was using the master file.

“主文件”恐怖故事The "master file" horror story

每家公司都有一个主文件，大家都会把它收藏在共享盘之类的地方。然后有人往里面填了一些数字，一切看起来都没问题。主文件里对合理的数值范围做了一些假设。那个人并不知道文件已经变了。他们没做错任何事，只是不知道文件变了。呃，然后结果就完全错了。你用自己的测试框架构建的确定性代码也会发生同样的事。如果你没有办法追踪代码中的改动，就会让自己暴露在流程变更带来的盲点风险之中。输入是一样的。呃，稍微往后跳一点。所以，沿着这个例子说，你能打开其中任何一个吗？In every company there's a master file and is favorited in the share drive and whatever. And that person puts some numbers in there and everything looked fine. The master file had some assumptions of what range of values were reasonable. That person didn't know because the file changed. They didn't do anything wrong. They just didn't know the file changed. Uh and then the results were entirely wrong. And the same will happen with deterministic code you build with your harness. If you don't have a way to having a traceability even for changes you make in your code, you're leaving yourself uh vulnerable for blind spots on the process change. The inputs were the same. Erh skipping ahead a bit. So if we're following from the example, can you open any of them?

客人：是的。Guest: Yes.

主持人：证据已经高亮显示了。呃，同样，你可以通过这个示例看到自己关注的区域，只要点击底部，然后查看哪些内容被高亮显示即可。不过，这展示了信息流是如何传递的，以及哪些部分是由助手提出的——你可以查看凭据，追溯这些内容的来源。看看业内其他金融分析师的实现方式，你会发现，追溯能力可以一直深入到数据集层面：比如你可以知道是哪个连接器或哪家数据供应商提供了这些信息，告诉你苹果在当天某个时点的股价是多少。如果你是在做股票交易或投资相关的事情，这一点就极其重要。除非有人有问题，否则我觉得我们没必要在这里多花时间。Host: The evidence is highlighted. Erh, and again, you can see like the areas that you care about through the example by just like clicking on the bottom and then seeing the bits that get highlighted. But this is how the flow of information goes through and what parts are proposed by an assistant like you can show the receipts of where where this came from. Looking at other implementations of financial analysts in the industry you get traceability down to the data set like you know which uh connector or which supplier of data gave you the information that whatever Apple had this price at this point of the day. if you're doing something about trading or investing in stocks and shares. So, this is extremely important. I don't think we need to spend any more time here unless anyone has any questions.

哦，好的，很棒。这里有几件事你可以试试。嗯，你可以给智能体出一道小难题。只要在聊天框里输入内容，你就会看到一些证据条目，或者输入到后续流程中的某些内容是如何变化的；它也能追溯到这些变化是因为你在聊天中更正了某个事实。另一个例子是，如果你想做出更改，可以点击发生变化的底层数据，这样应该就能一路追溯到它的来源，以及它为什么会影响后续计算。Oh, good. Cool. So, here's a few things you can try. Um, so you can add a little bit of a a curveball to the agent. Just typing in the chat and you will see how some of those evidence lines or some of the inputs that go into the rest of your process change and it it's able to trace that it came from you correcting some fact in the chat. And another example is h if you want to go and make a change and click on the underlying figure that changed h it should show you all the way back where this came from and why it matters for the rest of your calculation.

嘉宾：我接下来要说的话可能会让人觉得有些重复。嗯，不过再说一次，这个 harness 是一个示例，展示了如何应用这些原则。但我相信大家都在思考自己的使用场景、自己的公司、自己正在构建或使用的智能体，以及该如何把这些点连起来、把这些线接起来。我认为，我们可以把这个例子放在心里：当我们的用户，也就是我们正在构建的智能体的客户使用它时，他们对你的产品、你的智能体以及它输出结果的信任，将推动使用、普及、改进等等，并通过我们提到的那些循环不断发展，比如 tracing、评估等等。所以再说一次，我认为这里的核心是：构建 harness 时，要确保它能让使用你产品的客户和用户信任智能体给出的结果。Guest: I'm going to say something that might feel repetitive. Um but again we are we are this this harness is an example of how to apply the principles but I I'm sure everyone is thinking about your use cases your companies the agency you're building the agency you're using and how like to to to connect the dots and to to I don't know put the wires together. I think this is an an example we can think about when our users our and customers of the agents that we're building are using it and again the trust that they have in your product in your agent and in the output is what will drive usage adoption improvement and so on through the loops that we mentioned through tracing evolves and so on. So again, I think um what's underlying here is when you're building a a harness, make sure that you can um make this happen to uh your customers that are using your users that are using so that they trust the results that your agent is giving them.

再说一次，可能有点重复，但我们一路上得确保把所有东西都串起来。继续之前，有没有人能举个例子，说说有证据链的话，什么时候能帮你省去很多麻烦？我看到有人点头，像是在痛苦地表示认同。不过如果有什么想分享的，我很乐意听。要是没有，我们就继续。基本上每次你想做点什么，都会遇到这种情况。我最近就在尝试解决这个问题。我很喜欢能把依据和所有相关材料都拿出来给人看的想法。这基本上是我每次和智能体对话时都会遇到的事，不管我是在写代码，还是在做演示文稿之类的。我会问：“这个数字是从哪儿来的？”它就说：“嗯，我们之前聊过这个，诸如此类……”我就想：“要是能直接把具体资料给我看看就好了。”你举演示文稿为例挺好的，因为我自己也遇到过类似的情况：差不多一个半月以来，我一直在尽可能地避免调整 Google 幻灯片里的文本框大小。Again, maybe repetitive, but like making sure we uh draw get everything together as we go. Before we move on, anyone has any example of a time where having the evidence trail would have saved you a lot of pain? I see nodding like painful acknowledgement, but if anything you want to share, more than happy to to hear you out. Otherwise, we can carry on. basically every single time you try to do something. And so I'm playing with this. I really like the idea of being able to show the receipts and everything. That's basically every single conversation I have with an agent, whether I'm coding or like working on a deck or something. I'm like, "Where did this number come from?" Like, "Well, we talked about this blah blah blah blah blah." I'm like, "It would be nice if I could just be able to show me specifically like the paper I like the that you use a presentation as an example because I had an example myself where I was trying as hard as I could to not have to resize text boxes in Google Slides for a good like month and a half.

所以我当时特别想弄出一个能帮我做幻灯片的工具。我有一套很不错的工作流，它会深入分析问题，找出所有相关文档，并帮我提炼出我想向团队展示的要点。结果在我们交流的过程中，我一直没意识到：当我要求修改并给出一个示例时，模型会死死盯着这个示例，甚至把它放到演示文稿的每个地方。我当时完全没注意到。你可以想象，我拿着一个色彩鲜艳的示例给高层领导看，他们就问：“这个数字和假设是从哪儿来的？”我当时就说：“这太离谱了。我不知道这东西是从哪儿冒出来的，真是太离谱了。”但这种事确实经常发生，特别让人恼火。而这种连带后果真的会损害你对这套工作流及其产出结果的信任。So I was forcing my way into having something that could help me build slides. and I had a really nice harness that would like dig into the problem, get all the relevant documents and help me highlight what are the key points that I wanted to present to the team. And turns out in my conversations, I never caught the fact that when I ask for a modification and give an example, the model will fixate on the example so hard that it will put it everywhere in my presentation. I didn't pick it up. So, you can imagine I was showing some colorful example to senior leadership and they were like, "Where did you get that number and that assumption?" And I was like, "That's crazy. I don't know where that came from. That's crazy." But yeah, it tends to happen. It's really annoying. And that trace of consequence can really hurt the trust that you have in your harness and the results it produces.

我继续说。另一个原则是Let me carry on. So another principle is

原则 4：问责制Principle 4: Accountability

关于问责。嗯，这建立在证据的基础上，因为只有掌握了一些事实、有了某种参照，你才会做出决定。呃，我确实想这么说，但我们都有那么一个朋友，对自己并不了解的事情却说得特别自信。所以这算是个小小的反例，不过总的来说，我希望大家能达成共识的一点是：归根结底，无论 AI 智能体做了什么，我们才是负有责任、需要承担问责的人，对吧？嗯，还有件挺有意思的事：人们在讨论 AI 会不会……你知道，我们会不会因为 AI 而没有工作。我总会回到这样一个事实：我之所以能被聘用、能拿到报酬，部分原因是我需要对自己为公司创造的成果负责，而不是因为我坐在电脑前敲了点东西。on accountability. Uh and this builds on the evidence because you only make a decision when you have some facts when you have some point of reference. Erh well I do want to say that but we all have that friend who speaks very confidently about something they don't know. So little bit of a counter example, but generally I think if there's one takeaway I would love for everyone here to feel aligned with is we are ultimately the responsible people, the accountable people for whatever AI agents do, right? Um and there's a funny thing where people discussing that AI is going to um you know we're not going to have any jobs because of AI. I always go to back to the fact that I get higher, I get paid partially because I am accountable for the results that I produce for the company, not for me sitting in a computer and typing some stuff.

因此，人类体验中有一个层面，信任、问责以及团队协作都至关重要。那要怎样才能强化问责呢？你确实需要能够看到内部运作，了解这个黑箱里面是什么，等等。呃，我不知道这是谁说的。我很喜欢这句话，但一直没法引用出来。不过，智能体不会承担现实世界的后果，你会。所以我很喜欢举手表决这种做法，但这可能是我唯一能用的资源。也许这就是原因。有人见过这句话的文字版，或者听人说过吗？嗯，我不知道。这是 Claude 给我的。对。So there is an element of the human experience where the whole trust and accountability and working together as a team matters a lot. And how do you keep accountability high? You do need to be able to see under the hood, inside the black box, etc. Erh, I don't know who said this. I loved it. I never able to quote it. But agents face no real world consequences. You do. So I I love the show of hands thing, but it probably is the only resource I have. Maybe that's why. Has anyone seen either written or been told? Well, I don't know. That's what Claude gave me. Yeah,

主持人：感觉很棒，对吧？就像，哦，这说得通。是啊。是啊。一切都既往不咎。别担心。智能体做了什么，我们就承担相应的后果。所以，在我们的工具框架中加入问责机制，可以大大减轻你承受的压力。你不用总是在脑子里琢磨：“但你觉得它在这里是不是做了错误的假设？”之类的。呃，所以你想要减少歧义，而不是消除歧义。呃，我敢肯定，在座很多人都会怀念自动模式出现之前的日子。嗯，谁想回到自动模式还不存在、每一项更改都得按回车键批准的时代？Host: that feels great, right? Like, oh, that makes sense. Yeah. Yeah. Everything is forgiven. Don't worry. We face the consequences of what agents do. So, having accountability mechanisms in our harnesses reduces a lot the amount of stress you need to go through. Just going in your head like, "But do you think it made the wrong assumption here?" Whatever. Erh, so you want to reduce ambiguity and not remove it. Erh, I'm pretty sure a lot of people here will remember fondly the days prior to auto mode. H, who wants to go back to before when auto mode existed and you had to approve every single change by tapping enter?

嘉宾：是啊。是啊。那段时光真美好。Guest: Yeah. Yeah. Lovely times.

主持人：我现在还这么做。也行。Host: I still do it. Fair enough.

客人：有道理。Guest: Fair enough.

主持人：你做的这些，比如说100次点击里，有多少次失败了？我是说，拜托。这只是一个单行改动，而这个很重要。谢天谢地我看到了这个。是的，我只是更倾向于……Host: And from all the let's say 100 taps you did, how many of them you failed where? I mean, come on. This is a oneline change versus this is important. Thank god I was looking at this. Yeah, I just pref

嘉宾：而且这基本上就是难点的一部分，设计本来就是有意为之的。所以你审查的是任务，而不是单个改动。我们只是提醒你，设计流程框架时，要有意识地设置这些关卡，确保你仍然能够对产出的内容负责。嗯，要确保任何重要的判断，任何重要的消歧（不是那些无关紧要的），对吧？你还是能掌握一些控制权。嗯，没有这些会怎么样？我不知道。随机机器说没问题。答案真棒。批准每一项计算。光谱的另一端感觉也不太妙。嗯，或者逐行审查智能体提交的每个 PR。对，这样写得不太好，对吧？不过，确实会带来大量繁琐且不必要的工作，除非你以合理的方式、根据使用场景来设计它。Guest: and and that's basic that's basically part of the hardness like the design is intentional. So you review tasks instead of individual changes. And we're just providing you with a a reminder that when you're designing a harness, you want to make sure that you include on purpose those gates to make sure that you can still be accountable for what is being produced. Um so make sure that any judgment call, any disambiguation that is important, not the trivial ones, right? H you can get some hold of it still. H what happens without it? I don't know. The stoastic machine said it was fine. Amazing answer. Approving every calculation. The other side of the spectrum doesn't feel great. H or reviewing every PR line by line that your agents ship. Yeah, that it's not really well written, right? But yeah, it leads to a lot of either a lot of toil and unnecessary work if it's not designed um in a way that you know makes sense for the use cases.

呃。那么，关于如何设计良好的问责机制，最重要的启示是什么？设置少数几个信号非常明确的关卡。你举的任务例子，以及把任务拆解成便于推理它们之间如何关联的方式，都是很好的例子。人们只会在功能测试通过后才进行审查，在此之前的任何事情……我觉得你可以稍微含糊一点，说“嗯，看起来应该没问题”，但你不会把任何没有通过所有测试的东西推到生产环境，对吧？所以你会设置一些关卡，但这些关卡必须能提供非常明确的信号。Erh. So what is the main takeaway on how to design a good mechanisms for accountability? Few gates with very high signal. Your example of the tasks and breaking it down in a way that you can reason about the the way they connect. It's a really good example on where people do h reviews only on functional tests being passed. anything prior to that. I think you can be a little bit handwavy with yeah sure this looks about right but you wouldn't push anything into production that hasn't passed uh all your tests right so you put some gates in place but those gates need to be very high signal um

主持人：另外，作为补充，我觉得我们用了一些编程助手领域的例子。再说一次，这并不是我们在 Monzo 处理问题的方式，因为我们的背景不同。但这正是我们想要引入 harness 的东西。我觉得我们用 AI 编程的例子，是因为它们离我们很近，而且我们很多人每天都在用这些工具。不过，我们在这里应用的，是 Claude Code 做的同一件事：它有一项功能，可以让你衡量自己想在这个过程中参与多少。我们在构建金融领域的 harness 时，也可以用同样的方式引导用户。这确实说得很到位。嗯，再回到涉及资金的领域，有些事情你可以交给智能体自行判断。比如说，如果某件事有几个可能的取值，你可以说：“那就跑三个场景吧。”Host: and uh to to complement I think we're using some examples in the coding assistant world and and again this is not how we for instance approach things at at Monzo because of like a different context text, but when we this is bringing to to our hardness. I think we use the AI coding examples because they're close to us and they're the tools many of us are using on a on a daily basis. Uh but when we apply here is the the same thing that claude code does when they have a feature that allows you to gauge how much you want to be involved in the process. When we are building our financial harnesses, we can also guide our users in the in the same way. That is pretty spot on. Um, and again, just going back to the domain with money, there's some stuff you can leave the agent to figure out. For instance, if you have a few possible values for something, you're like, well, do three scenarios.

这大概会没问题。但在做出任何决定之前，你还是想把它们并排审查一下。你不会把王国的全部钥匙都交给一个你不一定相信会把你的最大利益放在心上的东西。倒不是说它会把我的利益放在心上，不过，嗯，重要的是你要知道，糟糕的决定会影响你之后做出的其他决定。所以，你设计的关卡必须适合其用途。你需要非常了解具体用例，才能把关卡设在真正重要的地方。否则，你只是在每次做决定前都加上一句“先问问我”，这样又会变得几乎没有有效信号。嗯，至于代码，你不一定需要询问，因为测试会说明一切。这就是凭证派上用场的地方：你只需要确认所有测试都通过了，因此就会很有信心。It's probably going to be fine. But you want to review them side to side before making any decision. You wouldn't hand over entirely the keys to the kingdom to something that you don't necessarily trust is going to have your best interest in mind. Not that it will have an interest at mine, but um so this is a it's very important that you know that bad decisions carry forward into your other decisions. So the gates you design need to be fit for purpose. You need to know the use case really well in order to be able to put the gates where it matters. Otherwise, you're just adding before any decision, ask me and then it becomes very low signal again. Um and well for code you don't necessarily need to ask because the tests will speak for themselves. That's where the receipts come in handy because you only need to check that everything is passing and you will have a lot of confidence because of it.

所以，如果你回到演示中，会发现有点好笑的是，这个计划在问责方面做得并不怎么样。唯一的步骤就是，我会检查我们计算出的所有数据、各种情景，然后决定每个月需要存多少钱。我的意思是，这样还是挺好的，因为不会有钱从我的银行账户转出去，但确实有一个问责机制，我觉得它能阻止其他任何事情发生。So if you go back to the demo, funnily enough, this plan is not very high on accountability. The only step is I will check everything that we calculated all the scenarios whatever and I will make a decision on how much I need to save every month. I mean still nice because no money is moving out of my bank account anywhere but yeah there's a accountability mechanism I feel confident that that will gate any anything else

嘉宾：我想再举一个例子，比如在这个挑战和决策中，假设我们接受这一点，并将其标记为已作出的决策。如果我们之后使用这条信息——比如说，20k 与第三笔付款在同一天到账——来影响最终决策，那我们肯定也要为此承担责任。Guest: I think as a as a as an extra example like in this this challenge and decision here likely as we let's say we accept this and this is now marked as a decision we made if we ever use this information like the 20k landing on the same day as payment three as something that influences the final decision. This definitely comes with the with the accountability.

主持人：是的。Host: Yes.

嘉宾：往上翻。不，我其实，呃，有点在即兴发挥。但如果你稍微往上滚一点，Guest: Go up. No, I actually like I'm improvising a little bit. But if you scroll up a little bit,

主持人：嗯，这又不是说我在夸自己的设计有多好，恰恰相反。不过你看，这里的结构有点奇怪，稍微有点像一棵树，分出一些枝杈后又汇合到一起。正如 Philip 所说，这是另一种问责机制：对于某些决策，模型要么会先做出假设，然后尝试向你确认，比如“嘿，这样对吗？这是你想在这里使用的数字吗？”要么会尝试弄清楚这些内容如何融入整个系统。我们认为，要展示这一点，一个好办法是设计一个能让你追究责任的 harness，确保你可以追踪所有已经做出的决策、是谁做出的，并且清楚地知道：这是智能体自行假设的内容，我让它继续沿用在后续流程中；还是我提供了修订后的输入。Host: h this is like again not not priding myself of any good design at all, anything the contrary. But h you see we have a little bit of a weird structure there. Looks like a tree a little bit with something branching out joining back in. That's another accountability mechanism as Philip is saying where there's some decisions the model will either assume and just try to verify with you like hey is this right? Is this the number you want to use here? Or just try to figure out things how they fit into the overall um system. We thought a good way to showcase that you need to design a harness that gives you the accountability is making sure that you can trace all the decisions that have been made, who made them uh and have that sense of well this was something that the agent assumed and I let carry on in the rest of my flow or it's something that I provided a revised input to.

再说说原因吧，我是一个非常看重视觉呈现的人，这可能是某种症状，我也不知道。呃，我认为原因在于，对话并不是沿着一条直线展开，而是会分支出去，再回到主线上。因为当你出于财务目的与智能体聊天时，即使已经做出决定，你也可能会对某件事进行推理，而且你完全有权纠正信息、补充更多背景。因此，我认为，创建一套 harness 时，重要的不只是设置一些关卡来验证事项、确保你仍然承担责任，还要能纠正方向，并确保你可以安全地这么做，而不是出现这种情况：“哦，就因为我在聊天里打了这句话，我之前所有的分析现在都没了。”And the reason again I'm a very visual person that could be symptom of something I don't know uh but the reason I think is kind of like a not a straight line conversation but rather branching out and then branching back in is that when you're chatting with an agent again for financial purposes you may reason about something after a decision has been made and you are well in your right to correct information and add more context. So, I think it's important that when you're creating an a harness that you not only build those gates for validating things, making sure that you're still accountable, but also for correcting the course and making sure that you can do so safely and it's not like, oh, because I typed this sentence in the chat, now all my analysis is gone.

这件事在我试图解决问题时已经发生过很多很多次了。可能我当时思绪很散，一边对着手机口述，中途说了一句话，结果整个分析都没了，因为我没法只删掉那一句话，同时保留其他所有内容。所以，这又给了我们一种方式，让我们能够对输入系统、用于财务分析的内容负责。你想切到幻灯片吗？可以。好，太棒了。我们可以试几种做法。嗯，你可以把鼠标悬停在那个写着婚礼示例中宾客人数的芯片上，然后点击它，把它添加到聊天里。接着你可以说：“哦，实际上，这个数字不对。比如说，宾客人数至少会有150人。”这样就会让你的工具重新思考计划，看看这项输入会对其他内容产生什么影响，并建议你修改计划，或者直接更改输入，诸如此类。H that has happened to me many many times trying to solve a problem and maybe I was like dispersed dictating to my phone I said something in the middle of it and then the whole analysis is gone because I can't really take away that one sentence and please keep everything else as it was h so this gives us another means to be accountable for what goes into the system for financial analysis and what Do you want to move to the slides? Yes. Okay, cool. So, uh, a few things we can try. Erh, you can go and and hover into the chip that guest that says how many guests we have in the wedding example. And you can add that that will add it to the chat and you can say, "Oh, actually, it's not this number. H we will get no fewer than 150 guests for instance. Erh that will get your harness to rethink the plan see the consequences of that input into other stuff and suggest a either a modification of your plan or just change the input whatever.

这里重要的是，你要有意识地建立反馈机制，并且要自行判断，确保你会根据这样一个事实做出重要决策：这项输入可能会改变你的整个分析。嗯，你需要密切留意，确保这些变化是合理的。你还可以试试另一种方法：点击步骤五，呃，就可以开启一个新的分支对话；我想聊天窗口底部有个“分支对话”按钮。The important part here is you build the feedback mechanism on purpose and you have some judgment over making sure that there is some consequential decision making on the fact that this input may change your whole analysis. H you need to be on top of making sure that those changes make sense. The other thing you can try is on if you click on step five erh you can open up a new branch conversation like a branched conversation I think is the button on the chat at the bottom

客人：在这里。Guest: here.

主持人：是的。是的。所以你可以新建一个分支。有人用过斜杠吗？Host: Yeah. Yeah. So you can open a new branch. Has anyone used the slash

主持人：顺便问一下，云端那个东西呢？你用起来体验怎么样？Host: btw by the way thing on cloud? How has been your experience using it?

来宾：有时能用，有时不能。Guest: Sometimes it works, sometimes it doesn't.

主持人：有时候管用，有时候不管用。感觉你像是在插一句话，但又不太算真正的插话。Host: Sometimes it works, sometimes it doesn't. And it feels like you're doing an aside, but not really an aside aside.

主持人：它会成为上下文的一部分，对吧？Host: It becomes a part of the context, right?

嘉宾：但你有选择权吗？比如说，也许我会在谈话中间问一个愚蠢的问题。Guest: But do you get a choice? Like maybe I have a stupid question in the middle of the conversation.

主持人：不，我别无选择。如果我提出一个问题，它就会成为本次会议的一部分。Host: No, I don't have a choice. If I ask a question, then it becomes the part of the session.

嘉宾：对，对。所以在这里，我们想做的是——回到我们为什么要这样做——我们想让你们看看，如果我们设计某种东西时，带着这样的意图，它会是什么感觉：嗯，这真的应该是个题外话。我跟 AI 聊天时会问太多蠢问题了。真的太多了。这让我觉得很羞耻，但最后我只好另开一个聊天，因为我不放心它不会影响我正在做的事。然后我会花更多时间说：“嘿，这是个蠢问题。请不要把它用于其他任何事情。”这就是菲利普刚才说的“请不要犯任何错误”那回事。Guest: Yeah. Yeah. So in here, what we're trying to do, and going back to why we're doing it like this, we want to show you how it could feel if we design something with the intent of, well, this really needs to be an aside. I have too many stupid questions when I'm chatting with AI. So many. And it's shameful, but I end up having to open a separate chat because I don't trust it's not going to contaminate what I'm doing. And then I will take more time just being like, "Hey, this is a stupid question. please don't use for anything else is the please don't make any mistakes thing Philippe was saying

主持人：h 不太好用。所以不管你设计什么，如果有反馈机制，都要确保它符合你实际想要的使用方式。呃，哎呀，酷。然后说到 h，最后一点、最后一条原则就是验证。所以Host: h doesn't work quite well. So whatever you design if you have a feedback mechanism make sure that you tailor it to how you actually want to use it. Erh oops cool and then as for h the last point the last principle will be verification. So

原则 5：验证Principle 5: Verification

你希望有另一个人帮你把把关，而这也建立在问责原则之上，因为你也要为这次审查负责。呃，也许这说得太抽象了，但有没有人遇到过这种情况：有人仅仅因为一句“不是，但是 Franchesco，你来审查一下”，就把一个糟糕决定的责任推给你或其他人？这种事确实会发生。you want a second pair of eyes and this builds on the principle of accountability because you are accountable for that review as well. Erh, and maybe that's too abstract, but has anyone been in a situation where someone h passed the blame on a bad decision to you or anyone else just on the basis of No, but Franchesco, review it. That happens.

嘉宾：这种情况确实会发生。嗯，大概一个月前，我们有过一次挺有意思的讨论：有几个人开始觉得，在 VIP 编码中，确保某些东西质量过关或正确的重担，最后落在团队审核者肩上，这让人觉得非常沮丧，因为你拿到的东西看起来都挺对，但最终确保它确实正确的责任还是在你身上。验证这一原则，说到底就是建立更多反馈回路，让你不会有盲点，但又不把任何责任推给别人。归根结底，这只是从另一个来源获取额外信息，以避免偏见，也避免你陷入自己的思维和流程中。但这并不会免除任何责任或问责。Guest: That happens. So, um, we were having a funny conversation like a month ago where a few a few people are starting to have thoughts that the burden of something being good or correct with VIP coding ends up being in the hands of the reviewer in a team that feels really demoralizing because you're just being given a bunch of stuff that sure looks about right, but then it's your responsibility responsibility ultimately to make sure that it is right. This whole principle of verification is just building more feedback loops so you don't have blind spots but without relegating any of the accountability. Ultimately, this is just extra information from a different source to avoid biases and avoid you getting stuck on your own thinking and your own processes. But it doesn't remove any of the responsibility or accountability.

嗯，是的，有意思的是，真正的专家往往不会检查每一个单元格。这更多是我在数据领域与财务团队合作时的经验。他们可以打开电子表格，只看几个他们知道能大致算出总额的单元格；如果感觉不对，就会告诉你。比如说：“我们怎么在营销上花了这么多钱？我觉得不对劲。”呃，这有点像凭直觉反应一下。不需要费太多功夫，也不必彻底重做计划，或者质疑自己做出的每一个决定，但你确实需要一个独立的参照点。呃，我记得有个例子，不过想不起来了，好像是关于一条特别出名的推文，说埃隆·马斯克有不知道多少万亿美元，然后有人说：“嗯，他可以给全世界每个人一百万美元，自己还剩下钱。”Um yeah, funny enough, expert experts often do not check every cell. And this is more on like my experience working with financial finance teams in data. A they could open a spreadsheet and just look at the you know a few cells they know will give you a ballpark total and if it doesn't feel right they will let you know. It's like, how did we spend this much money on marketing? I don't think that's right. Uh, and it's a little bit of a gut reaction. You don't need a lot of effort. You don't need to like redo your plan entirely or challenge every decision you're made, but you do want to have a separate point of reference. Erh, I think I I had an example, but I don't remember what was it was something about a tweet that got really famous about Elon Musk having I don't know how many trillion dollars and someone saying like, well, he could give a million dollars to every person in the world and still have money.

然后他就说：“等等，什么？这感觉不太对。”就是你读着读着会想：“等等，什么？看起来好像有道理。”接着你又会说：“不，不，这不对。”而你之所以会想去检查对方做的计算，多少是凭一种直觉。所以，有东西能帮你审查一下，给你一个粗略的答案，或者在有些地方看起来不对劲时提出质疑，是非常有帮助的。And he was like, wait, what? That doesn't feel right. Like you read it and you're like, wait, what? Looks plausible. And then you're like, "No, no, that's not right." And that needed need that needed for you to look into the calculations the person made is like a little bit gut feeling. Um, so just having something review and give you a rough answer or some challenge if something doesn't look right is extremely helpful.

主持人：抱歉。供应。Host: Sorry. supply.

嘉宾：这是个非常好的问题。呃，我一直在试着想出一些更接地气的例子，不过我觉得这是个很值得讨论的话题。啊Guest: That is such a good question. Uh I was trying to come up with a examples that feel more down to earth, but I think that's a very conversation to have. A

黄金示例与外部参考点Golden examples & outside reference points

一个很好的论据是：如果你拿到了数据应该是什么样子的黄金样例，那你的流程为什么会产出与之不同的结果呢？为什么不直接拿黄金样例来核对？这是我想从金融领域抽象出来的一点。你有很多信息来源，真的很多。除了自己的数据之外，你总能找到其他参考依据。所以，如果你使用的某些信息源声称某个事实——比如某个数字——不管是什么，你都应该看看自己没用到的其他资源，尽量增强你获取到的信号。这样就能避免陷入反馈循环：你手头只有现有信息，只有黄金样例，而你觉得一切看起来都没问题。构建测试框架时，尽量发挥一点创造力，想想还可以用哪些其他参考依据来帮助你做出更好的判断，或者发现哪里不太对劲。good argument is if you're getting golden examples of how the data should look like, why would your process give you something different to that? Like why don't you just check with your golden examples? And this is something I I wanted to abstract away from the world of finance. You have a lot of sources of information, a lot of them. You always have another point of reference outside of your data. So if some sources you're using are claiming I don't know whatever fact a number you have the internet you should always try and enhance the signal you're getting by looking out look at all the resources that are not the ones that I'm using and you avoid that h feedback loop where you only have the information you have available you only have your golden examples and everything looks right to me Always try to get a little bit creative when building a harness to try and guide which other uh points of reference it could use to help you make a better decision or to know if something is not quite right.

那是……吗？Does that

主持人：是的。我想，我们通常会尽量和相关人员一起合作，弄清楚代理应该使用哪一组信息，才能给出最好的答案。我们会试着了解怎样才能做到最优，这样就能再给你举一个可能有用的例子。我认为，黄金样例能为你提供真实基准，确保你处于正确的范围内，但没有哪个领域是孤立的。我以前在某家公司做 BI 分析师时，这种情况经常发生：我觉得自己写的东西很不错，查询也没问题，得出的数字看起来合理，我也检查了所有内容，没发现任何异常。可我把仪表板展示给某位利益相关者时，对方立刻就说：“这不对。”我当时就想：“你是觉得你懂 SQL 比我还多吗？”Host: Yeah. And I guess we generally try to like work whoever it is to like understand what would be the best set of things that agent should use the best answer. So trying to understand how that would have been the optimal to give you another example that could be useful. I think the golden examples give you your ground truth for making sure that you are within the right space but no domain is isolated. So what happened to me a lot of times when I was a BI analyst in some like company is that I thought what I wrote was really good. So my query was good. The number I got seemed reasonable and I was checking everything and nothing looked off. And then when I presented the dashboard to some stakeholder immediately went like that's not correct and I was like what do you mean? Do you know SQL more than I do?

你当时和我一起做了分析吗？结果发现，这个人来自另一个团队，他们只是大致知道那一年卖出了多少东西。所以他们凭直觉就知道那个数字不可能是对的，因为这意味着我在另一个地方的信息有误。把这一点放到智能体领域来看，我提供给智能体、放在测试框架里的黄金示例，会受到偏见、选择偏差等因素的影响，取决于具体领域；我挑选这些示例是为了这个领域，而且我觉得它们适用于这个领域。但当你看一个稍微有些关联、却不完全相同的领域时，可能会发现这其实毫无道理——只要看看旁边的另一项数据，就知道这个数字是错的，我甚至不需要重新检查自己的计算。好。所以有个概念叫“薄片判断”：你会从自己的经验中截取一些薄薄的片段，并根据自己在某个领域的经验，知道哪些事情感觉对、哪些感觉不对。Did you do do the analysis with me? And turns out this person was from a separate team and they just had an idea of how many whatever sales were made in that year. So they knew instinctively that number can't be right because it means my information in this other place is incorrect. So it's not if we translate that to the agentic world the golden examples I will give my agent in my hardness will be based on like biases and selection bias and so on based on the domain and I picked them for the domain and I think they're good for the domain but when you look at a slightly tangential domain you may find out that actually this is nonsensical like this number is wrong just by looking at this other thing next to me I I don't even need to look again into my calculations. Cool. Erh. So there is a concept called thing slicing where you have thin slices of your anecdotal experience and you know what things feel right or feel wrong based on your experience in your domain.

无论你在做什么分析，模型都会综合其他所有人的经验。因此，在构建你的测试框架时，你需要确保有一个反馈机制，这样就能从模型自认为了解的内容，一直验证到你自己认为了解的内容。如果把我刚才那句绕来绕去的话倒过来理解，它其实就是我们整个框架和所有原则的出发点：你要努力建立一个共享的领域、共享的上下文。本质上，你是在把自己的语言转化为模型的训练数据集。你只是在和模型一起梳理，比如：好，这就是我对这个问题的理解，或者这就是我们公司的做法。Whatever analysis you're doing. the model will have an average of everyone else's experiences. So when you're building your hardness, you need to make sure that you can uh have a feedback mechanism so that you can verify from what the model thinks it knows all the way to what you think you know. And if you flip that convoluted sentence I just said it's the it's how we started the whole framework and all the principles where you're trying to get to that shared domain shared context. You're essentially translating your language into the training data set for the model. You're just working with the model to be like okay this is how I understand the problem or how my company does this.

然后模型会帮忙，比如说：“好，那我该怎么把它纳入我的训练数据集？”而我们这里做的正好相反：我们只是说：“好，我觉得我们已经到了一个阶段，可以对这个领域做出合理判断了。不过我们还是建立一个反馈循环，这样我就能根据自己的经验告诉你，这看起来是不是完全跑偏了，还是挺合理的。” 所以，嗯，可能没问题，但它看起来也可能跟你的完全不一样。我觉得可以这么说：嗯，看起来差不多对。结果后来发现，你们公司确实很在意那 5% 的差异。当你在为团队构建测试框架时，意识到这 5% 可能会成为整个团队无法接受的关键因素，这会带来天壤之别。 嗯，如果你没有办法做到这一点，就会出现一些问题：要么逐行检查所有内容，要么什么都不检查；或者你根本不知道“差不多对”应该是什么样子。如果你没有这个参照点，你的测试框架也不会有。and then the model is helping like okay this is how would I fit it in my training data set here we're doing the opposite we're just saying okay I think we got to a point where you can make a reasonable judgment over this domain but let's have a feedback loop so I can tell you from my experience if this looks entirely off or if it looks reasonable so um yes it's probably fine may look nothing like yours I think is a way good way to describe it like yeah it looks about right and then turns out your company does care about that 5% difference it makes a world of a difference when you're building hardness for your team that you're aware that that 5% it will be a deal breaker for the whole team. Um there's some stuff that happens if you don't have um a way to do this. So checking everything line by line or checking nothing or you know no sense of what about right looks like and if you don't have that point of reference your harness won't have it either.

所以你需要构建那种外部验证点。我先跳过其中一些内容。至于验证，我们还会在 H 决策中加入那些注释，为模型自以为了解的一些内容添加外部参考点。事情就这么……还挺令人满意的，就像它……So you need to build those kind of like external validation points. I'm going to skip through some of this. And for the verification, it's those notes in the H decision as well where we're adding uh an external point of reference to some of the stuff the model thought it knew. It's kind of satisfac satisfactory when the thing just like

嘉宾：分支会回到主干。对，没错。而且我再次认为，这会留下一个追踪记录，让你能够快速浏览、凭直觉了解情况；这也是因为你能理解智能体经历的思考过程，而不只是看到最终输出。我回到这里。是的。和其他示例一样，如果你还在跟着看，可以尝试几件事。你可以针对一个带标签的挑战开启一个分支。这基本上就是一个独立的智能体，我们让它负责审查计划、审查假设，然后提出一些有益的质疑。这样看起来合理吗？嗯，只要点击打开它，然后你就会看到，正如 Philippe 刚才所做的那样，它会合并回主干，并实际修改计划。因此，这会带来实际影响；我们认为，在设计 harness 时有意识地考虑这一点，也是另一个重要环节。Guest: the branch go comes back. Yeah, exactly. And again, I think it creates a a trace to to um be able to like the the thin slice and the quick instinctive look also comes from from this because you understand the thought process that the agent went through and not only the the output only. I go back. Yes. So, as with the other examples, if you're still following along, uh there's a few things you can try. So, you can open a branch uh on a tagged challenge. That's basically a separate agent that we tasked with reviewing the plan, reviewing the assumptions, and then doing some healthy challenge. Does this read reasonable? Um so just click it open it up and then you will see how just Philippe did it will join back into the trunk and actually change the plan. So there is a consequence and we think this is another important part of being intentional with the design of your harnesses.

我们本可以把它留在“让 LLM 充当评审者”这一步。这样一来，每做一个决定、制定一个计划，或者其他任何事情，我们都可以再启动一个智能体来审查一下，然后就此了事。问题在于，如果没有反馈闭环，我们的财务决策也好，其他任何决策也好，都不会有任何改进。所以，这也是设计的一部分：你需要培养这种判断力，为自己的各种用例打造合适的 harness。另一件可以尝试的事是再打开一个，然后点击“忽略”。我们是有意这么做的，因为正如我之前所说，有时某些对话实际上并不会带来任何改变，或者与工作流程并无实质关系，而这方面你会比我们更清楚。因此，你需要这样的参照点来验证结果。嗯，简单总结一下，我们先把这页内容放一会儿。如果你还在尝试这个 harness，那么，构建真正可靠、能处理财务数据和资金的 harness，其框架是建立在这五项原则之上的。We could have left it as LLM as a judge. So for every decision, for every plan, whatever, spin another uh agent and then do a review and call it a day. The problem is if there's no feedback loop, our financial decision or decision whatsoever doesn't improve at all. So that's part of the design, the taste you need to build h to create fitforpurpose harnesses for whatever use cases you have. Then the other thing you can try is opening another and just clicking on dismiss. And we did that on purpose because sometimes as I was saying before some conversations don't actually change anything or not material to your process and you will know better. So you need to have that kind of like point of reference for validating things. Um so just to summarize and leave this on the screen for a moment. Um, if you're still playing around with the harness, uh, the framework for building really good harnesses that you can trust with financial data with money, it's built on these five principles.

所以，要有一个共同领域、保持一致、拿出证据、承担责任，并确保你有意识地建立一些验证闭环，去检验自己认为的东西，你懂吧。呃，什么？没有彩纸？真让人失望。好吧，有几个案例，我们认为有必要先明确一下。哦，抱歉，你刚才举手了。So having a shared domain, having consistency, showing the receipts, keeping your accountability, and making sure that you create intentional loops for verification outside of what you think, you know. Erh, what? No confetti. Disappointing. Okay. So, there's a few cases where we think it's important to define. Oh, sorry. You had your hand up.

主持人：是的。那么，我只是想确认一下自己的理解。你带来了一个像这五项原则这样的框架，可以以此为基础构建你的 harness。你还带来了一个 harness 作为示例。所以我在想，我应该什么时候构建 harness？harness 应该有多通用，或者多具体？因为你是用具体的 harness 来解决问题的，而且……Host: Yes. So, I just confirm my understanding. You brought us a framework like this five principles. You can build your harness upon it. And you also brought one harness as an example. So, I'm I'm thinking about when should I build harness, how generic or how specific the harness should be? Because you solve problems with the specific harness and

问答：应该构建多少个 harness？Q&A: How many harnesses should you build?

比如说，你工作中有多少个测试框架？只有一个，所有相关的事情都在这个框架里解决，还是有多个更具体的框架？这些框架有多通用？你多久搭建一个？比如说，搭建一个要投入多少精力？你每个月搭建一个，还是每六个月搭建一个？like for example, how many harnesses do you have in your work? It's just one and you solve everything that you have for this in this harness or you have multiple more specific harness. How generic and how often do you build one? Like how much effort do you put in building one? you make one every month or you make one every six months.

嘉宾：这是个很好的问题，当然也没有明确的答案。嗯，我觉得正如你所说，我们可以把它看作光谱的两端：一端是我所有事情都只用 Claude Code 或 Codex，然后把所有东西都嵌入技能指令之类的内容里；另一端则是每次想做一件小事，我都从头开始构建。当然，做到某个时候肯定会让人疲惫，因为有时候你花在创建工具上的时间比实际使用工具的时间还多。嗯，还是那句话，没有明确的答案。我觉得关键在于，你认为这个使用场景是否足够值得投入。比如，你想要可复现性；如果你发现使用预构建的 harness 得到的结果经常出错，那么你就会更想自己构建 harness，因为这样能让你获得更多控制权，也能兼顾确定性和非确定性。Guest: That's a that's a great question and there's no clear answer of course. Um I think it's we could we could trace as you said like the two ends of the spectrum as I just use clo code or or codeex for everything and I just embed everything into skills instructions so on so forth and on the other end of the spectrum is for every small new thing I want to do I'll build it from the ground up and like of course that will get tiring at some point because like you're spending more time creating your tools than actually using the tools sometimes Um again there's no clear answer. I think it comes to um when you feel the use case is worth enough. So you want reproducibility, you want it to feel few the results you are getting using the pre-built harnesses are very frequently getting it wrong because the value you get from building the harness is you get more control again with the deterministic bit together with the nondeterminism.

对吧？嗯，我们之所以要在金融分析这个场景下讨论案例中的这些原则，是因为如果我们在一个不具备这些原则的 harness 中处理这类用例，就很可能会遇到 Francesco 在逐步讲解时提到的一些陷阱。所以在这个案例中，为了避免这些反复出现的陷阱，搭建这样的东西是有道理的。不过我还是觉得，关键在于你认为有多少用户会使用这个用例，以及你所看到的问题有多重要。比如说，如果你要解决的任务并非业务关键型，那么让人在环路中审核输出并引导智能体，即使这需要多来几轮，可能也完全没问题。但如果你要把它交给内部用户使用，比如说，你要为信贷分析师团队构建某种工具，那么就会有 50 个人用它来辅助做出公司的关键决策。Right? Um the reason why we're talking about the the principles in the case in the um scenario of a financial analysis is because if we do this type of use case in a harness that doesn't have it we'll probably have some of the f pitfalls that Franchesco talks through each of the steps. So then in this case it makes sense for um to avoid these pitfalls we are seeing over and over and over again to build the thing. But again, I think it comes down to how many users you believe will go through that use case, how critical is the problem that you were seeing. So like if if there's a non-b businessiness critical um task that you were solving, you were probably fine to have a human in the loop reviewing the outputs that you get and steering the agent even if that takes a few extra turns. But if you're putting this in the hands of a uh internal user, for instance, let's say you were building something for your team of credit analysts, um you're going to have 50 people using something to support key company decisions.

对这一点，你肯定不希望出现太多差异。所以我再说一遍，我觉得没有明确的定论，但这些是我在做决定时会考虑的因素。也许我可以帮你想想反事实的情况。我们认为，这套方法尤其适用于金融领域的流程，因为这里的利害关系很大，而且你需要大量的反馈循环。即使是在个人或公司层面，我们在这里呈现的那些使用场景也有一些非常具体的特点。但试想一下，如果你只是想用 AI 生成社交媒体内容，也许你的某个使用场景就是像大家都在做的那样，用 AI 生成的视频刷屏 Instagram 和 TikTok。这里有些步骤你可能就没那么在意了。比如收集证据——为了什么？H 验证或问责就更不重要了。Yeah, you absolutely don't want a lot of variability on that. So again, there's no clearcut I would say, but these are the things I would consider in in making the decision. Maybe I can help with uh the counterfactual. So, we think this is a good recipe specifically for financial harnesses because there's a lot at stake. You want a lot of feedback loops. There's very specific things about those use cases that somehow we're representing here even at the personal level or the company level. But imagine you want to do you want to use AI just to produce like social media content and maybe one of your use cases is just flooding Instagram and Tik Tok with AI generated videos like everyone is doing. There's some steps here that you probably don't care that much. Evidence gathering. What for? H verification or accountability even less so.

所以，这主要是针对金融类用例设计的，也可以扩展到不同层级，但一切都取决于它是一个信任系统这一事实。如果你的用例不一定属于这种情况，那么这个方案可能就没那么有用；至于你想构建多少个 harness，我觉得 Filipe 说得更好，不过……So this is specifically geared towards like financial use cases and it can be extended to different levels of it but everything hinges on the fact that is a trust system. H this recipe may not be as useful if your use cases are not necessarily like that and in terms of how many harnesses you want to build. I think Filipe said it better, but

别把斧头磨得太锋利。Don't over-sharpen the axe

你很容易陷入不断磨斧头、却从不真正用斧头的境地。H 所以，如果你试着用某样东西，却得不到理想的结果，可以尝试添加几个技能、几个命令，再加一些额外的东西，但还是得不到你想要的结果或体验。你会发现自己在不同聊天之间复制粘贴文本，还得开启新会话。这感觉像是一套复杂繁琐的工具链。所以，把那些不得不重复做三五次的事情……好吧，只做一次，并围绕它来设计。you can easily fall victim for doing a lot of axe sharpening and not actually using the axe. H So, at some point, if you're trying to use something and you're not getting the right results, you can try adding a couple of skills, a couple of commands, a couple of extra things, still not getting the results or the experience you want. You're finding yourself copy pasting text from one chat to another, opening a new session. It feels convoluted harness. So you all the repeated things that you had to do three five times. Okay. Just once and you design it around it.

主持人：是的。是的。Host: Yes. Yes.

嘉宾：还有一个相关的问题。关于你们的婚礼策划任务，对吧？我觉得一种比较简单直接的思路可能是使用智能体，比如场地预订智能体。呃，你们试过吗？效果怎么样？或者你们对此有什么看法？Guest: Some more related question. So for your wedding planning task, right? I think one of like one of the naive way of looking at this is like maybe agent right like agents, venue agents. Uh, have you have you have you tried that and what's the outcome or like your opinion on that?

主持人：我先坦白一下，我订的是最贵的 Claude 套餐，而且我确实把额度用满了，而且可不是好事——Anthropic 肯定因为我而亏钱。我试着把所有场地、鲜花、婚礼相关的事情都研究了一遍，做规划，还列出各种不同的情景，比如我们要花多少钱等等。我发现，很多时候，只要用新的网页搜索功能，再加上一些改进过的工具之类的，它就能给我列出马德里排名前五的摄影师，这就已经够用了。效果好极了。我什么都不用再做，只要等它转着处理，就能看到比较结果，甚至还能联系他们。不过，在一些需要更多细节的场景下，我确实想要更多的控制权和可预测性，而且这又牵涉到财务方面的问题，所以我发现自己会想：我得搭建一个可靠的东西，不能再重新算一遍了。呃，为了让这种痛苦感更真实一点，我记得有一天我在车里，心想：我可以口述一些内容。Host: I will start by confessing that I have the most expensive cloth subscription and I do max it out and not in a good way as in Anthropic is definitely on the on the red because of me. I tried research everything for all the venue, the flowers, the wedding, do the planning, do the different like scenarios for how much we will have to pay etc. And what I found is that in many cases it was good enough to give me the list of the top five photographers in Madrid with the new web search with like a refined h tooling etc. Worked like a charm. didn't have to do anything else just spinning and having comparisons and even reaching out to them but for some more nuance use cases where I do want to have more control and predictability and this is linking to the financial nature of them I did find myself being like I need to build something robust I'm not recalculating this again uh and to to make the pain very real I remember myself one day because I was in the car and I was like I can dictate some stuff.

语音模式，然后我当时在开车，来回大概一个小时，你知道的，去一个小时，回来一个小时。我在那段时间里把同一段对话重复了三遍，就因为每次我想做一些与财务相关或相邻的规划时，总会出点问题，我就会想，嗯，是的，这确实有帮助。回到你举的例子，假设你是做 BI 的，对吧？嗯，我觉得以前没人会帮你验证 SQL 查询，对吧？所以现在看来，我们已经进入 SQL 查询的下一个阶段了，对吧？现在你在构建工具，因为测试框架更容易搭建之类的。你有没有找到办法，在和领导层沟通时把这一点说清楚？因为对我来说，每次出现 AI，人们就会期待它百分之百正确。你明白我的意思吧？Voice mode and then I was driving and it was about an hour drive back and forth, you know, an hour and an hour and I re had I had the same conversation three times in that t in that trench just because every time I was trying to do some financially related or adjacent planning, something will go wrong and I'll be like so yes, that helps. So getting back to your example of being a BI right like um I don't think back in the day there were people who were validating your SQL queries right um so this seems like we are it's like next level of SQL queries right now you're building tools because harnesses are easier to build and stuff have you found ways to kind of make this point clear to leadership ship in in conversations that like because because to me it feels every time there is an AI there is an expectation that it will be 100% correct like you see what I'm getting at

嘉宾：所以，你的问题是如何管理预期，以便人们进行自主式自我冲浪时……Guest: so is your question about managing the expectations so that when people do agentic self surf

主持人：这不是工程类销售，而更像是这样：假设你做出了这个东西，然后要跟公司的高层领导讨论一项财务决策，对吧？接着他们就会不断追问你。等你说“哦，我用了 AI 和 harness”，他们就会说：“那这必须百分之百正确才行。”你懂我的意思吧？你有没有找到办法，能让他们明白这仍然是 SQL 查询？Host: it's not enginic sales more so like let's say you build this and and you're talking to your senior leadership to make a financial decision in your company Right? And they start drilling you and drilling you and and the moment that you say, "Oh, I use AI and harnesses, they're like, well, this has to be 200% correct or something like you know what I'm saying? Have you found the way to to kind of make sure that they understand this is still SQL queries?"

嘉宾：有点剧透，Guest: Bit of a spoiler,

主持人：但明天有一场演讲。Host: but there's a talk tomorrow.

客人：我不知道。你有吗？[清了清嗓子]Guest: I don't know. Do you have any? [clears throat]

主持人：是的，是的，是的。不，我觉得，在这个 AI 优先的世界里，我们有时就像你刚才说的那样，会忘记它只是一个工具，是实现目的的手段。我们得先弄清楚到底发生了什么，对吧？所以，我觉得我们观察到的是——再说一次，结合 Monzo 的经验，我们一直是从高度注重影响的角度来处理这件事的。所以，重点不是“我们应该用 AI 来做这个，因为 AI 可以做这做那”。而是，我们想要实现什么结果？我们要如何衡量成功？我们是不是想减少与 BI 团队来回沟通的次数？呃，我们甚至没有 BI 团队，不过你明白我的意思。嗯，比如验证一个新仪表盘。好，那我们就可以跟踪这个指标，实际看看 AI 是否能帮助我们实现这个结果。嗯，我们是不是想减少内部咨询工单？比如有人说数字看起来不对的时候提的工单。Host: Yeah. Yeah. Yeah. No, I think um it's with the AI first world, we sometimes as you were saying forget that what's that that's only a two and a means to an end be before like what's what's going on right so I think what we would have noticed and again from Monzo's experience we've been approaching it from an highly impactbased um point of view so it's not about we should use AI for this because AI can something something something. It's like what's the outcome we're looking for here? What how do we measure success here? Are we looking for less uh back and forth with a BI team? Uh we don't even have a BI team, but you get you get what I mean. Um to validate a new dashboard. Okay. So, we can track that and actually see if AI can support us in in generating that outcome. um are we looking for um less I don't know internal ask tickets because the number looks wrong.

好，那我们就跟进这一点。至于我们采用什么方法、具体怎么做，在一定程度上其实并不重要，如果你明白我的意思的话，因为归根结底，它会成为一个工具，帮助我们更快地达成那个结果，同时提供更好的护栏机制，等等。所以我觉得，重点与其说是建立对 AI 本身实现方式的信任，不如说是就我们希望借助 AI 推动什么结果达成共识。我想我们刚才还提到了另一个问题。我的问题是，你展示了一个有一定复杂度的测试框架，对吧？Okay, so let's track that and then the approach that we do the way we do that doesn't really matter to a point if that makes sense because it will be again a tool that can get us to that um to that result probably faster with better guard rails and so on and so forth. So I think it's less it's less about gaining trust on the implementation on the AI itself but more about aligning on what's the outcome we're trying to drive with the AI. I think we had another one there. My question is more you have show a harness that has some complexity right

问答：Skills 与专用测试框架的比较Q&A: Skills vs. a dedicated harness

你有 UI，也有数据库之类的，但我也构建了一些会调用其中某些步骤的流程。不过这些都属于 skill，对吧？它会向你提问，基于 Python 脚本运行数字等等。所以我想问的是，你怎么看待这样的界限：是让一个 skill 完成所有这些事情，还是创建一个独立的工具？你试过这样做吗？you have a UI you have databases say but I have also build processes that invoke some of these steps but they are all at skill right that it ask you questions it runs the number based on a Python script and so on so my question is more like where you see the boundaries between like having a skill that does all these creating like a separate tool or have have you tried that?

嘉宾：当然。对，当然。嗯，我觉得这又和我们刚才谈到的用户群以及你对可复现性的要求非常相似：当这件事变得足够重要，以至于我们无法接受非确定性方法带来的那些权衡时，问题就变得关键了。我通常会这样描述技能命令，也就是基于提示词的方法：“嘿，请遵循所有这些要求，千万别出错，拜托拜托拜托。”因为这就是实际情况，对吧？我们把大量条理清晰、组织得非常好的指令交给机器，并希望它能照着执行。如今的智能体已经足够出色，大概有 99% 的时候都能按你的指令执行。嗯，可那 1% 呢？如果遇到某个导致事情无法正常工作的边缘情况呢？Guest: Absolutely. Yeah, absolutely. Um I think it it again very similarly to to what we were saying about your user base and the reproducibility you want is when this becomes critical enough that the tradeoffs we are getting with the nondeterministic approach are are not uh we're not willing to have them. I usually describe the the skill command like the the promptbased approach as hey please follow all of this and make no mistakes please please please like again because this is what's going on right we are we are giving a a lot of organized very well organized instructions to a machine and hoping that it it will perform these instructions agents are good enough nowadays they will probably perform your instructions 99% of the time. Uh, but what about that 1%? What about an edge case that makes things don't work?

如果你把它分发给内部用户，而某个内部用户在其用户级别的 cloud.mmd 文件中写了与您要求代理执行的内容相冲突的指令，该怎么办？你明白我的意思吧，就是说，呃，当你开始把它分发给许多用户，并用于各种场景时，有些地方会变得一团糟，而且这些情况是你无法预料的。然后你可能会发现自己……我们在 Moz 有一个内部的代理扩展贡献机制，多个团队都可以……我遇到过一些对我有用的东西，就想在内部分享一下。比如插件、技能之类的。我们审查指令时，经常会看到一些内容试图涵盖这样的情况：如果用户尝试做这件事，就告诉用户这个技能不支持。你看，这又是在试图涵盖很多情况。而确定性——我说的确定性，是指后端代码，比如 Python，或者我们使用的其他东西——由框架开箱即用提供的那部分，是我们可以处理的，或者……What about if you distribute this to your internal users and your internal user has a instruction in their user level cloud.mmd file that goes against what you are telling your agent to do. You see my point is like um there are many when you when you start distributing it to a number of users and use cases things can get messy in places you are not able to predict and then you can find yourself I we have a at Moz we have an internal like contribution for agent extensions where multiple teams can well I've come across something that works with me let me share it internally um so like plugins skills and so on and very frequent ly we when reviewing the instructions we see things trying to cover for if the user tries to do this tell the user back that this skill does not support that and you see like it's again trying to cover for many things that with the determinism and when I say determinism is the backend code like the python the whatever we are using uh that the harness provides that out of the like out of the box is something we can cover or

主持人：还有人有其他意见或问题吗？Host: anyone has any other comments, questions.

客人：好，太棒了。Guest: All right, cool.

主持人：那个东西在哪儿来着？算了。算了。算了。好，好，好，好。快进。快进。啊，有意思。呃，我本来想开个玩笑，结果点这个东西点得太快，完全错过了。不过金融挺有意思的。我说它有意思，是因为我觉得这是个特别有趣的领域，而且还有一个额外的优点：以前金融给人的感觉就是在处理电子表格，特别像那种枯燥的工作，到处加加数字、写写公式什么的。可你要是问任何一位会计师，他们会说这全是创造力和解读，因为他们会想出各种最奇怪的做法。我觉得在这种模棱两可的情况下，AI真的很擅长，真的非常擅长。你会看到一些电子表格里的数据，然后发现有些现金流不太对劲，因为有些收入被记到了一个月之后，于是你就会想：啊，这就是创造力。Host: Where is the thingy? Never mind. Never mind. Never mind. Cool, cool, cool, cool. Skipping ahead. Skipping ahead. Ah, fun. Erh, I wanted to make a joke and I completely missed it for clicking this thing too quickly. But finance is funny. And I say funny because I find it to be a super interesting domain with a the added advantage that back in the day finance felt like just working with spreadsheets, very like boring work, adding numbers here and there, formulas, whatever. If you ask any accountant is all about creativity and interpretation because they will come up with the weirdest things and I think in that level of ambiguity AI is really good really really good you have some bits that are the spreadsheet and then you realize some of the cash flow didn't quite make sense because some of the revenue was attributed a month later and you're like ah creativity.

所以，因为我们身处金融领域，而你的思维方式不同，看待问题的角度也会不同：你可能觉得问题很简单，只要让一个智能体在 10 分钟内解决就行；也可能看到问题的真实面貌——它既涉及一些必须始终可靠运行的事情，也需要对字里行间的内容进行大量解读，并留有发挥创造力的空间。所以，抱歉，这个笑话讲得很差，但这就是我想要讨论的内容。呃，归根结底就是……So because we are in this space of finance and depending on where is your mind mindset at you could look at a problem as something very simple that you can just get an agent to fix within 10 minutes or you could look at it as it actually is which is a mixture of things that need to run reliably 100% of the time and a lot of interpretation and freedom of creativity in between the lines. So yeah, sorry, very poorly delivered joke, but that's what I wanted to go through. Uh, and the bottom line being

要点：有意识地围绕信任进行设计Takeaway: design for trust on purpose

你设计的测试框架需要经过深思熟虑，才能提升可信度。别想当然地认为，哦，因为这是由 AI 驱动的，所以它就能达到那个——怎么说来着——200% 的准确率。除非你设计好所有机制，确保无论运行什么流程，结果都能在一定程度上值得信赖，否则它很可能做不到。嗯，而且模型会越来越好。过去六到九个月里，我们已经看到了这一点。说实话，事情发展得太快了，我都有点应接不暇，但模型仍然需要赢得信任。它不可能百分之百正确；而且当我们谈到钱的时候，哪怕只有百分之一的错误，有时也会造成很大的损失。所以，是啊，我们有点跳过前面的内容，直接进入了提问环节，不过如果还有人想补充什么，……the design of your harnesses needs to be intentional to increase trust. Don't assume that oh because this is running on AI is going to be what was the the quote 200% accurate. It will likely not unless you design all the mechanisms so that whatever process runs you know you can trust the results to a degree. Um and yeah models are going to get better. We have seen the last six months nine months in saying I'm getting a little bit of whiplash even with you know how fast things are going but they still have to earn the trust. it's not a 100% right and in when we're talking about money 1% wrong sometimes is a lot of money. So yeah, we kind of skipped into the questions, but if anyone has anything else,

主持人：对，简单问一下，比如在用户界面里，你们是如何体现可信度的？我是不是就直接问它：“给我看看你是怎么计算这个的”？还是它会把代码展示出来？Host: yeah, just really quick, like in the UI, how do you demonstrate trust? Do I like you just ask it? Show me how you calculated this or does it show you the code?

主持人：你能举个例子吗？比如，怎么证明这一点？Host: Can you show an example? Like how do you prove it?

嘉宾：本意是把所有东西聚合在一起。所以框架里的所有步骤都应该会失败，比如说，哦，但我知道这是从哪里来的；但我知道这是作为代码运行的；但我知道它经过了测试。所以这应该是一种让人沉浸其中、觉得值得信赖的体验。不过话说回来，这只是我们想出来的一种实现方式，所以并不是特别直接。Guest: The intent was that it was a aggregation of everything. So all the steps in the framework should have failed like, oh, but I know where this is coming from, but I know this is running as code, but I know this is tested. So it should have been an experience to kind of like immerse into this feels trustworthy. But again, it's one implementation we came up with. So not not super direct.

主持人：是的。不过我认为，关键在于，没有任何单一做法能在任何时候都赢得用户的信任。所以对有些人来说，验证会更重要；而另一些人则更加怀疑，他们想看到实际的计算过程、数据等等。所以我还是认为，这一流程指南能够涵盖各种可能的情况。Host: Yeah. But I think the the takeaway is there won't be a single thing that will work to earn the user's trust at any point. So I think that's for some people the verification will be more important. Some others are a bit more skeptic and they want to see the actual calculations and numbers and so on. So I think again that the process guides and covers for a multitude of u of possible possibilities.