为 AI 规模的提交量重建 CI：Gautam Korlam 谈合并队列与审计证据

简要描述Brief Description

Rebuilding CI for AI-Scale Commit Volume — Gautam Korlam， SonarSource

目录Table of Contents

• 编码代理将瓶颈转移到了 CI 中
  Coding agents move the bottleneck into CI
• SubmitQueue 与合并前的组合测试
  SubmitQueue and testing combinations before merge
• 为人类速度打造的队列
  Queues built for human speed
• 软件工厂与反复修复循环
  The software factory and repeated repair loops
• CI 故障与反馈成本
  CI failures and the feedback cost
• 增量构建、缓存与并行工作
  Incremental builds, caching and parallel work
• 不止于通过 CI：强制执行策略
  Enforcing policy beyond passing CI
• 基于风险的合并与审批策略
  Risk based merge and approval policies
• 已测试的提交、覆盖项和审计证据
  Tested commits, overrides and audit evidence

编码代理将瓶颈转移到了 CI 中Coding agents move the bottleneck into CI

大家好。呃，欢迎参加我的演讲。呃，今天我们要聊聊如何重建 CI，以应对 AI 规模的提交量。这些想法来自我十多年来在构建和合并基础设施方面积累的大量经验。我想带大家了解一下我们在软件工厂中遇到的所有新瓶颈，尤其是在高度受监管的行业和银行业中。这些变化意味着什么？我们如何在保持快速迭代的同时，让流程更加合规、可审计？那我们就开始吧。 我叫 Goautam，是 QAR 的联合创始人兼 CTO，最近加入了 Sonar，担任首席工程师。在被收购之前，我曾在 Uber 工作近十年，负责单体仓库的构建和合并基础设施。从构建、CI/CD、IDE 等等，到确保开发人员保持高效，这些工作我都参与过。Hello everyone. Uh welcome to my talk. Uh today we're going to be talking about how we are rebuilding CI for AI scale commit volume. And this comes from a lot of experience from build and merge infrastructure that I have been working on for a decade plus at this point. And kind of want to walk you through all the new bottlenecks we're seeing in software factories and also specifically for highly regulated industries and banks. How does that translate? How do we keep things more compliant and auditable while being able to move fast? So let's dive in. My name is Goautam. I am a co-founder and CTO at QAR and recently joined Sonar as a principal engineer. After the acquisition pre previous to I was at Uber for almost a decade working on build and merge infrastructure for mono repos. So everything from build, CI/CD, ids, you name it to make sure developers are productive.

现在，在 Solar 的 QAR 环节，我正努力确保所有已签入的代码都经过验证，并且所有由代理编写的代码在合并前都达到高质量标准。所以在开始之前，我想先简单回顾一下背景。呃，在……And now at QAR as part of Solar I'm trying to make sure that code that's been checked in is verified and all the code written by agents is of high quality before it goes in. So before I get started I want to just paint a bit of history picture. Uh at

在合并前提交队列并测试各种组合SubmitQueue and testing combinations before merge

在 Uber，我们在这些东西成为 agent 之前，就已经面对 agent 规模的涌入量了。我的意思是，我们有一个巨型 monorepo，每天要提交数千次变更，而且构建运行时间很长。这意味着，如果你把所有操作都串行化，一次合并一个变更，以免弄坏主分支，那么整个过程就会花很长时间，甚至长达数天——在这个案例中是 20 天——才能确保所有检查都通过。反过来，如果你跳过所有这些步骤，直接让大家把变更合并到主分支，我们就会一直看到主分支变红，因为变更之间会互相冲突；即使每个变更单独通过了 CI，合并在一起后也可能出问题。于是我们的 iOS 监控就会一直显示红色，这对开发人员来说效率很低。为了解决这个问题，我们构建了一个名为 Submit Queue 的系统，并为此发表了一篇论文。我们会测试可能提交进来的各种组合，确保能够缩短等待时间，这样就能并行测试这些变更。Uber we had agent scale coming volume before they were agents. So what I mean by this is we had one giant monor repo and we were doing like thousands of changes a day and with very long running builds. This meant that if you just kept everything serialized and merged them one at a time so as not to break your uh main branch you would take a really long time up to days so 20 days in this case to make sure that everything was green. And if he skipped everything and just let everyone merge to me, we would see red main all the time because changes would conflict with each other and even though the CI would be green on one of them combined, they might actually break. So we would see like our iOS monitor would be like red all the time and that was not productive for our developers. To fix this, what we did was we made a system called summit cube which we have a paper on and we tested all the likely possibilities of commits that are coming in uh and making sure that we can like reduce the wait time so we can test these things in parallel.

我们还可以通过测试所有组合，预测哪些情况可能会出问题。因此，一旦发现某个 PR 可能导致问题，我们就可以把它从队列中移除，并告知作者：他们必须先修复问题，然后在变基后才能重新提交。最后，只有当主干上的所有提交组合都通过测试时，才会将它们直接合并到主干中，这样才能保持主干干净。我们要确保为保持主干干净而付出的额外开销是值得的。直到今天，Uber 仍在运行这套系统。我相信现在 GitHub 和其他代码托管平台都推出了合并队列，但在当时还没有这样的系统，所以我们就自己开发了一个。[哼了一声]We could also predict what could break by testing all the combinations. So once we know that a PR might break, we might remove it back from the queue and let the author know that they have to go fix it before they can push it back up after rebasing. And finally like to keep the trunk clean only if it passes with all the combination of commits coming in it would be let into go directly onto the trunk. So you want to make sure that the overhead was worth it for keeping our trunk clean. And this is still running today at Uber. I'm sure now we have like merge cues that GitHub came up with and all the other code hosts but for back then there was not a system. So we developed this. [snorts]

为人类处理速度而构建的队列Queues built for human speed

所以这种方法的问题在于，我们目前所有这些队列——无论是 CI 还是代码审查——都是 SDLC 中的集中式瓶颈。这些队列都是按照人类的速度设计的，而一旦按人类的速度来构建，就会出现这样的情况：当你转向使用智能体时，它们就跟不上了，因为过去人类每小时只会提交几次变更，所以我们可以一次进行一项测试和审查。如今有了智能体，开发者一天可能会提交好几个 PR，而且他们不一定会等着排队；他们都想同时进入队列。这意味着 CI 和代码审查队列会越积越多，承受的压力也越来越大。再加上软件工厂和后台智能体会在各种工作流中触发，你突然就会发现一切都慢得几乎停滞，因为你的瓶颈仍然是 CI 和代码审查，而它们已经超负荷了。那么这对当下的团队意味着什么？So the problem with this approach is currently all these cues that we have. So that's either PCI or review are a centralized bottleneck in the SDLC. So all these cues were built for human speed and once you build it from human speed what happens is they are not able to keep up once you go to agents because humans generate changes used to generate changes like few times an hour. So we could do testing and review one at a time. Now with agents developers are writing like several PRs uh in many in a day and they don't always wait for the queue they all want to like come in at the same time which means your CI and review cues are piling up and having pressure and add to that like the patterns of software factories and background agents triggering on like various workflows suddenly you see like everything slowing down to a crawl because your bottlenecks are still going BCI in review and they're over capacity in this So what does this mean uh for teams today?

对吧？所以，稍微从更宏观的角度来看，自从大约 3 年前我们开始使用 Guitar 以来，我们自己的提交速率几乎增长了 10 倍，简直令人难以置信。团队规模还是一样，我们仍然很小，却能推送多得多的代码；我们甚至也能从 Guitar 的行业趋势中看到这一点。这张图展示了 Guitar 随时间审核的 PR 数量。如果你看看增长轨迹，就会发现它几乎是指数级的。我们几乎每个月都以环比 25% 的速度增长，审核、修复 CI 和处理其他事项的 PR 数量都在增加。我认为，这也得到了 GitHub 等业内公司的数据佐证；他们公布的数据也显示 PR 数量惊人地增长。如今，每个团队都在经历这一趋势——既然我们这样的小团队都看到了这种增长，我相信大型团队和银行在推进 AI 应用的过程中，增长幅度会更高。Right? So just to put things into perspective uh our own uh rate of commits has almost like 10xed since we started guitar uh about 3 years ago and it's just like uh unbelievable with the same size team we're still very small we're able to like push so much more code uh and we even see this in the industry trends on guitar. So this graph shows you the number of PRs that we see that guitar reviews over time. And if you look at the growth trajectory, it's pretty exponential. Every month almost we're growing 25% month over month of just the amount of PRs that have to review fixed CI and other things. So this is also I think like corroborated by evidence that industry players like GitHub have been putting out where they're seeing like uh staggering growth in PRs and this is now coming for every single team not if our small team is seeing this I'm sure like the larger teams and banks are seeing it like at a much higher degree as they are going through their AI adoption journeys.

[哼了一声] 所以我刚才说的问题是，如果智能体在开 PR，而你让它们各自通过 CI 系统检查所有内容，它们可能都能通过；但等你把它们合并回主分支后，主干仍然可能出问题。那么你要怎么解决这个问题？怎么才能在如此快的节奏下让一切保持绿灯？这非常重要，关键在于我们如何解决提交队列的问题：这些改动本身并没有彼此进行测试，所以 CI 系统需要做出某种改变，架构设计也需要相应调整。[snorts] So the problem I was talking about right so if agents are opening PRs and if you make them uh check everything through the CI system uh individually they might pass but once you kind of merge them back on domain your truck might still be broken. So how do you like solve for this like how do you keep things green uh at this high rate of pace right so this is very important kind of like how we solve submit Q the changes themselves are not uh testing against each other so there needs to be some sort of change in either the CI system as well as how you architecture

软件工厂与反复修复循环The software factory and repeated repair loops

代码和队列都需要变得更智能，才能解决这个问题。我们还看到，大家都在尝试采用所谓的软件工厂：让智能体根据规格说明编写所有代码，再由人类负责制定规格说明并进行审查。但正如我提到的，现有的工厂流水线是按照人类的速度构建的，这意味着随着智能体提交更多变更，流水线的每个环节都会变慢。比如在构建和测试环节，编译失败时可能需要反复重新构建；也可能有集成测试，或者有些测试不稳定，无法在本地复现，这些都会拖慢整个流程。因为你必须重试，而对于复杂的构建，很多时候这可能要花上几个小时。再加上现在还有审查和审批环节。code and in the cues to be smarter to be able to solve this and the problem we're also seeing with everyone trying to adopt like software factories which is having agents write all the code uh from the spec and then just having humans be responsible for the spec and review is that the factory line that exists today was built for human speed as I mentioned which means that as agents push more changes uh every part of the factory line is going to be slowing down. So for example with build and test you might have lots of rebuilds whenever uh compilation breaks or you might have integration tests you might have flaky tests that are not reproducible locally and that just slows everything down. So because you have to retry them and it could take hours in many cases with a complex build. Add to the fact that now you have review and approval.

所以你们很多人可能已经在用 AI 来审查代码了。希望你们不是每次审查都完全依赖人工。理想情况下，应该先让 AI 做第一轮审查。如果没有这样的机制，审查人员就会成为瓶颈。即使审查过了，你们可能还想增加一些标准，以确保代码得到全面批准。因为现在行业里的情况是，很多人只是走个过场就批准了，因为他们跟不上 PR 的高增长速度。他们现在只能依靠其他信号来应付过去。所以流程必须改变。我们现在面临的另一个问题是，很多检查都在运行。我们保存的很多记录都会过期。也就是说，旧日志过期后就不存在了。So a lot of you probably already use like AI to review your code. Hopefully uh you're not doing every single review with with humans. You ideally are reviewing with AI as a first pass. Now the reviewers become the cue if you don't have that system in place. And even after you review, there might be some criteria you might want additionally to make sure that it's approved fully because what's happening now in the industry is a lot of folks are just rubber stamping them because they cannot keep up with the high rate of PR. They were relying on other signals now to be able to just get through. So the process has to change. And the other problem right now we're having is like a lot of these checks are running. There's uh a lot of records that we keep that are expiring. So like your old logs don't exist as they expire out.

没有盖章注明实际检查了哪些内容，因为好像只有一个批准章。因此需要对这些事项留存记录，以便进行审计和合规检查，以……There is no stamp of like what was actually checked because there is like just an approval stamp. So there needs to be some record of these things for auditability and compliance checks to

CI 失败与反馈成本CI failures and the feedback cost

发生。我们看到的另一个问题是，随着代码量增加，修复所需的时间也带来了大量额外开销。呃，至少在我们的平台上，有记录显示 13% 的 PR 遇到过 CI 失败。这包括一次或多次 CI 失败。这意味着在所有这些情况下，都需要额外往返一轮：要么回到开发者的机器上，要么由智能体去理解日志、找出失败根因、修复问题、推送另一个提交，然后不断迭代，直到 CI 变绿。而这还只是 CI，因为大多数时候，如果人工评审者想查看一个 PR，他们不会在 CI 变绿之前查看，因为这是第一道防线。所以这也会拖慢评审流程，因为两件事无法并行进行——通常我们希望等某项内容定稿后再查看，而现在每个实验室都要为评审付出成本。happen. The other issue we're seeing is that as the code volume increases the time taken for repair also is adding a ton of overhead. Uh what we see on our platform at least 13% of PRs had a recorded CI failure. This is just one or more CI failures. Which means in all these cases there is a separate round trip to go either back to the developer machine or for an agent to understand the logs root cause of failure fix it push another commit and then keep iterating till it's green. And this is just CI because most of the time if a human reviewer wants to look at a PR they're not going to take a look unless CI is green because that's the first line of defense. So that also like slows uh the review process because they can't go in parallel because usually we want to look at something once it's finalized and every lab is now paying for review.

所以，如果收到审阅意见后你又进行一轮修改，结果又回到同样的状况，就会感觉像是在重复这个循环。那么我们该怎么改进呢？比如说……So if your review comments come in and you're doing another pass same at break again and it kind of like repeats the cycle. So how do we make this better like what

增量构建、缓存与并行工作Incremental builds, caching and parallel work

随着智能体提交越来越多代码，我们该怎么做才能让这个问题更易于管理？我们已经发现了这一点，呃，显然在……但现在在 Guitar Steam 也是如此。最显而易见的做法当然是让 CI 尽可能快速，这意味着你可能需要删掉很多耗时的检查。可以尝试把这些检查转移到合并后再执行，甚至让 CI 流程本身尽可能快，因为这样修复问题的效率会高得多。你可以并行处理任务、进行分片，也可以使用缓存。我们也会聊聊我们用过的一些技巧。你还要确保评审足够快。不只是做深入评审，也可以分多轮评审，让开发者更快获得反馈。这样他们就能据此判断，哪些高风险变更需要深入检查，而对于低风险变更，甚至可以让 AI 批准并合并。what do we do to make this problem more manageable as agents are pushing down more code. So we have discovered this uh obviously back to but also in guitar steam right now. But the most obvious thing is of course like make your CI really quick and fast which means that you may want to like remove a lot of the expensive checks. Try to offload them to like post merge even and try to make the CI step itself really quick because your repairs will be much more efficient. So you can paralyze things, you can chart, you can do like uh caching. And we'll talk about some of the techniques that we have used as well. You also want to make sure the review is fast. So not just like a deep review, but you also want to have maybe multiple passes that are like giving quicker signal to your developers. So they're able to like use them as a way to understand in what scenarios they may want to go deeper for higher risk changes or maybe for like lower risk ones even like let the AI approve and merge them.

对于那些已经在 CI 中运行的检查，如果能把它们左移，尽早发现问题的成本就会低得多。其实，这对 CI 之后才会进行的检查也很有意义。比如，你可能有成本高昂的回归测试，或者一些在生产环境中可能会失败的检查。当然，并非所有情况下都能把检查左移，但只要尽可能做到这一点，就能减少后续流程中的瓶颈。而且，智能体在本地通常掌握着更多上下文，因此更容易修复遇到的问题。你还需要制定策略，这对受监管行业尤为重要。我们在 Uber 有个教训：由于软件出现故障，司机未经全部必要检查就能加入平台，而这源于一次功能标志变更。结果，我们失去了在伦敦运营的牌照。And for the checks that are already running in CI, if you can move them to the left, it's way cheaper to catch them. And this actually uh is very relevant to checks that happen even past CI. For example, you might have like expensive regression tests or like things that might fail in production. Uh not every time it's possible to like move things to the left, but as much as possible, if you can move them, it means you have less bottlenecks down the line. And locally, agents have done more context anyway to fix the issue that they're facing. And you also want to have policies. This is very important for regulated industries. So a story we have at Back at Uber is we lost like the license to operate in London because there was a glitch in the software that allowed drivers to get on the platform without all the proper checks and that was because of a feature flag change.

所以我们不得不建立一套流程，确保每项会影响伦敦、且需要经过 SDLC 的变更，都有某种追溯链路：这项变更会影响伦敦吗？如果会，有哪些控制措施？这样一来，我们就成功拿回了许可证。现在，你们中很多人可能也正面临这类挑战：智能体让人更难弄清代码库里究竟发生了什么。那么，该怎么做呢？So we had to put in a process in place so that every change affecting London that was going through the SDLC would need some sort of like a a lineage of okay does this change affect London? If so, what are the controls in place and we were able to win our license back? Now, a lot of you might be facing these kind of like challenges where agents just make it much harder to understand what's happening in your codebase. So, how do

强制执行策略，而不只是确保 CI 通过Enforcing policy beyond passing CI

你如何确保这些政策确实得到了遵守？所以，你需要有办法在 CI 之外强制执行这些规则，因为代码能编译通过，并不意味着它遵循了特定规范。比如 HIPPA，也可能是 Tora。还有很多其他政策，除非你的代理在另一端强制执行，否则它们根本不会了解这些政策。你还需要确保记录下这些规则已被执行，以便进行审计。顺便简单讲讲我们是如何加快 CI 的。我们的 CI 原本已经算快了，但在你想要推送更多代码时，还是非常烦人。我们的 CI 大约需要 10 分钟，我们想大幅缩短这个时间。所以我们开始思考如何缓存输出；我们的构建系统使用 Rust。you ensure your policies are actually being respected? So, you need to have a way to like enforce these rules beyond CI like just because the code compiles doesn't mean that it's following a particular specifications. It could be like HIPPA. It could be like uh Tora. There's so many other policies that these agents are not going to be aware of unless your agents are enforcing it on the other end. And also you want to make sure that you record that it was enforced for auditability. So just a quick story about how we made our CI faster as well. So we were at a baseline, we were I would say like still fast but like very annoying when you're trying to like push more code. uh we were around like 10 minutes for our CI and we wanted to like bring that down significantly. So we started think looking at how can we cache outputs uh we use rust for our build system.

所以我们开始尽可能多地缓存并重复利用缓存结果。我们还研究了如何减少编译器中的变动。因此，我们又做了一些改进，确保编译部分更容易缓存，也不那么容易失效。在此基础上继续改进，通过分片并消除这些开销，我们把 CI 耗时缩短到了几乎不到一分钟（针对增量步骤）。这意味着冷启动的 CI 预热后，增量构建会快得多，我们也能推送更多代码。 另一个问题是代码审查。CI 通过并不代表一切就绪。有意思的是，我们测量 PR 从提交到上线的 P90 耗时时，发现绝大部分时间都花在了 Uber 的代码审查上。说实话，即使是今天，大多数团队花在审查代码和其他类型测试上的时间也更多，直到他们确信可以把代码合并到主分支。So we started like caching as much as possible and reusing them. And we also looked at ways to like reduce churn in the compiler. So we added some more fixes to like ensure that the compilation part is more cachable and is invalidated less often. And by adding more fixes on top of this, by sharding and by removing like these overheads, we're able to bring the time of CI almost like less than a minute for incremental steps, which means once the cold CI is warmed up, incremental builds are way way quick quicker and we can push a ton more code. The other issue is also review. So just because the CI is green, what was funny was when we were measuring our uh P90 time of like PRs making into production, the vast majority of time was spent in review back at Uber and even today to be honest like most teams spend more time reviewing code and doing other kinds of testing before they are comfortable merging the code into main.

因此，除了 CI 之外，这件事也需要纳入考虑。因为现在的情况是，智能体当然会同时生成代码，而 AI 智能体也会在你的代码审查系统中发现问题；这些自动审查发现的问题，你现在也得回头处理。你要么在本地为智能体获取上下文，要么让 CI 上的智能体反复迭代并修复问题，然后由人类审查 AI 首轮处理后剩下的内容。这也需要纳入整个流程，这样你就不会把宝贵的人力时间浪费在越来越多由智能体生成的 PR 上。嗯，这里有个我们的例子。嗯，我们当时在重构代码库，以加快 CI 速度。于是我们用 nextest 来并行运行测试，并启动了一个共享数据库，确保只需初始化一次，所有数据库测试都能复用它。So this is something that needs to be addressed on top of CI as well because right now what's happening is agents are of course like generating code at the same time the AI agents are also like finding issues on on your uh code review system and these automated reviews are also findings that you would have to now go back and address. So either you have to pull context locally for your agent or have an agent on CI iterate and fix and humans are able to review whatever is left after the AI took its first pass. And this needs to be part of the equation so you're not wasting precious human cycles on more and more agent generated PRs. Uh here's an example for us. Uh we were refactoring our code base to have faster CI. So we were using next text to parallelize our uh tests and we had this like shared database that we were spinning up to make sure that we could set it up once and all our database tests would reuse it.

但问题在于，我们没有设置一些防护措施，这意味着 CI 非常繁忙时，可能会发生交叉污染，导致测试变得不稳定。Guitar 实际上是一名代码审查员。它在我们合并 PR 之前就发现了这个问题，所以我们之后不必浪费时间处理。GAR 也帮我们修复了这个问题，因为它提出了修复方案、提交了修复，并将工程师列为共同作者。通过这样做，我们能够在代码一路进入……之前避免代价高昂的错误。But the problem was that there were some guards we didn't put in place which meant that when CI gets very busy uh there is like uh crosscontamination and test might become flaky. Guitar actually is a code reviewer. It caught it on our PR before we merged it. So we didn't have to waste time after. And GAR also fixed the issue for us because it suggested the fix, committed it, and added the engineer as a co-author. By doing this, we're able to like prevent expensive mistakes before the code made it all the way into

基于风险的合并与审批策略Risk based merge and approval policies

主要内容。我们在 Qar 做的另一件有趣的事是，随着越来越多客户开始这样做，我们会利用策略和标准来判断何时适合批准变更。很多变更都是常规操作或低风险变更，例如文档更新、依赖项版本升级；当然，前提是它们不会破坏任何 API。安全补丁，甚至是内部工具中一些风险较低的变更，都有可能自动获批。Qar 开箱即支持这些功能，你还可以添加额外的自定义规则，用通俗易懂的英文编写，并据此设置审批和合并条件。我们看到越来越多 AI 原生团队开始采用这些功能。在我们的平台上，我认为各团队采用自动审批和自动合并的速度增长很快，从大型团队到拥有 100 多名、甚至 1,000 多名开发者的团队都是如此。main. The other thing that we do at Qar uh interestingly and as we see more and more of our customers start doing is we're using policies and criteria to decide when is it appropriate to actually even like approve changes. There's a lot of changes that are like routine or lower risk documentation dependency bumps of course if they're not breaking any APIs security patches uh even things like uh lower risk changes for internal tooling that can potentially be auto approved. Uh so guitar supports these things out of the box and you can also add additional custom rules which are plain English that you can use to like create the approval and merge on and we are seeing like teams uh who are like very AI native starting to adopt these more and more. We see I think on our platform the rate at which folks are adopting like the auto approval and automerge grow really quickly across teams even from large teams to like 100 plus,000 plus developers.

这里举个审批策略的例子。如果只是依赖项版本升级和文档变更，我们应该批准它们。对于支付或 o 等敏感内容的任何更改，可能还需要一位人工审批者参与。此外，当然不能让做出更改的智能体批准自己的更改，因为那就不是零信任了。在这种情况下，智能体生成 PR 后，PR 会得到批准。如果出现不符合标准的新更改，我们就必须升级处理，交由另一位审批者审批。当然，这可以通过你自己的智能体来完成；如果你想尝试，也可以使用开箱即用的 Gitas。如今只依赖 CI 还有另一个问题：正如我提到的，在很多情况下，CI 显示绿色并不意味着代码符合规范。在这个例子中，CI 通过了，而且我们还有一些相关元数据，比如重跑时某个不稳定测试的情况是……So here's an example of approval policies. So if it's a dependency bump and doc changes, we should be approving them. Any changes under sensitive stuff like payments or o might need a human approver on top. And then of course you don't want to have the agent who made the change approve its change because that would not be uh zero trust. So in this case as agents are generating PRs the PRs are getting approved. When there is a new change that doesn't meet the criteria then we would have to escalate to another approver. And this can be done with your own agents of course but also gitas out of the box if you want to try it. And the other problem with uh just relying on CI today is that just because CI is green as I mentioned doesn't mean the code is compliant in many cases. So in this case uh the CI is passing and there is some metadata that we have about it on the rerun a flaky test uh is

已测试的提交、覆盖项和审计证据Tested commits, overrides and audit evidence

会失败，这意味着当你强制推送来修复那个问题时，分支现在就会指向另一段代码。因此，这里的问题是，随着时间推移，大多数 CI 数据都会过期。所以，除非你记录了发生过什么以及修复了什么，否则你将无法满足合规要求，也无法审计实际发生过的情况。这就是为什么我认为，智能体的执行轨迹，以及每次推送后的结果之类的信息都非常非常重要。无论你能在 PR 还是其他记录系统中保存什么信息，在你的构建产物过期后，这些信息都将成为事实来源。这里的控制措施是确保更改经过了未参与编写该更改的人员批准和测试。如果你没有这些信息，就可能会有问题。举个例子。左边是一个失败的流水线，另一个是通过的流水线。going to fail which means that when you force push to fix that thing now the branch is pointing at another piece of code. So the problem with this is that over time most CI data is going to be expired. So unless you have a record of what happened and what was fixed, you will not be able to like be compliant or be able to audit what actually happened. Which is why I think things like agent trajectories and like your uh outcomes after each push are very very important. So whatever information you can record on the PR or any other system of record is going to be a source of truth as your artifacts expire. So the control here is making sure that the change was approved and tested by someone who did not write it. If you don't have that information, it could be a problem. So here's an example. We have a pipeline that failed in the left and a pipeline that passed.

但这里的区别在于，如果你能妥善记录失败发生的原因，而且其他人能够批准它，因为这是已知的故障模式，[嗤笑]并且相关记录也随着最终合入的变更一并保存下来，那么这项内容实际上就通过了你的检查。相比之下，如果你没有记录事情的经过，却只是盲目信任提交代码的代理，那就不一样了，因为它可能只是一个质量并不高的测试。所有合规检查都没有留下记录。因此，在更敏感的行业中，除了代码审查之外，你还要确保将所有这些决策记录为规则或 PR 上的合规检查。那么，我们应该保留哪些类型的记录呢？以下是我们会保留的一些记录，我觉得它们只是一些示例。But the difference here is that if you record properly why the failure occurred and someone else was able to approve it because it was a known failure mode and [snorts] it was saved with the change that made it in. This thing actually passes your checks versus something where you don't have a record of what happened but you just blindly trust the agent pushing code because it could be a test that was actually uh not high quality. There was no record of all the compliance checks that were like recorded. So you want to make sure that on the uh more sensitive uh industries you want to have all of these decisions recorded as rules or compliance checks on top of your PR in addition to code review. So the what are some of the kinds of records that we want to keep. So here's a few that we keep on which I think are just examples.

这对不同的人来说可能会有所不同。比如说，是哪个编码代理创建了这个 PR？这个 PR 是从哪里来的？发起它的会话是什么？实际是谁启动的？它可能是由自动化工作流触发的，也可能只是工程师通过提示词发起的。对于运行的每个 CI，都要记录 CI 所验证的提交，以及保存测试报告，而不只是记录状态。还要确保每项发现都有解决过程的记录：是被驳回了？还是进行了跟进？或者是在 PR 中修复了？然后要确保审批人不是由同一个人提示的机器人。还要确保所有覆盖操作都明确注明。如果构建失败，或者跳过了控制代码覆盖率检查，就要确保说明该检查为何失败，以及在这种情况下为何可以接受。It could be different for folks. So for example, which coding agent made uh the PR? Where did the PR originate from? What was the session that originated it? And who actually started it? Could it be a trigger from like automated workflow or could it be just an engineer doing prompting? And for each of these like CIS that run, what is a commit where CI was trusted and a save test report, not just a status? And making sure that each finding has like a record of how it was resolved. Was it dismissed? Was it followed up? Was it like fixed on the PR? And then making sure the approver is someone a apart from the bot that was prompted by the same person. And making sure that any overrides are explicitly named. So if there is a red build or like a control code coverage skipped, you want to make sure there's a reason for when why that check failed and why we're okay with that in that scenario.

并确保实际合并的代码包含所有这些内容，而不是只合并了一个叠加在上面的小改动。这样每次都会执行所有检查。因此，在这个案例中，我们要确保智能体不能批准自己的更改。触发它的工程师也不能是作者。如果现在已经落实了这些实践，我们就可以开始消除一些瓶颈，因为 CI 加速后，构建就不会再受阻。这样一来，整个流程会顺畅得多；而且，如果能把一些检查提前到流程前期，也能大幅减少测试负载。在 CI 方面，如果大部分审查都能自动化，只在特殊情况下才升级给人工处理，甚至还能自动批准低风险或内部更改，那么就能进一步消除瓶颈，减少需要人工处理的事项，让软件工厂充分发挥其潜力。And making sure that the code that merges actually has all this and it's not like a small change on top that was merged. So all the checks are done every single time. So in this case we want to make sure that agents cannot approve their own changes. Engineer who prompted it is also not an author. And if you have these practices in place now we can actually start removing some of these bottlenecks because the CI being faster will unblock your builds. So your flow is much nicer and making sure you can like move some checks left will also make sure that your test load is going to be reduced quite a bit. On the CI side, if you can automate review for the most part and only escalate to humans for exceptional cases and potentially even auto approve lowrisk or internal changes, then you remove further bottlenecks and then you have less that has reaching humans uh for a software factory to perform to its uh potential.

当然，每次审批和合并之后，你们都要记录发生了什么，因为随着工厂逐步扩大规模，如果不了解过去发生了什么，这些背景信息对于你们不断迭代、逐步改进软件会非常有帮助。嗯，我就说这些。也很希望大家试试 Guitar，看看如何改进你们的软件工厂。那就这样，非常感谢你们……And of course after every single approval and merge you want to record what happened because as factories are going to ramp up if you don't have an idea of what happened in the past that context is going to be very useful on how to like iterate and make your software better over time. Uh that's it for me. Uh would love for you folks to also try out guitar and see how you can improve your software factories. And with that uh thank you so much for for your