超越 RAG：从零构建面向组织知识的关系型上下文引擎

简要描述Brief Description

超越 RAG：从零构建关系型上下文引擎 — Peter Werry，UnblockedBeyond RAG: Build a Relational Context Engine from Scratch — Peter Werry， Unblocked

目录Table of Contents

• 不再受阻，以及 RAG 只是检索的一半
  Unblocked, and RAG as only half of retrieval
• 上下文分散，但它有其结构
  Context is scattered, and it has structure
• “我上周合并的 PR”：相似度搜索会搞错的查询
  "PRs I merged last week": the query similarity search gets wrong
• 模型需要的四样东西：架构、身份、验证、重试
  Four things the model needs: schema, identity, validation, retries
• 桩引擎及其聊天标签页
  The stubbed engine and its chat tab
• 通过抽样三百份文档发现架构
  Schema discovery by sampling three hundred documents
• 身份解析：Peter 与 GitHub 登录名的区别
  Identity resolution: Peter versus the GitHub login
• 查询规划：强制工具调用与当前日期
  Query planning: a forced tool call and the current date
• 验证：允许列表、隐藏字段、租户隔离
  Validation: allow lists, hidden fields, tenant isolation
• 重试循环，以及精确匹配失败时的全文搜索
  The retry loop, and full text search when exact match fails
• 聊天代理基于同一个引擎进行推理
  The chat agent reasons over the same engine
• 回顾：可复用的思路
  Recap: the reusable ideas
• 问答：扩展 Schema、数据摄取、Slack、MCP、混合搜索
  Q&A: scaling schemas, ingestion, Slack, MCP, hybrid search

解除阻塞，以及 RAG 只是检索的一半Unblocked, and RAG as only half of retrieval

好的。我是 Peter，来自 Unblocked。这位是我的同事 Brandon。今天，呃，首先我想简单介绍一下我们在做什么。我们构建的是一个上下文引擎。我们公司的名字叫 Unblocked。我们为人类提供组织上下文，当然，也越来越多地为智能体提供组织上下文。 这场分享原本安排了几个小时，现在缩短到了一小时，所以内容相当紧凑。这意味着我会很快地带大家走一遍工作坊。如果过程中没能完全掌握所有步骤，也不用担心，因为代码仓库都在线上，之后你们可以按自己的节奏走一遍。 这里的核心观点是，RAG——今天我们所说的 RAG，指的是语义相似度搜索或向量检索。Okay. So, I'm I'm Peter. I'm from Unblocked. Um, over here is my colleague Brandon. And today, uh, well, first of all, I just want to talk a little bit about what we do. So, uh, we build a context engine. Our company's name is Unblocked. And uh we deliver organizational context to humans and increasingly of course agents. Um so the session was originally a couple hours. It's it's now one hour so it's quite condensed. U which means that I'm going to step through the workshop pretty quickly. If you don't get uh you know all the steps nailed down as we go don't worry about it because the repos up online um and you can run through it at your leisure um afterwards. Um, so the thesis here is that rag and I'm just going to say that rag in our parlance today is going to be semantic similarity search or vector retrieval.

好了，RAG 只是检索的一半，今天我们要构建另一半，也就是把自然语言转换成真正的数据库查询。这里再放一次设置用的二维码，供还没完成设置的人使用。现在我先跳到下一部分，简单展示一下，希望它能正常显示出来。不行，当然不行。给我一秒钟切换一下。哎呀。希望大家现在能看到这个。这是代码仓库。设置步骤很简单。这里有一些依赖项和前置要求，需要大家逐项完成。在整个工作坊期间，我们会依次讲解一系列拉取请求。不过网络基本上已经断了，或者至少没正常运行。我们换这个试试。Okay, rag is only half of retrieval and today we're going to build the other half which is turning na natural language into real database queries. So here's the again the QR code for setup um if people haven't already done so. I'm gonna now skip um on to the next thing and I'm just going to briefly bring up hopefully that renders. No, it won't. Of course. Give me one second to switch. Whoops. And hopefully people can see this. Uh this is the repo here. So setup steps are pretty simple. Um there are a few sort of uh dependencies up here, prerexs that that uh you'll need to run through. Um and the the course throughout the course of the workshop, we're going to be running through a series of pull requests uh and the network is basically gone or at least isn't running properly. Let's try this one instead.

好在这次演讲不需要网络。嗯，希望我们开始之前你已经把所有内容都准备好了。嗯，因为这个显然是行不通的。不过无论如何，我们会按顺序讲解六个拉取请求，它们是逐层叠加的。所以，我会……好了。我会在讲解过程中提到这些。嗯，我先切回这里，这样大家都能看到。然后回到主题演讲。现在我得再把它延长一点了。抱歉。Well, the good thing is we don't need network for this talk. Um, so hopefully you got everything before we started. Um, because this one's obviously not going to work. But at any rate, uh, we're going to be following a series of six pull requests and they stack on top of each other. So, I'll be There we go. I'll be referencing these as we go. Um, and I'll just go back to here so that everyone can see that. And back to the keynote. Now I'm gonna have to extend it again. Sorry.

上下文分散，但也有其结构性。Context is scattered, and it has structure

主持人：好了，可以了。好，希望大家都能看到。很好。那么，我们的智能体需要上下文，而这些上下文散落在工程团队日常使用的各种工具中。你会在 PR、审查评论、Linear 这类工具中的问题和工单、Jira 文档[清嗓子]和维基，以及 Notion、Confluence 里找到它们，当然，还有 Slack 和 Teams 等平台上的对话。这些内容都可以进行语义查询，但它们也有自己的结构：作者、日期、状态、标签、关系等等。你知道，我很喜欢这么说：即使是两年前的思路，在 AI 领域也已经是远古历史了。但即便在两年前，人们的本能反应还是：不如把所有内容都扔进向量存储，检索出来，再重新排序，希望能找到正确的内容。不过，这行不通。Host: There we go. All right. Hopefully people can see that. Great. So um our agents need context and context is scattered across every tool that engineering teams live in. So you're going to find it in PRs, review comments, um issues and tickets like in linear and jura docs [clears throat] and wikis, notion confluence, and of course in conversations and places like Slack and Teams. And all of that is semantically queryable, but it's it also has structure. It has authors, dates, uh, states, labels, relationships, etc. And you know, the the instinct I I love to say this like even two years ago is ancient history in the realm of AI, but the instinct even two years ago was let's just dump this all in a vector store and retrieve it and rerank it and hopefully we get the right thing. Um, but that's that's not going to work.

所以我想先退一步，把这件事和关于上下文的更大讨论联系起来。嗯，在一个提供组织上下文的上下文引擎中，正如我所说，它包含的不只是代码——我们会把所有东西连接起来。因此，左侧散落在你所有代码、PR、文档等内容中的上下文会被输入进来，而我们的目标是从另一端获得彼此相关、有依据的上下文，并且专门针对当前任务进行定制。嗯，在这个模型中，我们不仅要关注语义关系，当然也必须关注连接所有数据的结构关系。所以再说一次，RAG 的做法是对查询进行嵌入，找到含义相似的文档，然后把这些内容塞进提示词里，对吧？嗯，这对内容类问题很有效，比如找出代码、找出修改过它的 PR、找出讨论过它的 Slack 线程；而且，如果把它套进一个智能体循环里，这种方法能做成的事情还挺多。So I'm just going to step back and and kind of relate this to the larger discussion about context. um in in the context of a context engine that serves organizational context um this is this is made up of more than just code as I said we're connecting everything together so the scattered context uh on the left in all your code and PRs docs etc goes in and the idea is to get you know grounded context that relates to each other out the other side that's specifically tailored to the task at hand um and in that model uh we have to care not just about the semantic relationships, but of course the structural relationships that connect all the data. So again, rag is about embedding the query, finding docs with similar meaning, and then we take that stuff and we stuff it into the prompt, right? Well, um, that shines for content questions like find the code, find the PR that changed it, find the Slack thread that debated it, but um, and if you wrap it in an agent loop, you can get pretty far with that.

你也可以把这个东西接入一些能递归遍历文件的工具之类的，它可以作为一个很好的起点。这样你就能在语义理解上抢占先机，然后你的智能体就可以递归遍历 GitHub 上的实际文件系统，嗯，这样你就能做很多事情了。And you can also wire this thing up to like um, you know, tools that can recurse files and it acts as a good jump off point. you get a good semantic head start and then from there your agent can recurse over the actual file system in GitHub um and you can get pretty far

“我上周合并的 PR”：查询相似度搜索结果不正确"PRs I merged last week": the query similarity search gets wrong

不过，根据我们的经验，实际情况并不完全是这样。工程师经常会问这样的问题：“我上周合并的这些 PR”或者“我的团队上周合并的这些 PR”。他们基本上是在寻找某个特定的东西，知道从哪里开始，但不知道目标在哪里，不过他们确实知道其中的关系。举个例子，我们从 RAG 的角度来看看“我上周合并的 PR”。你会把这句话转换成嵌入向量，找到最接近的向量，再对排名前 K 的结果重新排序。这样一来，你可能会找到一些提到上周合并某个演讲的 PR，但却找不到实际的关系，比如作者、合并状态，以及它是否真的是上周合并的。所以最终可能会得到错误答案：找到一个语义上相关的 PR，但它其实是三年前合并的。with that. But in our experience um our this is not really the way things go in practice. engineers often ask questions like this PRs that I merged last week or my team merged last week. Um they're they're basically, you know, hunting for something in particular and they know where the starting point is, but they don't know where the destination is, but they do know the relationships. So, as an example, PRs I merged last week, let's think about this in a rag context. you'd you'd you know embed that sentence, you'd find the nearest vectors, you'd rerank it top K. Uh you get PRs that mention merging about a talk last week, but uh you wouldn't get the actual relationships like the author, the merge state, um and whether it was merged last week actually. So what you could end up with is the wrong answer where you pull up PR that's semantically related but it was merged three years ago.

嗯，所以这实际上并没有回答这个问题，因为这个问题有一个结构层面，还有一个时间层面。因此，检索在这里分为两部分：语义检索，另外，今天我们要讨论的是结构化查询。关键在这里：我们要让模型……嗯，在这种范式下，模型实际上并不回答这个问题。它会写出……Um and so that doesn't actually answer the question because there's a structural there's a temporal aspect to that question. And so retrieval has two halves here semantic and again today we're going to be talking about this structured queries. And here's the key. We're going to take uh the model doesn't actually answer the question in this paradigm. It writes the

模型需要的四样东西：模式、身份、验证、重试Four things the model needs: schema, identity, validation, retries

访问数据库的查询。好吧。那么，如果说生成查询就是我们唯一需要做的事，那就太天真了。为了生成查询、确保生成过程可靠，并且或许还要考虑一些安全问题，我们需要关注这里列出的这四点。首先，LM 在创建查询之前需要了解数据库架构。它还需要理解数据中的一些关系。举例来说，如果你传入一条自然语言查询，说“给我看看 Peter 尚未合并的 PR”，它就不知道该怎么处理，因为它会生成一条使用 Peter 作为用户名的查询，而实际的 GitHub 登录名可能并不是 Peter。因此，它需要能够将两者对应起来。当然，最终生成的查询中也可能会有一些不可信的内容。query that accesses the database. Okay. So, um it would be naive to say that generating the query is the only thing that we have to do. Um in order to generate the query and to do it reliably um and and to take care of maybe even some security concerns, we need to look at all four of these points here. So that first of all the LM needs to be aware of what the schema is before uh creating the query. It needs to understand uh some of the relationships in the data. So for example, if you pass a natural language query that says show me Peter's open PRs, um then it's not going to know what to do because it'll synthesize a query that says Peter when really the GitHub login name might be different. So it needs to be able to map that across. Um, and of course you're going to have some uh untrusted queries coming out the other end.

嗯，也许使用 in parlance 中的函数运算符，它可能会直接在服务器上执行 JavaScript，而我们大概不希望这样。嗯，最后，如果第一次运行时出现错误，我们希望能够自行纠正这些问题。下面是一次性完成后呈现的样子。我们想要实现的是这样的效果。嗯，它会映射……我同事的名字叫 Rashine。他的用户登录名是 Rashab。然后我们还希望能够映射诸如 state 这样的内容，并获取 limit 和 sort 之类的运算符。所以这就是我们要构建的东西。我们将从一个完整的 UX 和一个存根引擎开始，嗯，这个引擎一开始还不具备相应能力，然后我们会逐步完善它。到了第三步，这个东西就会真正运转起来。这样我们就能向它发出查询，看看它会做什么。Um, maybe using the function operator in the in parlance which might execute JavaScript directly on the server and we probably don't want that. Um, and then finally, if there are errors the first time around, we want to be able to self-correct those things. So here's here's what this looks like in one single shot. What we want is something that looks like this. Um it it maps uh my colleagueu's name is Rashine. His user login name is is Rashab. Um and then we want to be able to map things like the state and get maybe uh operators like limit and sort. So uh this is what we're going to build. We're going to start with a full UX, a stubbed engine, um, that is not capable to start with, and we're going to gradually build this up. And then at step three, the thing will come alive. And so we'll actually be able to issue queries to it and see what it does.

还有，呃，如果你之前没看过，现在正好可以扫一下，把它弄进去。我在这里帮你。And uh, if you hadn't seen that before, now is another good time to scan that and get it in. I'm here to help you.

嘉宾：是的。Guest: Yeah.

主持人：是的，任何代码仓库都行。哪个都无所谓。嗯，如果你想走捷径，也可以直接让一个代理处理这件事，让它帮你把所有东西都配置好。这样也可以。Host: Yes. Any repo is fine. Doesn't matter which one. Um and if if you want to take a shortcut, you can just point an agent at the thing and let it set everything up for you. That works.

被存根化的引擎及其聊天标签页The stubbed engine and its chat tab

好，我们开始吧。首先我们要做的是这个。嗯，到现在你应该已经运行过这个数据导入流程了。如果还没运行也不用担心。嗯，还是那句话，你可以最后再回来处理。我先快速展示一下这个东西最精简的界面是什么样的。现在这个东西已经在运行了。如果我在这里输入任何内容，显然是不会起作用的。嗯，你应该会看到类似这样的东西。嗯，还有一个聊天标签页。这是一个已经完整实现的智能体，可以替你完成工作。我们稍后再回到这个。这个就是代码里的接口约定，你可以去看看。嗯，这部分不用担心，我们会跳过。嗯，基本上它就是入口，我们会围绕它来构建。Okay, let's move. So, this is what we're going to do to start with. Um, by now you you probably should have run this ingestion thing. Don't worry if you haven't. Um, again, like you can revisit this at the end. I'm just going to quickly show what the the kind of barebones interface looks like for this thing. So, now this thing is running. Now, if I type anything in here, this is obviously not going to work. Um, and what you should see is something like that. Um, there's also a tab for chat. This is a fully implemented agent uh that can uh do work for you. So, we'll come back to that in a sec. And so, this this is the contract in the code if you wanted to read. Um, we're going to don't worry about this. We're going to skip through this. Um, but basically like that's the entry point and we're going to be building around that.

那么，我们首先想谈的是So the first thing we want to talk

通过抽样三百份文档发现数据模式Schema discovery by sampling three hundred documents

这部分讲的是模式发现。嗯，在摄取过程中，如果你看一下摄取脚本的工作方式，就会发现它实际上必须先构建一个模式，才能下载并存储内容。嗯，不过你可以想象，你可能并不知道这个模式。所以，摄取可能发生在别处的某个服务中，而你想要构建一个完全动态的模式发现引擎。嗯，也就是无模式的情况。因此没有模式可供读取。嗯，不像 PG 之类的数据库，我们可以读取表定义。嗯，所以我们要做的是对文档进行采样，以推断出模式，这就是整个诀窍。我们只需要把它重建出来。我们会取出所有节点并将它们合并起来。所以这里会对 300 个文档进行采样，并将它们折叠成一棵树。嗯，空值会被跳过，因为缺失字段无法用于约束类型。about is schema discovery. Um during ingestion, if you if you look at the way that the ingestion script worked, it it actually had to build out a schema in order to download the content and store it. Um but you can imagine that you may not know the schema. So the ingestion may be happening in a service elsewhere and you want to build like a fully dynamic schema discovery engine. Um schemalis. So there's no schema to read. um unlike you know PG or something where we could read the table definitions. Um so what we'll do is we'll sample the documents to infer the schema and that's the whole trick. So the we're just going to reconstruct it. We're going to take um all the nodes and merge them down. So this is going to sample 300 documents, collapse it into a single tree. uh nulls are skipped because missing fields won't be able to constrain the type.

然后，呃，我们要在这里用枚举类型做一些挺有意思的事情。现在，呃，有了枚举类型，我们就能用一个挺有趣的过程式技巧。这只是其中一种做法。我建议用过程式技巧来实现这类事情，因为你可以把内容交给 LLM 来做提取，这样确实可行，呃，但结果相当不确定。因此，你始终需要某种结构化的框架，通过运行过程式代码来验证提取出的内容，或者直接用它来完成提取。在这个示例项目中，负责提取的是过程式代码。我们来看看。接着运行一次。现在我们得到的是这个东西。也就是说，schema 已经运行了，呃，发现流程也运行了，并为我们创建了一个 schema，概述了 GitHub 中拉取请求的 schema 是什么样的。And then uh we're going to do something kind of interesting here with enum types. And so now uh with with enum types, we can do an interesting sort of procedural trick. This is just one thing. I would suggest using procedural tricks to to do this type of thing because you could pass it through an LLM and do extraction and that works u but it's fairly non-deterministic. So you always need some kind of structural um harness around it that is running procedural code to either validate the things that you pull out or to actually do the extraction itself. And in this sample project um it's the procedural code that's doing the extraction. And let's go and show that. So we'll do and then run it again. And now what we get is this thing here. So the schema ran uh the the discovery process ran and it created a schema for us that outlines what the pull request schema looks like in GitHub.

它甚至还帮我们创建了一些索引。现在，这里是故意用 Markdown 格式标记出来的，因为It even created some indexes for us. Now, this is like marked down on purpose because

身份解析：Peter 与 GitHub 登录名的区别Identity resolution: Peter versus the GitHub login

我们会把这个 schema 传给 LLM，让它在生成实际查询时使用。第二步就是我几张幻灯片前提到的身份解析。这是因为，比如说，我的 GitHub 登录名和 Peter 的很像，但又不完全一样。所以，如果我问“嘿，找出 Peter 提交的所有 PR”，但没有进行某种身份映射，它就只会匹配 Peter 这个名字，结果什么也找不到。所以我们得想办法解决这个问题。[吸鼻子] 再说一次，我们会这么做。乍一看，这里似乎有点依赖 schema。不过，要动态地确定这些映射完全是可行的。你可以让 LLM 基本上为你生成这些映射。we're going to take this schema and pass it to the LLM when we synthesize the actual uh query. Now, the second step is the thing that I referred to a few slides ago about identity resolution. And this is because like my lo GitHub login um is similar but it's not quite the same. And so if I were to ask a question like um hey find all the PRs by Peter um it if without doing some sort of identity mapping it would just say you know it would just match across Peter and it's not going to find anything. So we have to be able to solve that. [snorts] And again, um, we're going to do this. At first blush, this seems like we're coupling to the schema a little bit here. Um, but you know, it is absolutely possible to determine this dynamically. And you can have uh your LLM essentially synthesize the mappings for you.

不过在我们的案例中，我们只会用这些简单的小技巧。嗯，就是非常简单的模糊匹配。嗯，莱文斯坦距离就可以。你可以直接按名称进行匹配，然后就行了。哦，其实在继续讲那个之前，我想先给你看一样东西。我们再做一遍同样的操作。可以调出来。So, but in in our case, we're just going to do these simple little tricks. Um very simple sort of fuzzy matching. Um Levenstein distance is fine. You can just like match across the names and uh there you go. Oh, actually I want to show you one thing before we get on to that one. So let's do the same thing again. We can pop up

嘉宾：哦，对，不好意思。当然。等它变大时告诉我。好，谢谢。提醒得好。好，那我们再运行一次我们的流程。现在，呃，哦，实际上，我得等到下一步才行。抱歉，这部分还没接通。所以，我们继续做真正能带来收获的部分，也就是第三步。Guest: Oh yeah, I'm sorry. Absolutely. Tell me when it's big. Good. Thanks. Good call out. Okay. So, let's run our thing again. And now, um Oh, actually, I have to I have to wait till the next step. I apologize. That's not going to be wired up just yet. So, we'll go on to the thing that will really give us the payoff here, which is step three.

查询规划：强制工具调用与当前日期Query planning: a forced tool call and the current date

所以这就是实际的查询计划步骤，可以说它算是最简单的部分。它是 LM 部分。嗯，我们在这里会传入 schema，提供一个工具，并强制它调用该工具来进行名称提取和合成。嗯，之后它就会执行计划。还有一点，就是时间因素。如果把当前日期和时间传给 LLM，它们其实很擅长生成时间查询。所以在提示词中加入这一点也很重要。这里就是名称提取部分。在这个例子中，我们会强制它调用工具。现在，我来给你看看这样做的效果。好了，现在我可以这样问：“打开 Peter 创建的 PR”。然后，砰。我们看到它生成了这个匹配子句。你在这里看不到，但它把 Peter 映射到了我的 GitHub 用户登录名 puery，并生成了查询，这就是最终结果。So this is the actual query plan step and it's it's it's arguably sort of the simplest part. It's the LM part. Um and what we do here is we pass the schema give it a tool and force it to call that tool for name extraction and synthesis. Um and then it will execute the plan. There's one other thing which is the temporal aspect. uh LLMs are actually pretty good about generating temporal queries if you pass them the current date and time. So that's another important thing to do in the in the prompt. And so here this is the name extraction part. We're going to force that tool call to happen in this case. And at this point I am going to show you the the payoff for this. So there we go. So now I can do things like this. open PRS by Peter and bam. So what we'll see here is that it uh generated this um this match clause and uh you can't see it from here but it it mapped Peter to my GitHub user login puery and generated the query and that's the outcome.

这很酷，不过，嗯，我们得进入下一步，因为尽管我们……So that's cool, but um we need to get on to the next step because even though we

验证：允许列表、隐藏字段、租户隔离Validation: allow lists, hidden fields, tenant isolation

现在能够查询数据了，但我们也有一些安全方面的顾虑。嗯，可能会产生不可信的输出，因此它可能会使用 where 或 function 子句，直接在服务器上运行 JavaScript，这意味着现在这里成了一个提示注入点。嗯，你的 schema 中也可能有隐藏字段，比如 ID 或元数据字段，从而泄露内部信息。所以我们需要进行清理，嗯，而且我们不会通过拒绝列表来处理这些内容，而是使用允许列表。也就是说，默认拒绝。你基本上需要创建一个允许的操作符列表，以及，嗯，可能还需要创建一个允许的字段列表。就我们而言，这是动态 schema 发现，因此我们不会这么做。嗯，不过我们可以指定一些已知字段，比如元数据字段。这对于租户隔离之类的场景很有用。are able now to query the data, we have some safety concerns. Um it could be the case that it produces untrusted output and so you know it could use the wear or function clauses and literally run JavaScript on the server which means that this is now a prompt injection point. Um, you could also have hidden fields in your schema like ID or metadata fields that leak internals. So, we need to sanitize um, and we don't deny list these things. We allow list them. So, it's deny by default. You want to create basically an allow list of allowed operators and uh, you know, potentially an allow list of allowed fields. In our case, this is dynamic schema discovery, so we're not going to do that. Um, but we could have some fields that we know about like metadata fields. And this is a a useful thing for like tenant uh tenant isolation.

所以，如果你有一个承载多个租户的集合，我不建议这么做。你最好按租户拆分集合。不过，如果你确实这么做了，那么可以设置一个包含租户 ID 的元数据字段。你需要对 LLM 隐藏这项信息，同时也要验证输出。如果它输出了这个元数据字段，就需要把它清除掉，然后在下面注入你自己的元数据字段。再说一次，从技术上讲，这其实并不难，你只需要编写代码就行了。这一切都是按步骤处理的。我不建议在这部分使用 LLM。所以，还是用允许列表，不要用拒绝列表——或者说，拒绝一切。最后一步会自然衔接到下一步。验证结束时，我们显然会返回一个错误，然后把它输出出去。So if you have a a uh collection that houses multiple tenants, I don't recommend that. you should probably separate your collections by tenant, but if you did, then you could have a metadata field that contains the tenant ID and you would want to hide that no that um information from the LLM, but also validate the output. So if it outputs that metadata field, you need to wipe it out and then you need to inject your own metadata field underneath. And again, like technically this is it's not actually that difficult to do. You just have to write the code. It's all procedural. Um I I wouldn't recommend using LLMs for this part. And so again, allow list, don't deny or deny everything, I should say. Now the last part is um we're going to this kind of dovetales into the next step. At the end of validation we're obviously going to return an error and it's going to get dumped out.

目前这个错误就只是……在那里而已。我们暂时不会对它做任何处理。不过现在我来给你看看它是什么样的。嗯，哎呀。停留在那个页面上，然后运行我们的服务。现在我要再编写一个查询，检查标题中是否包含 authentication，而我们已经禁止使用正则表达式。所以现在可以看到，它拒绝了这个查询。And right now that error is just it's just there. We're not going to do anything with it just yet. But I will show you now what this looks like. Um, oops. Stay on that one and run our service. So now I'm going to write run another query that checks for authentication in the title and we've disallowed regular expressions. So now what we can see is that it rejects the query.

重试循环，以及精确匹配失败时的全文搜索The retry loop, and full text search when exact match fails

最后，把这一切串联起来的就是重试循环。计划可能会以几种不同的方式失败，对吧？嗯，它可能无法通过验证。也就是说，LM 实际上可能会输出一些不符合我们验证器规则的内容。它也可能在执行过程中失败。比如说，我们假设它把一个包含项目和排除项目混在了一起。嗯，那显然会失败。不过我们可以自我纠正。我们可以把错误传回给 LLM，告诉它：“嘿，这样不对。”嗯，当然，我们可以让它改进，直到达到最大尝试次数。接下来我会切回我们的项目，演示一下效果，这样我们就能往上看一层了。哎呀，不好意思。请见谅，我其实得照着这边屏幕上的内容念。And finally, this is the thing that kind of ties it all together is the retry loop. The plan can fail in a couple of different ways, right? Um, it can fail validation. So maybe the uh the LM actually outputs things that are illegal according to our validator. Or it could actually fail during execution. Um, so for example, let's imagine that it it mixed an inclusion exclusion project. Um, then that's obviously going to fail. And then but we can self-correct. We can take that error and pass it back into the LLM and say hey that was wrong. Um maybe you can do better up to some number of max attempts of course. So I'm going to show now what this looks like back over to our project so we can punch up a level. Whoops. Sorry. You have to forgive me. I actually have to read this off of the screen out here.

好。现在，如果我们运行同一个查询。很好。我们可以看到，经过了几次尝试才得到正确结果。不过，到了第三次尝试时，这里出现了一点问题，因为我们说过不要用正则表达式，也不要用 FTS。于是现在它说要直接使用 match，但这样就变成了精确匹配。因此，它会尝试查找“authentication fails”。这意味着我们还可以再做一件事，就是这样：添加全文搜索。这很基础。显然，我们需要创建一个索引，并在同一个 match 中结合作者和状态筛选条件。这样一来，我们应该就能回答这样的问题了。来看看结果如何，看看它是否真的会按我说的那样执行。Okay. So now if we run that same query. Perfect. So, what we can see is that it took several attempts to get it right. Um, by the third attempt though, there's a bit of a problem here because we said no regular expressions, no FTS. Um, so now it said we're going to just use match, but now it's an exact match. So it tries to find authentication fails which means there's one more thing that we can do which is this. We'll add full text search. It's pretty basic. Um we need to create an index obviously and we'll combine it with author and state filters uh in the same match. And so now we should be able to answer a question like this. And let's see what happens. And see if it is actually going to do what I said it's going to do.

聊天代理基于同一引擎进行推理。The chat agent reasons over the same engine

好了，就是这样。现在最终的效果是，我们进入聊天选项卡后，就可以提出一些更复杂的问题，让智能体在幕后进行推理。比如，你可以问：“我们上周做过与身份验证相关的工作吗？”这里可以看到，它生成了查询。现在它正在提取数据，并对数据进行推理。好了，就是这样。Bam. There we go. And so now the the final payoff here is if we go into the chat tab, we can ask some sophistic more sophisticated questions that um you know an agent can actually reason about behind the scenes. So you can say things like um did we work on anything authentication related last week. So here we can see that it generated the query. Now it's pulling the data and kind of reasoning about it. There we go. That's it.

回顾：可复用的理念Recap: the reusable ideas

简单总结一下我们构建的内容。我们基本上快到尾声了，还会留出很多时间提问。这里简单回顾一下：有些问题是 RAG 无法处理的。比如有关状态或时间性的结构性问题。用传统 RAG 回答这类问题时，无法获得正确的数据。因此，查询体验区和聊天代理背后的引擎是一样的。大家之后想试用的话，尽管去试。它们用的是同一套底层机制。当然，唯一的区别是，聊天代理会进行推理，而查询界面只是用来查询。这里再总结一下可复用的思路：具备架构感知能力的提示、动态架构发现、身份解析、验证和错误重试。So, just a quick summary of what we built. We're basically like reaching the end. There's going to leave lots of time for questions. So just to recap here, um there are questions that that rag can't touch. All those structural questions around things like state or temporality. Um you don't get the right data out when you ask those types of questions with rag, traditional rag. So the same engine behind the query playground and the chat agent, uh they're they're the same. So if you guys want to play with that after, go ahead. Um it's it's the same backing thing. The only difference is of course that uh the chat agent actually does reasoning whereas the query interface is just for queries. So just to to recount on this here are the reusable ideas here. Schema aware prompting dynamic schema discovery identity resolution validation and error retries.

错误循环实际上是最重要的部分之一。所以我们构建了一个具备筛选和聚合功能、支持按时间范围限定、还能进行全文搜索的系统。嗯，它经过了验证，能够自我纠正，还能为一个功能完整的聊天代理提供支持。因此，你可以把它带到自己的组织里，直接部署。我建议在内部部署之前，最好再多花点时间完善验证框架，不过，嗯，它确实能回答所有关于你们 PR 的问题。就这些。我们现在留了很多时间来提问。大家都成功设置好了吗？我只是想了解一下。没有？好吧。如果你们愿意，之后可以来找布兰登和我，我们会在这儿外面待一会儿。哦，也可以去 P16 展位找我们。The error loop is actually one of the most important things. And so we built something that's capable of filtering, providing aggregations, it's time scoped, has full text search. Um, it's validated, it self-corrects, and it powers a full-blown chat agent. So you could take this into your organization and just deploy. I I would recommend maybe working a little bit more on the validation harnesses before deploying this internally, but um yeah, it'll it'll answer all your questions about your PRs. And that's it. So, we've left lots of time for questions now. Did Did everyone uh was everyone able to actually set it up? Just curious. No. Okay. If if you want uh you can come and fetch Brandon and I afterwards, we'll just kind of linger outside here. Oh, at the and at the P16 booth as well.

是啊，不过那个要到今天晚些时候，大约四点才开始，我们会……Yeah, but that that starts later today at around four o'clock and we'll be

问答：扩展架构、数据摄取、Slack、MCP、混合搜索Q&A: scaling schemas, ingestion, Slack, MCP, hybrid search

整个星期都在那里。there all week.

主持人：好。有问题吗？Host: Cool. Any questions?

嘉宾：是的，Guest: Yes,

主持人：我有个问题。我觉得你把基础知识讲解得非常好。Host: I have a question. I think you did a great job breaking down like basics.

嘉宾：是的，Guest: Yeah,

主持人：我脑子里经常会想，好吧，那如果要大规模应用，比如说，你们是怎么生成 schema 并把它放进系统提示词里的？这要怎么扩展到大规模？而且这也取决于公司的规模：你们怎么容纳所有这些数据？怎么决定选取哪些数据？处理完所有这些数据要花多长时间？成本又是多少？我关心的就是这些。Host: a lot of my mind goes like okay so at scale how does for example you you generate the schema and you put it into the system prompt how does that work at scale? It's like also depending on the size of your company, how are you fitting all this data? How do you choose what data and like how long does it take to process all this? What's the cost? That's where my mind is.

嘉宾：是的，这是个好问题。嗯，我先重复一下这个问题，让大家都能听到。演示显然展示了一个相当刻意设计的例子，只专注于拉取请求。嗯，所以问题是，如何扩展这种做法？比如，如何在多个不同的集合、多个不同的表等之间实现这一点？嗯，你可以在上面再加一个 LM 步骤，为这些架构集合准备一些简短的摘要和描述，然后根据查询内容加载对应的架构。LM 在这方面出奇地擅长。一年前还不是这样，但现在它们做得非常好了。嗯，这就是缩减上下文的办法，也回答了你关于如何扩展的问题。嗯，你对扩展还有其他方面的疑问吗？Guest: Yeah, it's a good question. So, you know, um I'm just going to repeat the question so everyone can can hear it. The demo showed obviously a fairly contrived example where you're just focusing on pull requests. Um and so the question is how do you scale this? Like how do you do this across multiple different collections, multiple different tables, etc. Um, so you you can add uh another LM step on top where uh this the schema collections have tiny little summaries and descriptions and then uh depending on what the query is, you can load the right schema for that query. LMS are surprisingly good at this. They weren't a year ago, but now they're very good at it. Um, so that's kind of the way to shave the context thing down. And that's to your question about scaling things out. Um, was there another aspect of scaling that you were interested in?

主持人：把所有数据都导入进来，我想这其中一部分应该是成本。Host: Getting all the data in and I guess like part of that is like cost.

嘉宾：对，还有活力之类的。Guest: Yeah, it's livveness and stuff too.

主持人：做这个大概要花多长时间？我该选什么数据放进去？Host: How long does it take to do like what is what data should I pick to put into this?

嘉宾：是的，这是个好问题。Unblocked，呃，Unblocked 会持续摄取内容。比如说，对于 PR 中的所有事件，我们都会收到 Webhook 回调。每次发生变更时，我们都会收到那个 Webhook 事件，摄取内容，进行 rem……，然后存储到数据库中。呃，这实际上就是一个分布式系统问题：如何从摄取层面扩展系统。存储也是一样。所以，是的，Guest: Yeah, it's it's a good question. So unblocked um at unblocked like we're continuously ingesting content. Uh we get web hook callbacks for all the events for example in in PRs. And so every time there's a change, we get that web hbook event, we ingest the content, we rem um and store it in the database. Um that that's just really just like a distributed systems question about how you scale that out from an ingestion level. Um you know, storage, same thing. So yeah,

主持人：那么，也许这样会让它更具体一些。你会不会说，比如，这些就是三个最主要的数据来源——像是 GitHub 之类的，或者其他什么？Host: so maybe this will make it more concrete. So what if you would say like these are the top three data sources like someone seems like GitHub maybe like something

嘉宾：嗯，Slack 肯定很重要，很多“为什么”类问题和决策都是在 Slack 上发生的。嗯，在这种结构化查询的场景下，人们通常会想到过去进行过的对话，然后就能向 Unblocked 发起查询，说：“哦，对，我大概三周前和某某聊过这件事。”所以，这正是结构化查询的理想用途。嗯，我会说，Slack 对回答“为什么”类问题最重要。嗯，很多决策背后的原因也记录在 PR 里。PR 的评论中包含了所有这些组织历史，但这些内容基本上就像掉进了黑洞，对吧？所以，你可以在向用户提供这些数据之前先对其进行处理，从中挖掘有价值的信息。比如说，在 Unblocked，我们会创建一种叫作“最佳实践记忆”的内容，它们是从过去的 PR 评论中提炼出来的。Guest: um so Slack for sure a lot of the like why questions and and decision- making happens in Slack um people uh in the context of this sort of like structural query um people usually think about conversations that they've had in the past um and they'll be able to query unblocked and say Oh yeah, you know, roughly three weeks ago, I was having a conversation with so- and so. And so that that's a perfect thing for structural query. Um yeah, like I would say that um Slack is the most important for why questions. Um a lot of the why decision-m is also embedded in PRs. PRs uh have all of that like organizational history in the comments and that's just basically black hole, right? So uh you can basically run over that data before you serve it and and kind of mine it for interesting insights. So for example at unblocked we we create these things called best practice memories which are distilled from PR comments in the past.

这样说得通吗？Does that make sense?

主持人：是的。嗯。还有别的吗？是的。Host: Yeah. Yep. Any else? Yeah.

嘉宾：问题。直接说到这栋建筑是如何……[嗤笑]Guest: Question. directly into how this building. [snorts]

主持人：是的，我的意思是，你当然可以这么做。如果你有一个结构化查询——如果你有一张结构化查询表，就可以通过 MCP 来表达。你甚至可以在底层架构每次发生变化时动态更新工具来实现这一点。嗯，这类东西展示了 MCP 调用另一端会发生什么。比如，你可以有一个只接受自然语言查询的 MCP 接口，然后在后端把这些查询全部转换好；或者也可以直接提供数据库接口。这取决于客户界面是什么样的，不过，没错。是的。Host: Yeah, I mean you you can definitely do that. So if you have a structured if you have a structured query table, you could express that through MCP. You can even do that dynamically by updating the tool every time the schema changes behind the scenes. Um this this type of thing is showing like what happens on the other end of that MCP call. So you you could you know have an MCP interface that just takes natural language queries for example and then translate that all on the back end or you could provide a direct interface to the database. It depends on like what the customer interface looks like, but yeah. Yeah.

嘉宾：是的。Guest: Yeah.

主持人：是的。Host: Yeah.

嘉宾：是的。对。第二部分更好。关于……的想法Guest: Yeah. Yep. Second part is better. Thoughts around

主持人：呃，你是想问怎么组合它吗？还是说，让我先确认一下我是否理解了你最后的问题。Host: uh how to mean how to combine it or let me just make sure I understand your last question.

嘉宾：是的。感觉这就像……Guest: Yeah. feel like this is like

主持人：不，完全不是。没有没有，它们绝对是相辅相成的。对，对。我的意思是，你可以用全文搜索来做这类事情，你知道，就像这就是全文搜索，对吧？所以它会识别关键词。不过你仍然需要某种机制来进行真正的语义相似性搜索，因为你们那里有一些想法和句子，它们经过编码后并不一定能在文本上匹配。所以你需要这种概念层面的匹配，对吧？所以就像我一开始说的，相似性搜索其实非常适合作为切入点。比如，你可以先进行向量搜索，然后再让它使用这类工具，因为这样它就能获得一些有意思的初始数据，并据此构造查询，去获取数据。Host: No, not at all. No, no, they're absolutely complimentary. Yeah. Yeah. I mean, you can have a full text search that does this kind of um you know, like that's full text search, right? So, it'll identify keywords. You still need some kind of mechanism to to do like the actual semantic similarity search because there you all have ideas and sentences that are encoded that aren't not ne necessarily textual matches. So you want you want that like conceptual matching, right? So as I said um at the beginning like there's similarity search is actually really powerful for jump off points. So you can for example start with a vector search and then from there give it access to tools like this because then it'll actually have like some interesting starting data and help it form queries to go and and fetch the data.

所以，是的，这确实是我们会做的事情。抱歉，你问题的第一部分是……So yeah that that's that's something that we do and so sorry the first part of your question was

嘉宾：所以你会拿……你知道，在任何情况下，这些都是文档，它们会以某种方式组成 JSON，这样你就可以把这些东西合并起来，查找特定的 token，然后你知道，提取这些 token，再通过这种方式提炼出类型。所以你可以看看源代码，就会发现至少这个算法的当前版本相当粗糙，对吧？它会做一些事情，比如确保类型信息的字符数介于 2 到 8 之间；至于枚举，它也会查找特定的模式。Guest: so you take um you take you know in in anyways like these are all documents and they're they kind of like form into JSON so you can merge those things down and look for certain tokens and then you you know, extract the tokens and then that's how you distill the types. So you you can look through the source code and you'll see that at least like this iteration of that algorithm is pretty hacky, right? It'll do things like look to make sure that the character count is between two and eight for uh for type information and then for enumerations it's it's looking for particular patterns as well.

主持人：是的。Host: Yeah.

嘉宾：嗯。是的。Guest: Yeah. Yes.

主持人：嗯。Host: Yeah.

嘉宾：现在，这个词的含义非常宽泛，所以我想先确认一下自己准确理解了你的意思。你说的混合搜索，是指由某个服务提供商同时进行语义相似度搜索和词法搜索，然后再进行排名融合，对吗？对。哦，对。是的。不是，所以，我们确实会这样做。答案是肯定的。对。Guest: Now, that that's a very overloaded term, so I'll just make sure I understand exactly what you're talking about. Do you mean hybrid search in terms of um a provider that does semantic similarity and then also does a lexical search and then does rank fusion? Yeah. Oh, yeah. Yeah. No. So, so we we do that. Yes. The answer is yes. Yeah.

主持人：是的。Host: Yeah.

嘉宾：是的。Guest: Yeah.

主持人：呃，就是纯粹靠经验反复试错。所以，我的意思是，这个问题没有什么简单的解决办法。嗯，在幕后，比如在 Unblocks，我们会使用很多静态模式，因为我们想把正确性做到绝对可靠。我想展示这个技巧，是因为你知道，在探索数据、进行实验，或者连接任意数据库时，这是个不错的技巧。我们通过和客户反复试验、不断迭代，摸索出了这个办法。我们会从文档存储中摄取大量内容，然后收到客户的反馈，说：“哦，你漏掉了我想找的内容。”接着我们会深入调查，弄清楚原因，最后大致确定了最合适的样本量。Host: Uh just empiric empirical trial and error. And so I mean there's there's no easy solution to this. Um be behind the scenes like at unblocks we're we're using a lot of static schemas because we want to get the correctness absolutely nailed down. This this was like a just a technique that I wanted to show because um you know for exploration of data or experimental um uh for experiments or either to wire up like arbitrary databases this is a good technique. So we we got this through some trial and iteration with customers and you know we we'd ingest a whole bunch of contents from the document store and we'd get responses back from customers that said oh you missed the thing that I was searching for and we'd dig into it and we'd figure out why and we kind of got to the sample size that made the most sense.

是的，Yeah,

嘉宾：这绝对是真的。对，肯定是这样。你总会——实际上，你肯定会漏掉一些东西。总会有某份文件里有点……比较大胆的内容。对。还有其他人吗？我刚才好像看到那边有人提问。哦，对。不好意思，你能把开头那部分再说一遍吗？Guest: it's absolutely true. Yeah, definitely. You're always actually gonna you're going to miss something. There's always going to be some document that has a little edgy thing. Yeah. Anyone else? I thought I saw a question over there. Oh, yep. Sorry, can you say that first part again?

主持人：有人提到你了解这个架构。Host: There's mention that you understand the schema.

嘉宾：呃，是的，我们的做法是强制调用某个工具，但这不是为了发现模式，因为那会在一开始就发生。它是为了提取名称。我们有一个验证器，用来监视智能体在做什么。比如说，如果它没有调用第一步，也就是名称提取……嗯，实际上有两种方法。一种是创建一个执行查询的工具，并将用户名作为工具参数传入。这样，如果用户名不正确或格式错误，你就可以当场报错。嗯，如果你想通过工具调用来提取名称，那就得监视智能体的操作。然后，如果它没有调用这个工具，你只需插入一条用户消息，说你忘了调用提取工具，现在就调用。对。Guest: Uh yeah, what we do is we force a tool to be called for not for schema discovery because that happens at the start. It's for name extraction. we so we have a validator that watches what the agent's doing. So like if it doesn't call the the first part which is name extraction. Um so there's there's actually two techniques for this. One is you create a tool that does the the query and as an argument to the tool you accept a username. So you can fail right there if the username is not correct or malformed. Um, if you want to do name extraction uh via tool call, then you have to watch what the agent does. And then if it doesn't call it, you just basically insert a user message that says you forgot to call the extraction tool. Do so now. Yeah.

主持人：是的。你如何应对这些变化？是的。Host: Yeah. How do you handle the changes? Yeah.

嘉宾：是的。所以请记住，在这个特定示例中，schema 完全是未知的，因此随着文档不断添加，schema 可能会更新。所以你们可以选择按一定的采样间隔重新生成 schema。以我们在 Unblock 的情况为例，schema 是静态的，并且与我们的代码库绑定；实际上，schema 由我们控制。如果 GitHub 决定更改 schema——当然，这种情况很少发生，因为那会对他们的客户产生重大影响——但如果他们确实这么做了，我们就会直接采用他们做出的更改，然后显然通过 CI 部署。好，还有其他人吗？Guest: Yeah. So remember like for this particular example the schema is is completely unknown and so as documents get added um that schema could update. So you your your choice is either you know on a sampling interval you you rerun the schema generation um in our case at unblock the schema is is static and it's tied to our codebase and we control the schema actually. So if the schema of if GitHub decides to change the schema um that of course rarely happens because that would have serious implications for their customers but if they did then we would just adopt whatever changes are made statically and then obviously deploy that through CI. Yeah. Anyone else?

主持人：太好了。我想这样的话我们可以，呃，节省一些时间。谢谢。Host: Awesome. I think we can uh reclaim some time in that case. Thanks.