让智能体掌控自己的推理：Du’an Lightfoot 与 Khaja Omer 的 Akamai GPU 部署实战

简要描述Brief Description

能够自主掌控推理的智能体 — Du'an Lightfoot 和 Khaja Omer，阿卡迈科技Agents That Own Their Inference — Du'an Lightfoot & Khaja Omer， Akamai Technologies

目录Table of Contents

• 认识杜安·莱特富特和卡贾·奥默尔
  Meet Du'an Lightfoot and Khaja Omer
• 智能体循环与推理所有权
  Agent loops and inference ownership
• 何时使用托管 API 或专用 GPU
  When to use hosted APIs or dedicated GPUs
• 研讨会日程与 Kubernetes 环境
  Workshop agenda and Kubernetes environment
• vLLM、批处理与服务指标
  vLLM, batching, and serving metrics
• 访问笔记本
  Access the notebooks
• 模块一：连接到推理服务
  Module one: connect to inference
• 模型精确率与首次请求
  Model precision and the first request
• 测量流式 Token 延迟
  Measure streaming token latency
• 启动一个代理并检查指标
  Start an agent and inspect metrics
• 比较工作负载的成本和运行时间
  Compare workload costs and runtimes
• 模块二：单位与内存预算
  Module two: units and memory budgets
• 模型文件和配置
  Model files and configuration
• 权重大小、带宽和 KV 容量
  Weight size, bandwidth, and KV capacity
• 模块三：预填充、解码与缓存
  Module three: prefill, decode, and caching
• 测量并检查 KV 缓存的使用情况
  Measure and inspect KV cache use
• 冷前缀缓存与热前缀缓存
  Cold versus warm prefix caching
• 模块四：稠密模型与专家模型
  Module four: dense models and experts
• 推理 Token 与性能限制
  Reasoning tokens and performance limits
• Omer 的优化研讨会
  Omer's optimization workshop
• 模块五：量化选项
  Module five: quantization choices
• 针对实际工作负载检查质量
  Check quality for the actual workload
• 基线负载测试与并发性
  Baseline load tests and concurrency
• 更改 Kubernetes 模型部署方式
  Change the Kubernetes model deployment
• 比较 FP8 性能
  Compare FP8 performance
• 第六模块：推测解码
  Module six: speculative decoding
• 配置草稿模型
  Configure the draft model
• 变慢而非提速
  A slowdown instead of a speedup
• 回到基线并讨论兼容性
  Return to the baseline and discuss compatibility
• 模块七：饱和度和服务标志
  Module seven: saturation and serving flags
• 从笔记本故障中恢复
  Recover from a notebook failure
• 模块八：调优与评估
  Module eight: tune and evaluate
• 研讨会资源和运行时问题
  Workshop resources and runtime questions
• 提高序列上限
  Raise the sequence limit
• 再次比较吞吐量和延迟
  Compare throughput and latency again
• 最后的注意事项与总结
  Final caveats and wrap up

认识杜安·莱特富特和卡贾·奥马尔Meet Du'an Lightfoot and Khaja Omer

我之前提过，我叫 Dewan Lightoot。我是 Akami 的开发者 AC 倡导者，这位是我的同事。I mentioned before my name is Dewan Lightoot. I'm a developer AC advocate at Akami and this is my colleague.

主持人：我是 Kaja Omar，大家叫我 R。我是 Akami 的高级软件工程师。今天我们要聊的是能够自主进行推理的智能体，以及如何在专用 GPU 上构建可用于生产环境的 AI 智能体。那我们为什么会在这里呢？情况正在发生一些变化。对我来说，我在工具调用出现之前就开始构建智能体了。我的第一个智能体是在 2023 年 1 月构建的。当时我会输入一个提示词，里面的上下文大约有 8,000 个 token，然后解析这个提示词，再用 Python 调用工具，最后把响应交还给 LLM，对吧？但在这个过程中，我发现了一件有意思的事：这可能会非常昂贵，对吧？于是我开始使用本地模型，琢磨该怎么做。接着我又发现了另一个问题：如果我使用的是较小的模型，它的能力可能不够强，而且我可能会受限于算力；GPU 的显存可能无法处理更长的上下文。Host: I'm Kaja Omar. I go by R. I'm a senior software engineer at Akami. And we're going to be talking agents that own their own inference, building production AI agents on dedicated GPUs. Now, why are we here? Things are kind of changing. And for me, I started building agents before tool calling. Back in January 2023 is when I built my first agent. I would take a prompt, that context was about 8,000 tokens at the time, and I would parse that prompt and then make a tool call with Python and then give the response back to an LLM, right? But as I was doing that, I found something interesting. It can get really expensive, right? And then I started using local models to figure out, okay, how can I do this? And then I learned something else. There's another challenge around that because if I'm using a smaller model, it may not be as capable and then I may be compute bound that memory in that GPU may not handle the larger context.

这就是输入和输出，以及多轮交互带来的工具成本。那么，怎样才能最大限度地利用 GPU、控制成本并满足生产环境所需的延迟要求，或者满足我们自己的使用场景呢？我们今天来到这里，是要聊聊那些往往需要租用智能的智能体，以及如何重新掌控这一切，让智能归自己所有。So that's the inputs and outputs and then the multiple turns in the tool cost. How do we get the most either the GPUs, the costs, the latency that we need in production or just for our own use cases, right? So the reason why we're here, we're going to talk about agents that are often renting their intelligence and then how you can take back that control and own it.

智能体循环与推理所有权Agent loops and inference ownership

说到智能体，它本质上就是一个循环。你调用一个 LLM，它可以访问工具和环境，然后做出决策——也就是在你调用它或给它提示后，推理该采取哪些行动。接着，你的工具、Python 代码或其他编程语言会执行这些行动，再把结果反馈给模型。模型会评估这些行动及其结果，然后更新计划、采取更多行动，或者达成目标，对吧？但这一切都需要生成 token，对吧？而根据你的应用场景，这可能会带来延迟方面的挑战。因此，我们要讨论如何在专用 GPU 上实际管理这一切。那么，拥有自己的推理能力意味着什么？首先，说到拥有自己的推理能力，你实际上是拥有 GPU，或者租用 GPU。When it comes to an agent, it's just a loop. You invoke an LLM, it has access to tools, it has access to an environment, makes a decision, a reasoning decision on what actions to take once you invoke it or give it a prompt. And then those actions are being performed by your tools, your Python code, your other programming languages and then given back to a model. The model evaluates that the actions, the results and then updates the plan, takes additional actions or the goal is achieved, right? But all that requires tokens to be generated, right? And depending on your application can lead to latency challenges. So we're going to talk about how you can actually man manage all of that on dedicated GPUs. So what does it mean to own your inference? First, when it comes to owning owning your inference, you actually own the GPU or rent the GPU.

所以这么做的话，就意味着你可以使用 GPU，也就是运行 AI、承载模型的硬件，这样你就能控制它们部署在哪里。这样一来，数据就不会离开你的集群、数据中心，甚至不会离开你的笔记本电脑，对吧？现在，这些智能以及模型的运行情况都由你掌控。如果你使用云服务，也可以采用私有 VPC 端点，或者把它部署在 VPC 中，这样它就不会离开你的 VPC。对吧？我知道大家的环境各不相同，基础设施的搭建方式也不一样。我们可以聊聊这些情况。So when you do that, that means that you have access to the GPU, the hardware for that AI to be ran on, for the models to live on, which means then you have access to control where it goes. So the data never leaves your cluster, never leaves your data center, never leaves your laptop, right? You now own that intelligence and what's happening with the model. And if you're in the cloud, you can also do something like private VPC endpoints or have it in the VPC so it never leaves your VPC. Right? I know we all have different environments, have different ways to set up our infrastructure. We can talk about all of those.

何时使用托管 API 或专用 GPUWhen to use hosted APIs or dedicated GPUs

那么，自有和租赁有什么区别？什么时候该买，什么时候该租？嗯，首先，如果你的使用量很高而且稳定，为什么会这样？对。Now what's the difference like when do you own when do you rent? Well, first if your usage is high and steady, why is that? Yes.

嘉宾：是的。演讲结束时，我会提供一个链接，里面有幻灯片以及我们展示的所有代码，所有内容。你可以下载下来带走。那么，谢谢大家。现在，如果使用率很高，什么时候该自己拥有（GPU）呢？对吧？如果你一直有工作负载在运行，因为如果你为 GPU 付费，你其实是在租用 GPU。这意味着什么？这台 GPU 应该始终在工作。否则，你付钱买的是什么呢？所以，这是你需要考虑的事情。接下来是数据：如果你需要确保数据保持私密，需要拥有数据，重视自己的知识产权，或者因为治理和合规方面的原因而不能使用公共 AI 服务提供商，那么这也是你想要自己拥有的另一个理由。然后是硬件，这也是我们需要考虑的，你自己的硬件。Guest: Yes. At the end of the talk, I'll have a link to the slide and everything that we showed code, everything. You can download it and take it with you. So, thank you. Now, when do you own if the usage is high? Right? If you're constantly have a workload running, because if you're paying for a GPU, you're renting a GPU. What does that mean? that GPU should always be working. Otherwise, what are you paying for? So, that's something you got to think about. Then, when it comes to the data, if you need that data to stay private, if you need to own the data, you care about your intellectual property, or you have some governance and compliance reasons why you can't use a public provider when it comes to AI, that's another reason you want to own. And then the hardware, that's something we got to think about your hardware.

如果你有能力承担成本，也有足够的资源，为什么不为自己的用例运行这些模型呢？这样你就能完全掌控一切。那什么时候该租用呢？如果你的流量很低、时有时无，只是偶尔需要智能能力，那么自己拥有一些 GPU 可能并不划算。接下来，我们来谈谈智能能力。能力更强的模型需要更多 GPU，因此成本也更高。而且，这些服务提供商通常会通过 GPU 提供这类智能能力，因为他们已经具备相应的基础设施。所以，使用这些服务提供商来获取智能能力及其带来的好处，可能更有意义。接下来这一部分通常是一个……If you can afford the capacity and you have the capacity, why not run those models, right, for your use cases because then you have that full control. Now, when do you rent? If your traffic is low, bursty, you just need intelligence sporadically, it might not be efficient for you to own some GPUs. Next, well, we start talking about intelligence. The more capable models, they're going to cost for those GPUs. And often times, those providers will provide that intelligence on the GPUs because they have the infrastructure for that. So, it may make more sense to use those providers to get that intelligence, get those benefits. And then the next part which is often a

研讨会日程与 Kubernetes 环境Workshop agenda and Kubernetes environment

挑战在于，你们的团队可能缺乏支持专用或自托管 AI 所需的知识和专业技能。今天的议程将从一些基础知识开始，包括我们将使用的基础设施技术栈、计算单元和内存预算。我们会了解预填充、解码和 KV 缓存，以及它们的具体作用。接下来，我们会比较稠密模型和混合专家模型，让你们了解它们在 GPU 上的实际运行方式，以及该如何选择。之后，我们会讨论如何优化推理：确定要采用自托管方案后，怎样才能充分利用 GPU、所运行的推理引擎以及整个工作负载？最后，我们会把这个智能体部署到自己的专用集群中。challenge is that your teams may not have the knowledge or the expertise to support dedicated or self-hosted AI. Now today's agenda we're going to start out discussing some of the foundation you need to understand like the infrastructure stack we'll be using units and memory budget. We'll look at prefield decode and KB cache and what that looks like. And then from there we'll look at dis dense models verse mixture of experts so you understand what that actually looks like on the GPU and why you would pick one or the other from there will discuss how to optimize your inference right once you figure out okay we want to do self-hosted how can we optimize this to get the most out of our GPUs the inference engine that we run and just our entire workload and then finally from there. We'll take that agent and deploy it into our own dedicated cluster.

话虽如此，我把时间交给你，Omar。好的，谢谢。接下来，我们会介绍一下你们每个人在这次研讨会中将获得的环境。我们有一个 Kubernetes 集群，里面有 200 多个 GPU 节点。你们每个人会分配到一块 GPU。我们已经进行了设置，让每个人都有自己的命名空间；在这个命名空间里，你们会运行自己的 Jupyter notebook，还有一个 VLM 推理端点，供你们进行各种尝试。此外，我们也会在上面部署 agent。这里比较有意思的是，你们命名空间中运行的 VLM 端点，将会在前四个模块结束后作为内容的一部分进行优化。我觉得这会相当有意思。With that being said, I'm going to pass it off to you, Omar. Right. Thank you. So, next uh we'll talk about uh the environment that you each of you is going to get for this workshop. Um we have a Kubernetes cluster with around 200 plus GPU nodes. Each of you gets a single GPU. Um and we have set it up in such a way that uh each of you gets sort of like your own namespace and within that namespace you will have like your Jupyter notebook running and then you will have your VLM inference endpoint running as well that you'll be playing around with. Um and then we're also going to be deploying uh the agent on there as well. Uh the cool thing here is that the VLM endpoint that we have running in your namespace, you're going to be optimizing it uh as part of uh um after the first four modules. And um I think that'll be pretty interesting.

我们将看看为此会用到的不同技术。你会反复重新部署这个 VLM 端点，直到对它进行优化，提高吞吐量、降低延迟，并确保满足所有 SLA。接下来我想谈谈 LKE 上的 GPU。我在 Okami 负责 Kubernetes 相关工作，所以这算是我的专长领域。如果有人有任何 Kubernetes 相关问题，随时都可以提问。不过我想先谈谈 Akami 提供的资源。我们会使用 RTX 4000 ADA GPU。我们也提供其他 GPU，不过这些是专用 GPU。eGPU 有 200……eGPU 配备 2 GB VRAM，内存带宽为每秒 260 GB，因此解码速度会相当快。We're going to look at different techniques that we're going to be using there uh to do so. And you're going to redeploy this VLM endpoint over and over again until you have sort of optimized it to increase the throughput uh reduce the latency and uh make sure like you're meeting all your SLAs's. And then the other thing I want to talk about is uh GPUs on LKE. So I work on the Kubernetes side of things at Okami. Um so this is sort of my wheelhouse. So if anyone has any Kubernetes related questions, uh feel free to ask anytime. But I want to talk about what Akami has. So we have the uh we're going to be using the RTX 4000 ADA GPUs. Uh we also have other GPUs available. Uh but these are dedicated GPUs. EGPU has 200 uh eGPU has 2 GB of uh VRAM with 260 GB per second memory bandwidth. So the decode will be pretty quick because of that.

呃，我们的 TFLOPS 性能很高，CUDA 核心数量也很多。因此，预填充也会非常快。呃，目前这个研讨会中我们会使用单个 GPU，但这并不意味着你之后自己部署时也只能用一个 GPU。我们还提供不同的方案，你可以在一个节点中使用多达四个 GPU。这样，单个节点的显存就能达到 80 GB。你也可以在上面部署规模相当大的模型。嗯，而且所有这些都由 LKE 统一编排，包括设置节点、部署驱动程序，以及为你完成所有配置。这样你就不必手动操作了。呃，基本上只需点击一下：点下按钮，所有内容就会部署好，然后你就可以开始使用了。嗯，更多信息也可以通过这里的链接查看。Uh we have high performance in Tflops. Uh large amount of CUDA core. So the pre-fill will be pretty quick as well because of that. Uh currently for your workshop we will be using a single GPU but that doesn't mean when you want to deploy this at the end on your own you can have different plans that we offer as well where you can have up to four GPUs in a node. So that goes up to 80 gigabytes of uh VRAM on a single node. So you can deploy uh decently sized uh models on there as well. Um and uh everything is sort of orchestrated by LKE. So setting up the node, uh deploying the drivers, uh setting it all up for you. So that way you don't have to do anything manually. Uh everything is sort of like one click. You click a button, everything gets deployed and you're off to races. Um yeah, for more information, we have a link here as well.

嗯，最后你们肯定会拿到所有这些幻灯片。这样你们就能访问这些页面，并且……Um and at the end, you'll definitely get all these slides. So you'll be able to get to the pages and

vLLM、批处理与服务指标vLLM, batching, and serving metrics

那，嗯，我再把它交还给 Dwan。then um I'll give it back to Dwan.

主持人：好。推理引擎方面，我们将使用 vLLM。你们中有些人可能听说过 Llama，它很适合本地运行；而 vLLM 则提供了一些更适合生产环境的优势。其中一项是开箱即用的连续批处理，另一项是分页注意力机制（PagedAttention）。它还支持兼容 OpenAI 的 API。因此，如果你使用的是支持 OpenAI 的任何 SDK，都可以搭配 vLLM 使用。它还允许你在为端点和应用设置推理时，进行一些非常有意思的调整。 刚才我提到了连续批处理。对于不了解它是什么、以及它具体如何运作的人来说，它发生在预填充阶段，对吧？提示词会陆续进入。通常使用静态批处理时，假设你有一辆坐满孩子的校车。Host: All right. So for the inference engine, we're going to be using V LLM. Now, some of you may have heard of Lama, which is great for local, but VLM offers some benefits that are better for production. One of those being being continuous batching out the box, the other being page detention. Now, it also supports the Open AI compatible API. So if you're using whatever SDK that supports Open AI, you can use it with VLM. It also allows you to do some really interesting tweaks when you're setting up inference for your for your endpoints and for your applications. Now, I mentioned continuous batching. For those of you that don't know what it is and how it actually works, this happens at the prefill stage, right? the prompts come in and normally with static batch, let's say you have a school bus full of kids.

嗯，要使用静态批处理来卸载并接入新请求，你必须一次卸载一个请求；等所有请求都卸载完后，再返回接入其余请求，然后开始下一批处理。但使用连续批处理时，你可以同时卸载一个请求并接入另一个请求。在预填充阶段进行 KV 缓存时，这样的效率要高得多；分页注意力也是如此。要理解分页注意力，可以想想 KV 缓存以及实际发生的情况：这里会占用内存空间。提示词进入后，最长的提示词会占用内存空间。分页注意力会为较短的提示词也分配这部分内存，对吧？而现在，分页注意力通过 C 和 VLM 统一管理这一切，确保 VLM 可以随着请求不断到来，持续处理、生成并对请求进行批处理。Well, in order for you to drop get new kids with static batching, you have to drop off every kid one at a time and then once all the kids are dropped off, you go back and get the rest of the kids and then start the next batch. But with continuous batching, you can drop off a kid and pick a kid all at the same time. This is way more efficient when it comes to the KV caching that happens during the preill stage. now or when it comes to page detention. Paged detention is when you think about the KB cache and you think about what's actually happening there's memory space that happens. So the prompt comes in and the longest prompt is going to take up memory space. When you when page attention that memory space gets allocated even for smaller prompts, right? But with page attention now that is all managed through c with the VLM that ensures that you can continuously pro process generate and batch those requests as they come through VLM.

它有一个调度器来管理批处理，这让整个过程非常高效。当然，还有其他服务引擎，我们会在本次研讨会中介绍其中一些。话虽如此，你们需要关注的一些指标包括吞吐量，对吧？要了解 GPU 的运行表现，可以关注首个 token 的生成时间：也就是说，你发送一个提示词后，需要多久才能收到第一个 token？接下来是 KV 缓存利用率。我们会查看这些指标，并展示如何从 VLM 和 QEP 中获取这些数据。在本次研讨会中，你们会看到这些指标，并更好地理解它们。说到这里，我们接下来要……It has auler that kind of manages the batching, but it makes this really efficient. Now, there are other service engines that are out there, and we'll talk about some of those in this workshop. With that being said, some of the metrics you're going to want to care about is going to be throughput, right? To understand how your GPU is performing that time, the first token, understand that, okay, you sent a prompt. How long does it take to get the first token back? from there that KV cache utilization. We're going to look at these metrics and show you how to get all of that out of VLM and then the QEP. So these are some of the metrics as we're going through this workshop you'll you'll see and be able to better understand. And with that being said, we're going to

访问这些笔记本Access the notebooks

访问这个环境。如果你能扫描这个二维码，就拿出笔记本电脑，跟着一起操作吧。如果不行，也可以跟着我们一起完成这场工作坊。在这个 Jupyter 笔记本中，我们准备了九个不同的实验环节，所有资源和依赖项都已加载到这个环境里。如果你想查看代码，回家后自己试试，我已经把代码仓库公开放在 accommoders 上了。我会把这个链接、演示文稿以及我们分享的所有内容都放在同一个二维码里，供你下载。如果你需要我生成一个短链接，我也可以。好，那我们现在开始工作坊的第一个模块。大家都连接好了吗？access the environment. So if you can scan this QR code, pull out your laptops if you want to follow along. If not, you can follow us as we go through the workshop. In this Jupyter notebook, we have nine different labs that we'll cover and all of the resources and dependencies are loaded into this environment. Now, if you want to look at the code and try this on your own at home, I do have the repo public on accommoders. I'll have this link along with the presentation and everything that we shared available with one QR code that you can download. And if you need me to make a short link, I will. All right, with that being said, we're gonna start with the first module in the workshop. If everyone is connected.

嘉宾：对，我这边也有点慢，所以我就用了自己的链接缩短器，因为我和 Russ 很快就把它做出来了。是啊， 乐彩Guest: Yeah, it was kind of slow for me too, which is why I went to my own leak shortener because I kind of made that really fast with Russ. So yeah,

主持人：希望吧，比如说，如果你确实有个热门的……Host: hopefully like if you do have a hot

模块一：连接到推理服务Module one: connect to inference

热点……如果有的话，我会用你的热点。嗯，不过我们会尽量让这次体验发挥出最佳效果。听起来不错吧？太棒了。好。那么在第一个模块模块中，我们要讨论推理栈。我们会连接到 VLM 端点，了解它，看看它如何生成 token。我们还会看看如何构建我们的第一个智能体，了解一些需要知道的事情，看看一些指标，然后了解我们要连接的模型。我们要用的模型是 Qwen 3 4B，不过还有几个模型存放在 PVC 上，你实际上可以为这个端点加载和卸载它们。现在，谈到 vLLM，有一点是，当你提供一个端点时，它只会有一个模型，对吧？所以如果你想提供两个模型，就需要另一个 vLLM 端点，对吧？spot, I would use your hotspot. Um but we'll try to make the best out of the experience. Sound good? Awesome. All right. So in the first module module, we're going to talk about the inference stack. We're going to connect to our VLM endpoint, understand it, see how it generates tokens. We'll also look at building our first agent. some things that we need to know, look at some met some metrics and then understand the model that we'll connect to. The model that we'll use is going to be Quinn 3 4B, but we have several models that sit on a PVC that you can actually load and unload for this endpoint. Now, when we talk about VL LM, one of the things about it is when you serve an endpoint, it just has one model, right? So if you want to serve two models, you're going to need another V LMA endpoint, right?

这意味着你可能需要一些模型路由，在它前面放一个网关来处理这些事情。它确实支持身份验证，但不支持授权。所以你可能需要在它周围加一个网关，对吧？Agent Gateway 通常是……Kerve 是另一个选择。嗯，还有别的吗？Which means you may need some model routing, a gateway in front of it to kind of handle that. Now, it does support authentication, but there's no authorization. So you may need a gateway around that, right? So agent gateway is often kerve is another one. Um, any others?

嘉宾：有 LLMD，还有各种各样的开源工具可以让你实现这些功能。不过我们主要使用的是 KServe。它在扩缩容、路由和身份验证方面都做得很好，对吧？它缺少的另一个功能是速率限制，对吧？如果你要处理请求，可能会希望设置速率限制。所以，你还是需要找一个网关放在它前面，用来管理这些请求并处理路由。我提到过，在运行时方面，还有其他可以用于服务端的服务端点或服务运行时。SGLang 是其中之一，TensorRT 是另一个，还有 llama.cpp。之后我们会更深入地讲解 llama.cpp，让你能更好地理解它。现在，在这个环境中，你连接到的是一个位于该命名空间中的 Jupyter Notebook。Guest: There's LLMD. There's there's all sorts of open source tooling out there that lets you do that. Uh but the main but the main one is like Kserve is what we use a lot. Um and it's it's pretty good and doing the scaling, doing the routing and the authentication as well, right? Another thing it doesn't have is rate limiting, right? If you're processing requests, you may want to rate limit. So again, you want to find a gateway to put in front of it to manage those requests and also handle the routing. I mentioned that when it comes to the runtime there are other service endpoints that or service runtimes that you can actually use for the server. SG lang is one, tensor RT is another and llama CPP which we do go over this later with a little bit more in depth so you can kind of understand it. Now in this environment you are connected to a Jupyter notebook that lives in the name space.

你还会有稍后就能看到的 VLM 服务器端点。这些都在各自独立的 Pod 中。然后你还会有另一个 Pod，也就是你可以为智能体启动的另一个 Pod。我们也会看到它。那么，我们开始安装依赖项吧。我们来安装依赖项。我之前已经大致跑过一遍了，所以有些内容已经加载到我的运行环境中了。我想，我们还不知道网络情况会怎么样，对吧？不过好在你发送请求后，处理过程会在集群内部进行。所以，当我们发送第一个请求，了解 VLM 中的配置时，你会发现模型名称里包含你在集群中的命名空间名称。kubeconfig 只针对你的命名空间。这样一来，环境中的任何人都无法访问其他人的命名空间。You also have the VLM server endpoint that you'll see here in a moment. And all of this is in its own pod. And then you'll have another pod. Well, another pod that you can launch for the agent. And we'll see that too. So, let's go ahead and install our dependencies. So, we'll install our dependencies. I've kind of already ran through this, so some of this is already loaded in my runtime. I figured we don't know how the internet will be, right? But the good thing is once you send a request, it happens within the cluster. So once we send our first request to understand how we're configured in VLM, you can notice the model that we have the name of your name space inside of the cluster. The coupube config is just for your namespace. So that way nobody can access anyone else's namespace in the environment.

然后我们还没有设置 API 密钥，不过你可以用 V L 设置一个。And then we don't have an API key set, but you can set one with V L.

模型精度与首次请求Model precision and the first request

接下来，我们将使用兼容 OpenAI API 的端点，让 API 能够连接到 VLM QN 34B 这款 40 亿参数模型。我记得我们用的是 FP8，也就是说它实际上有，呃，80 亿个参数。是 80 亿个参数吗？Now from here, we're going to use the open AI API compatible endpoint to be able or API to connect to the VLM QN 34B 4 billion parameter model. I believe we're using that at FP8, meaning that it actually sits at um eight billion parameters. Is it eight billion parameters?

主持人：不，是四十亿个参数。Host: No, four billion parameters.

嘉宾：40亿个参数。Guest: Four billion parameters.

主持人：我们使用的是 BF16。Host: We're using BF16.

嘉宾：哦，BF，原来是这么回事。Guest: Oh, BF, that's what it is.

主持人：四精度。Host: Four precision.

嘉宾：是的。所以我们用的是 BF16 全精度。因此，参数的比特数、字节数或数量都要乘以二。也就是 40 亿个参数，这意味着在 GPU 上占用 8 GB。现在我们调用模型后，就收到了模型的响应，也能看到实际是由什么提供服务的。这一切都在我们的集群内完成。速度挺快的，对吧？它就在我们的集群内。所以希望通过这次体验，既然我们没有调用公共 API，而且已经连接到这个 HTTP 服务，整个过程就会直接在集群内部进行。响应应该会快一些。最后，我们要进行追踪、发送一条追踪记录，这样就能真正弄明白……Guest: Yeah. So, we're using BF16 full position. So, you times the bits or bytes or parameters by two. So, it's 4 billion, which means it's eight gigs on the GPU. Now once we invoke we got our response for the model and we can see what it was actually served by. Now this was all within our cluster. That was pretty fast, right? It lives within our cluster. So hopefully with this experience since we're not hitting the public API, we're already connected to this HTTP. It's going to happen right inside of the cluster. The response should come back a little faster. Now we're going to trace send a trace in the end so we can actually understand

测量流式传输的令牌延迟Measure streaming token latency

它实际经过预填充、KV 缓存和解码阶段时是什么样的。这里我们有要进行配置的流。我们会查看各个块并进行一些计算，得出首个 token 的生成时间，以便理解这一过程。稍后我们就能看到这一切是如何生成的。现在运行一下——我刚才又运行了一遍。我们到下面来运行这个。这可能需要一点时间，不过我们应该能看到耗时。首个 token 用了 37 毫秒；至于输出 token 的耗时，也就是解码阶段的耗时，指的是首个 token 生成之后每个 token 所需的时间，对吧？这就是处理这些 token 时的内部 token 间延迟。我们正在计算这个指标，结果大约是每秒 40 个 token，这很可能就是我们的上限。how this actually looks like when it goes through the prefill the KB cache and the decode phase. So here we have our stream that we're going to configure. We're going to look at the chunks and do some calculations. We'll get that time to first token so we can understand that. We'll see how all of this is generated here in a moment. So if we run this, I ran that one again. We'll come down here and run that. This may take a little bit to go through, but we should see time. The first tokens took 37 milliseconds and time output token, the decode phase, which means every token after the after the time of first token, right? That's going to be that inner in interle latency, right? that inter latency that happens when those tokens are processed. So, we're calculating that and we're getting about 40 tokens per second, which is probably going to be our cap.

不过，当我们开始讨论吞吐量时，情况就有些不同了，因为这只是一个请求。但想象一下，如果我们有数百个请求，或者 16 个请求，吞吐量就会高得多，不过它们仍然会被限制在每秒 40 个 token，对吧？或者吞吐量也可能下降，但我们仍然可以获得更高的整体吞吐量。我相信 Omar 会深入讲解这一点。我想展示的一件事是：为什么不直接在 CPU 上运行呢？嗯，我不想在这个环境里用 CPU，因为这个用于 Jupyter notebook 的 pod，或者说容器，只有 1 GB，对吧？所以在上面运行本地模型会直接崩溃。我只是模拟了一下那会是什么样子。不过，如果你试图在 CPU 上运行本地模型，速度会慢很多，对吧？But when we start talking throughput, that kind of changes because this is just one request. But imagine if we had hundreds of requests or 16 requests, that throughput is going to be a lot higher, but they're still going to be capped at that 40 tokens per second, right? Or it could go down, but we can still get more on that throughput. And I believe Omar will go deeper into that. So one of the things that I want to showcase is the question is around why don't you just run this on CPU? Um I didn't want to do CPU on environment because this this um pod for this or this container for the Jupyter notebook is only one gig, right? So trying to run a local model on this is gonna it's just going to crash. So I kind of simulated what that would look like. But if you're trying to run a local model on CPU, it's going to be a lot slower, right?

它没法像 GPU 那样对那些矩阵执行复杂的乘法运算。这正是 GPU 的用途。它能并行完成所有这些数学运算，因此速度快得多，对吧？所以，我们在 CPU 上模拟这个过程。然后，我们实际上会在 GPU 上看看效果如何。我们达到了每秒 45 个 token。有一点让我觉得挺有意思：我经常会使用本地模型，可以说是经常运行它们。每秒 45 个 token，或者 39 个 token，对于某个特定用途的应用来说并不算差，对吧？如果你有特定用途，而且不在意延迟，也不在意这些因素——比如说，如果你有一个聊天机器人，就会希望它尽快生成第一个 token。但对于一个标准应用，如果你只是启动一个后台任务，那么每秒 39 个 token 可能也没那么差，对吧？It can't do the complex multiplication in those matrices like a GPU can. That's the purpose of a GPU. It does all those all that math in parallel, which makes it a lot faster, right? So, we simulate this on on the CPU. And then from there, we actually look look at this on the GPU to see how that looks. We got 45 tokens per second. And one of the things about this and which is interesting to me is that I do a lot of I would say local models and running them. But 45 tokens or 39 tokens per second isn't that bad for an application if you're running it for a specific use case. Right? If you have a specific use case and you don't care about the latency, you don't care about those like if you have a chatbot, you want that kind of that first token to come fast. But for a standard application where you just kick off a job in the background, well, that 39 tokens per second may not be that bad, right?

所以，有时，每秒生成的 token 数较低，或首 token 时间较短，未必那么重要，这取决于具体应用以及你关注的延迟，对吧？你可能更在意任务完成得有多智能，而不是它实际有多快。So sometimes a lower token per second or a lower time to first token may not be as important depending on the application and the latency that you care about, right? you may care more about the intelligence in the job getting done than you do actually how

启动一个智能体并检查指标Start an agent and inspect metrics

它实际上发生得很快，尤其是像我说的那样，如果它是一个后台进程。现在，我们要构建第一个智能体。对于你们当中那些听我们谈到智能体的人来说，这个智能体里没有工具。所以它实际上只是一个请求，只不过是智能体的基础。我们给它一个系统提示词，这样它就知道自己的角色，等到课程后面我们把它交给我们时就能派上用场。但现在，我们只给它一个提示词，并将它的系统角色设定为 Akami 的解决方案架构师智能体。你们可以看到，它成功生成了回复。这个回复很快，也能正常工作。但问题是，如果我们想大规模实现这一点，或者想构建一个不只是接收提示词并生成回复的智能体，它可能需要多次调用工具，对吧？fast it actually happens especially like I said if it's a background process. Now here we're going to build our first agent. Now for those of you that when we talk about agents there are no tools in this agent. So it's actually just a request but just the foundation of the agent. We're giving it a system prompt so it knows it role when we do give it to us later on in the course. But right now we're just giving it a single prompt and we're giving it a role as system as a solutions architect agent for Akami and you can see that it was able to generate the response. Now, this response pretty fast. It works. But then the question is, if we're trying to do this at scale or we're trying to build an agent that does more than just take it take a prompt and generate a response, it may need to make multiple tool calls, right?

所以，这时候你就会开始考虑 KV 缓存之类的问题，也会开始考虑上下文容量有多大。因为我们刚才谈到 KV 缓存时，实际上说的就是你能在那块 GPU 上容纳多少上下文，对吧？连续批处理的工作方式是：你向模型发送一个提示。可是，可能会有 100 个用户同时发送提示，对吧？而这些请求的大小各不相同。因此，如果使用 VLM 之类的模型，即使是较小的 GPU，也能处理一批请求。要是使用 Ollama 之类的工具，你会发现它很快就会限流，因为它处理这些请求的方式不一样。而 vLLM 的调度器在处理这类情况时要好得多。接下来，我们要设置并调用智能体，然后看看其中的各个步骤。So, that's when you start to think about things like the KB cache. You start to think about how much context because when we started talking about that KB cache, that's actually how much context you can actually fit on that GPU, right? And so with continuous batching, what happens is you send a prompt to the model. Well, you may have 100 users trying to send prompts at the same time, right? Well, continuous batch, they're all going to be different sizes. So, a smaller GPU can actually handle a bunch of requests if you're using some with something like VLM. Now, if you're using something like Olama, you'll see you'll get throttled a lot faster because it doesn't process those requests the same way. But VLM actually handles that with theiruler a lot better. And so here we're going to set up our our our agent and invoke it and we're going to actually going to look at the steps.

现在，我们为此使用的代码位于 common 中，你实际上可以查看所有导入的自定义代码。这里有一个 foundation.py，它可以让我们使用 Prometheus 从 VLM 获取一些测量数据，对吧？我们可以查看这些指标。说到指标，你可以使用 Prometheus，也可以使用 Grafana，对吧？我自己总是使用 Grafana。除了查看指标之外，我还总会在智能体中构建的一项功能——也是最先构建的功能之一——就是可观测性，对吧？我常用 Langfuse。你们有多少人听说过 Langfuse？市面上还有其他工具，不过它是开源的，你也可以把它部署在自己的集群中，与智能体并排运行，然后访问它。这样你的可观测性就是私有的，对吧？Now the code that we're using for this is inside of common and you can actually look at all the code that we actually are importing that is custom code. So here we have this foundation.py that allows us to do some measures measurements from with Prometheus from VLM right we can look at those metrics. Now speaking speak speaking of metrics you can use Prometheus and you can use graphana right at home I always use graphana also when I'm building my agents outside of just looking at the metrics another thing that I always build one of the first things I build into my agents is observability right lanefuse that's something I commonly use how many of you have heard of lanefuse now there are other tools out there but it's open source you can run that in your cluster as well right beside your agents and then have access that now your observability is private as Right?

你可以看到代理中的每一个步骤、每一条追踪记录、每一个跨度，以及各自消耗了多少 token、延迟是多少——不只是从生成第一个 token 开始的延迟，还包括每次工具调用的延迟，对吧？如果你现在想排查问题或进行评估，就可以在这些跨度中看到每一步的延迟。但你也会发现，哦，我用的是一个更小的模型，它调用工具的次数多得多，耗时也远超预期。这是为什么？这就要靠评估来找答案了，OM 会对此详细介绍。不过，让你能够查看代理的运行情况很重要，这样你就能了解使用较小模型时该如何优化。优化方法可能很简单，比如在提示词中移除一些工具，或者让代理稍微专精一些。接下来我们运行一下。现在我们来看看这些步骤。And you can see every step, every trace, every span in your agent with how many tokens, what was the latency, not just from the time the first token, but for every tool call, right? If you're trying to troubleshoot or you're trying to evaluate now, you can see all of the latency steps in the spans. But then you can see, okay, I'm using a smaller model and it's making way more tool calls. It's taking way more time than it should. Why is that? Well, that's where evaluations come in and OM will talk more about that. But it's important to have that visibility into your agents so you can understand when you're using a smaller model how to optimize. And that could be something as simple as a prompt removing tools or making the agent just a little bit more specialized. And so we'll run this. And so we're looking at the steps.

我们来看看最大 token 数，以及首个 token 所需的时间。随着 token 数增加，可以看到首个 token 的耗时（毫秒）会如何变化，对吧？随着提示词变长，生成首个 token 所需的时间也会更长，对吧？那么问题是，你要针对什么进行优化？是中位数，还是最大值？对吧？也就是延迟最高的那个。你要针对耗时最长的情况进行优化。We're going to look at the max tokens and we should see the time to first token. So you can see as we increase the tokens, what happens to the time to first token in millisecond as we increase the pump, right? As the prompts get larger, that time the first token takes longer, right? So the question is what do you optimize on? Do you optimize on the median or the largest? Right? The one that took the most time doing the latency. Well, you're going to optimize on the one that takes the most time

比较工作负载成本和运行时间Compare workload costs and runtimes

而不是中位数，对吧？这样你就能通过这种方式提高中位数。现在，[哼笑]，我们来谈谈租用还是拥有。我们要看看 token。生成了多少 token？我之前提到过，如果你的使用模式是突发式的，也就是说你不太常通过服务提供商调用模型；而如果你有一个智能体，[哼笑]，或者一些始终在运行的智能体应用，就可以比较一下两者的成本。所以，我们先生成一条响应，只是为了获取这些 token。接下来，我们实际看看账单，并比较哪种方式更高效。假设你每天处理 200,000 个 token，通过托管服务按量租用的话，每月大约要 1,500，对吧？但如果你租一台 GPU，每月大约要 1,000。rather than the median, right? And then that way you can increase your median by doing that. Now, [snorts] here we talk about renting versus owning. We're going to look at tokens. How much are being generated? This is where I I mentioned that if you're using bursty where you're not using invoking a model as much with a provider versus if you have a agent [snorts] or some agentic applications that are always working you can kind of compare that cost. So here we generate a response just to get those tokens. And now we're going to actually look at the billing and compare which one would actually be more efficient. So, if you're processing, let's say, 200,000 tokens per day on a hosted bill, rent it, that's going to be about 1,500 per month, right? But if you're renting a GPU, that's going to be about a,000 per month.

现在，GPU 的规模、模型的大小，以及它是否能按你的需求运行，这些都很重要。不过，在比较租用还是自有时，这些都是需要考虑的因素。你可以实际计算一下，分析其中的利弊。我们之前也谈过本地运行时的数据韧性、合规性、安全性、控制权等诸多方面。 最后说说推理运行时的生态。我提到我们在使用 vLLM，但你也可以使用 SGLang；如果你有共享较长前缀的请求，或者需要处理多轮对话，它就很适合。因此，很多组织会在智能体场景中使用 SGLang。至于 TensorRT-LLM，它专注于在英伟达硬件上提供峰值吞吐量；而 llama.cpp 则适合边缘设备，处理速度也非常快。Now, it does matter the size of the GPU, the size of the model, the if it actually performs the way you need. But these are some of the things you think about when you're trying to compare, should I rent versus own? You can actually do the math on that to break that down. And we already talked about these with the data resiliency, you know, the compliance, the security, the controls, and all of those good things when it comes to running locally. Now, before we go, the runtime landscape, I mentioned that we're using VLL, but you can also use SE lane, which is good if you're using if you have requests that share long prefixes, multi-turn chats. So for agents a lot of organizations are using SG lane for that. When it come to tensor well this is about that peak throughput on Nvidia hardware. It really specializes in providing you that and then llama CPP for that edge that that really fast processing.

它可以帮助你处理那些高度量化的模型，比如运行在 ARM 服务器上的模型。所以我们会在这里再调用一次，并查看客户端，确认一切仍然正常。我们正在验证是否可以连接。可以看到模型服务，也可以看到 ID；我相信还可以把整个内容打印出来，这样我们就能看到请求返回时的实际样子。所以我也要把整个内容打印出来。现在你可以看到了——其实我们把它格式化打印一下吧，这样看得更清楚。好了。请求返回时，你可以看到 ID，也可以看到最大模型长度，也就是上下文长度。这个是可以控制的，对吧？当然，在使用模型时，你可以调整这些参数；不过我们可以看到模型本身的所有信息。It can help you with those heavily quantized models like on the ARM server. So we're going to invoke one more time here and we're looking at our client just to verify that we're still good. We're verifying that we can connect. We can see the model serve and we can see the ID and I believe I can print the entire so we can see how that require that request actually looks when it comes back. So I'm going to print this whole thing as well. So we'll and now you can see and actually let's pretty print it. So we can see it even better. There we go. And so when that request comes back, you can see the ID, you can see the max model length, which the that context length. You can control that, right? You can tweak those parameters of course when you're working with a model but we can see everything about the model itself.

现在我们进入下一节。嗯，这一节会有一些……Now we're going to move on to the next session. Um this will have some taking

模块二：单位与内存预算Module two: units and memory budgets

如果你还能进一步探索集群、探索 VM，也可以在这个笔记本中随意操作。所以我先把它关掉。我们丢弃这些内容，然后继续讲单位和内存。现在进入下一部分，我会讲快一点，这样 Omar 就能有些时间。关于单位和内存，我们要讨论使用模型时每秒 token 数的上限。我们还会查看模型的实际配置，了解模型在 GPU 上实际是如何存放的，以及看起来是什么样的。这里，正如 Omar 提到的，我们正在使用……我把它缩小一点。看看能不能多显示一些内容。好。那么说到模型实际存放在 GPU 上的情况，4 billion 参数模型的模型权重，如果采用 BF16 格式，占用 8 GB。it further when you can explore the cluster explore vm you can feel free to do that in this notebook. So I'm going to close that out. We'll discard that and we'll move on to units and memory. Now the next section and I'm g kind of go faster so Omar can has some time. Now when it comes to units and memory, we're going to talk about the ceiling when it comes to token per second on using a model. We're also going to look at the model's real config and understanding how a model actually is sits and how actually looks um when it comes to a GPU. So here, like Omar mentioned, we are using and I'm going to make that a little smaller. Let's see if we can get more of that on there. All right. So when we talk about the the model actually sitting on the GPU, you have the model weights for the 4 billion parameter model, that model weights at BF16 is going to be 8 gigs.

所以我们有 8 GB，房间剩下的空间都要留给 KV 缓存。也就是说，你能在这块 GPU 上容纳多长的上下文？因为所有内容都必须放在里面。现在，当提示词传入时，说到预填充阶段，模型会读取整个提示词，并将其存储到 KV 缓存中。之后，每生成一个新 token，它都会更新这份预填充内容。所以，这份预填充内容会不断增长，对吧？也就是说，即使传入的提示词大小固定，在它经过模型处理并生成内容时，所占空间也会增加，对吧？再想想工具调用、系统提示词，所有这些都会传给模型，而且都必须放进 KV 缓存，这样模型才能高效地处理你的请求。因此，我们还在考虑使用 FP8 的情况。接下来会讲到量化，以及它究竟如何缩小模型，同时仍然实现你所追求的效率。So we have 8 gigs and the rest of the room is going to be for KV cache. Meaning how much context can you fit on that GPU? because it all has to fit within there. Now, when your prompt comes in, when we talk about prefill, the model is going to read that entire prompt and store it in the KV cache. And then every time it generates a new token, it's going to update that prefill. So, that prefill is going to grow, right? So, just because your your your prompt that goes in is one size, as it goes through and is processed through the model, it's going to increase as it generates, right? And then imagine the tool calls, the system prompt, all of these go to the model and all of that has to fit on the KV cache in order for it to be able to handle your request efficiently. And so we're also looking at um if we're using at FP8, which is going to talk about quantization and what that actually looks like to shrink a model and to still get the efficiencies that you're looking for.

现在我们要安装依赖，然后从这里连接并确认已加载的内容。再说一次，我们有 Quinn 3 四十亿参数模型、workshop 命名空间和集群。Now we're going to install our dependencies and then from here we're going to connect to verify what we have loaded. So again we have the Quinn 3 4 billion parameter model our workshop namespace the cluster

模型文件和配置Model files and configuration

API 密钥。说到模型，我们有 config.json 这个 JSON 文件。这个文件到底是什么呢？下载模型时，你会得到 config.json。比如说，你去 Hugging Face 下载一个模型，就会得到一组文件，这些文件都是必需的，对吧？你会得到 config.json 这个 JSON 文件，还会得到这些 save.tensors 张量文件。这些张量文件就是模型权重，对吧？而配置文件则会告诉 VLM 如何处理该模型，对吧？所以，你实际上可以查看并探索这些文件，了解它们具体是什么样的。这里展示的就是配置文件的样子。我们有 128 层，大小为 1,024。注意力头有 16 个。数值是 8。词汇表大小，对吧？模型是如何训练的？API key. Now when it comes to models we have the config.json JSON like what is actually that file when you download a model you get the config.json let's say you go to huggerface you download a model you're going to get a collection of files all of those files are needed right you're going to get the config.json JSON you're going to get these save.tensors tensors. Those tensor files are the model weights, right? But the configuration is what tells VLM how to handle that model, right? So you can actually look at this and explore those files to understand what they actually look like. So here we're showing what that configuration file looks like. So we have 128 layers. The size is 1,024. The attention head, there are 16 attention heads. The values are eight. Vocabulary size, right? How does the model train?

那个模型的词汇表大小是多少？对吧？你确实可以查看这些配置。能修改它们吗？不能，对吧？这些文件无法修改。但你可以微调，对吧？你可以使用更低的精度。你可以做这些事情来适配模型。但如果你试图修改这些文件，模型就会崩溃，无法加载。现在我们来看看每个权重所占的字节数，以及它们是如何计算出来的。这里我们看一个较小的模型：一个采用 BF-16 的 60 亿参数模型，参数量乘以 2，得到 1.5 GB；而 80 亿参数模型的参数量则是 8 GB。再来比较 BF16 和 FP8。谈到精度时，大多数模型训练时有时会使用 FP32，但通常会使用 FP16。What is the vocabulary size of that ma model? Right? You can actually look at these configurations. Can you modify them? No. Right? These files cannot be modified. But you can fine-tune. Right? You can use lower. You can do all these things to adapt the model. But to modify these files, if you try to do that, it's just going to crash. It's not going to load. Now here we're going to look at the bytes per weight and where it comes from and we calculate here we're looking at a smaller model at BF-16 for the 6 billion parameter model and the parameters you multiply by two 1.5 gigs and for the 8 billion parameter models what those params look like and it's gonna be eight gigs and if we compare the BF16 versus FP8 Now, when we talk about pre precision, most models when they're trained, they're going to be at sometimes FP32, but you'll often get it at FP16.

全精度下，也就是说，模型大小要乘以多少？乘以 4，对吧？所以这个 40 亿参数的模型在全精度下会占 16 GB，对吧？在 GPU 上把它量化到比如说 FP8 时，模型的能力会有一点损失，但不多，对吧？不过这时候你就要开始评估了：如果你的计算资源受限，手头有一个更大的模型，比如 700 亿参数，也许可以用 FP8，或者用 int4，对吧？一路量化到现在这个 700 亿参数的模型占用的空间不再是 32 GB，而是 16 GB，对吧？如果可行的话，对吧？我不知道在位置方面会损失多少，Full precision, meaning that you multiply whatever the model size by what? Four by four, right? So, this 4 billion parameter will be 16 gigs at full precision, right? on your GPU when you quantize it down to let's say FP8 well you kind of lose a little bit capabilities of the model but not much right but that's when you start to evaluate right if your compute constraint you have this larger model maybe 70 billion parameters that maybe you can do it at FP8 or maybe with int4 right all the way down to where now that 70 billion parameter models is rather than 32 is 16 gigs, right? If that's possible, right? I don't know how much you're going to lose for the position,

权重大小、带宽和 KV 容量Weight size, bandwidth, and KV capacity

但这需要你来评估。因此，评估难度很重要。我们来看看权重所在的 GPU。这里，BF16 要乘以 2，FP8 要乘以 1。我们正在为 Quinn 研究这一点。然后，在本课程中我们还会使用另一个模型，比如说，它有 80 亿个参数，在 GPU 上占用 5 GB。你可以看到，如果决定进行量化，在保留足以满足我们所讨论用例的智能水平的同时，能节省多少空间。我们来看看这张卡的每秒 GB 数。你取……我们实际在计算的那个值。好，我们知道 GPU 的速度是 360，也展示了如何计算每秒生成的 token 数，以及实际受什么因素限制。所以，我们会受限于 GPU 的实际速度，也就是它能运行多快。but you'll have to evaluate that. And so evaluation hardnesses are important. So we'll look at the GPU where the weights live. So here BF16, you multiply by two, FP8, you multiply by one. We're looking at this for Quinn. And then we have another model that we're going to use in this course where it's say at 8 billion parameters is five gigs on the GPU which you can see how much you save with the still intelligence that works for the use cases that we're discussing if you decide to quantize down. So we're looking at this card the gigabytes per second. you take the where we're actually calculating. Okay, we know how the speed of the GPU is 360 and we're showing how to calculate to get the tokens per second and where we're actually bound. So, we're going to be bound by the speed of the actual GPU on how fast it can go.

这里我们展示了 KV 缓存的计算过程。我会尽量讲快一点，这样我们就能进入量化和优化部分。不过，如果你把这些内容过一遍，就能读懂并理解 KV 缓存公式的工作原理。很多时候，我们可能不必深入推导，但当你要计算一些实际问题时，就得弄清楚了。比如，你想为 100 位用户提供服务，但手头只有一块较小的 GPU。问题是：我有这块较小的 GPU。举个很具体的例子：我曾经要举办一个工作坊，需要想办法用大约三四块显存约 20 GB 的 GPU 为 80 名学生提供服务，并为所有学生执行推理。我需要这么多 GPU 吗？能装得下吗？那么，这项工作负载的上下文有多长？如果大家同时发起请求呢？And we're showing the math for the KV cache here. And I'm trying to go a little faster so we can get into the quantization and optimization. But if you go through these, you can read through it and understand how the KV cache formula works. Oftent times we may not have to go through this and figure it out, but you will have to when you're trying to calculate, for example, you have a 100 users you want to serve. You have a smaller GPU. The question is I got this smaller GPU. Let's say I'll give you a perfect example. I had to do a workshop and I had to figure out how to serve 80 students and perform inference for all the students with about three or four GPUs that were about 20 gigs, right? Do I need that many? Is it going to fit? Well, how big is my context for the workload? What if everybody is invoking at the same time?

GPU 集群能支持吗？连续批处理能正常工作吗？所以，你要设置一次评估，对整个工作负载进行评估，确定它是否能装得下，还是会达到上限，对吧？会不会开始溢出到 CPU？这会导致应用变慢，或者涉及模型权重的内存预算与 KV 池之间的权衡。我们会看看这些情况，然后说明如何实际计算。比如，在这块 GPU 上，我们可以容纳 37,000 个 token；在这块 Sega GPU 上，模型权重占用 8 GB。这样一来，你可用的空间就能容纳 37,000 个 token，对吧？如果有多个用户发送请求呢？这会显示他们各自的上下文长度。如果有四个用户，他们都可以发送 8,000 个 token，对吧？对于输入、输出和工具调用，这样在这块 GPU 上同时就会累计到 8,000 个 token。Will the GPUs will the cluster support that? Will continuous batching work? So, you set up an evaluation to evaluate the entire workload and to determine, okay, is it going to fit or do you hit that roof line, right? Do you have start to have spill over the CPU? These is your application slowdown or the memory weights memory budget versus KV pool. So we'll look at that and then here's how you can actually get that the math. So here's our we can get 37,000 tokens on this GPU right on the Sega GPU this 8 gigs for the model weights. And then the room that you have is going to be 37,000 tokens, right? And so if we have multiple users sending requests, right? Well, this will show how many users in their context list. So if we have four users, they can all send 8,000 tokens, right? For input output tool calls, this going to add up to 8,000 tokens at the same time on that GPU.

现在我们展示了随着不同请求数量增加以及请求生成量增加，所带来的开销。这可以说明，在试图了解计算资源限制、GPU 以及所执行的工作负载时，应该如何进行计算。因此，你可以使用所有这些代码并加以应用，从而能够……And now we're showing the tax as this increases the different requests and the generation of that request. And this kind of shows how to do that math when you're trying to understand your compute constraint, your GPU, and then the workloads that you're performing. So you can take all this code and apply it and be able to

模块三：预填充、解码与缓存Module three: prefill, decode, and caching

根据你的使用场景进行计算，看看是否能在你的 GPU 上运行。接下来我们会讲到预填充和解码。下面再深入了解一下 KV 缓存中的预填充和解码。我们仍然使用 Quinn3 模型，但会实际深入看看具体情况。 如前所述，预填充阶段主要关注首 token 延迟，对吧？也就是从你按下发送开始，到收到第一个 token 需要多长时间——前提是你使用流式输出。如果不使用流式输出，你就会一次性收到完整响应。但如果使用流式输出，收到第一个 token 需要多长时间，对吧？这就是预填充阶段。接下来是解码阶段，它会衡量后续每一个 token 的生成情况。如果你查看 VLM 中的指标，并将其发送到 Grafana，就可以在仪表板上查看；你可以观察到，随着提示词变长，指标如何上升或下降，对吧？take your use cases and do the math on it so you can see if it fits on your GPUs. So we will go to the prefill and decode. So going a little bit deeper into the prefill and the decode in the KV cache. Again, we're using the Quinn3 model, but we're actually going to dive in to see how this looks. So we have the prefill, which like I mentioned is going to be focused on that time to first token, right? from the time you press send, how long does the first token, if you're streaming, right? Because if you're not streaming, you're just going to get the whole response back. But if you're streaming, how long does it take to get that first token back, right? And that's going to be that prefill stage. And then the decode is going to be that token, every token after that, that's all measured. And if you're looking at the metrics inside of VLM, you send that to Graphana, you can actually view that on the dashboard and you can watch, okay, as I increase my prompts, you can watch it increase, you can watch it decrease, right?

对。真正理解向 GPU 发送请求时，GPU 上实际发生了什么。接下来我们会逐步讲解。先安装依赖，然后查看环境，确保已经连接。没错，我们仍在使用 Qwen 34B。接着我们会看看自注意力机制，也就是 KV 缓存是如何处理的。这就是实际使用的公式。你可以看看这个 softmax，从而真正理解模型是如何处理 KV 缓存的。我们手动演示一下，这样你就能看到其中的数学计算。说到 KV 缓存，有一种朴素的方法，也有一种 KV 缓存方法。我认为后者是这样处理的：朴素方法是，先输入一个提示词，模型查看整个提示词，然后逐个将提示词存入 KV 缓存，并生成回答。Right. And really understand what's actually happening on that GPU when you're sending requests. So, we'll go through here. We'll install our dependencies. Then from there, we'll look at our environment. Make sure we're connected. Yes. And we're still using the Quinn 34B. And we're going to look at the self attention, meaning how does that KB cache get processed. So this is the actual formula. So you can look at that soft max to really understand how that KB cache is being handled within a model. We're just doing it by hand just so you can see the math on that. And so when it comes to the KB cache, there's the naive approach and there's the KB cache approach that I would say is handled. So the naive approach is you take a prompt, the model looks at the whole prompt and then one by one stores that prompt in the KV cache and generates a response.

KV 缓存的作用是这样的：它会说，好，我要获取提示词，从中学习，并把它直接并行存储起来。因此，KB 缓存让你能够从提示词中学习，以便引用其中的信息，而不是在处理 KB 缓存时，只采用朴素缓存方法那种固定序列。我们生成这个之后，[嗤笑]，就可以看看它是如何计算出使用 KB 缓存方法相较于朴素方法能减少多少开销的；我们会在代码中手动展示这一切，这样你就能真正把它拆解开来，并且……What KV cache does, it says, okay, I'm gonna take the prompt, I'm gonna learn from it, and I'm gonna store it right in parallel. And so the KB cache allows you to learn from the prompt to be able to reference the information rather than just doing the fixed sequence of the naive caching approach when it comes to handling that KB cache. And so if we generate this, [snorts] we can look how it computes the reduction of how much if you use the KB cache approach rather than the naive approach and we're just showing all that by hand in code so you can really break it down and

测量并检查 KV 缓存的使用情况Measure and inspect KV cache use

明白。现在我们来看看如何使用 KV 缓存，并测量首个 token 的生成时间。这里是 57 毫秒，随后解码耗时 25 毫秒。接下来我们继续，看看 VLM 实际上暴露了哪些内容。我们可以查看这些指标，这里展示的只是其中一些实际暴露出来的指标，以及如何在 VL 中查看它们。现在我们要调用提示词并逐步增加它，你可以看到 KV 缓存逐渐填满；这需要一些时间。我们会等几分钟，让这个过程继续进行。这样我们就能查看 KV 缓存的使用情况和相关指标。我们可以看到运行期间的峰值使用量，以及完成后的 KV 缓存使用量。这就是处理这些数据时所用的计算方法。understand. And now we're going to look okay using K KV caching and we'll measure the time to first token. And so here we have 57 milliseconds and then the decode was 25 milliseconds. And so we'll continue on and we can see what VLM is actually exposing. So we can actually look at those metrics and we're just showing some of the metrics that are actually being exposed and how you would view that in VL. Now we're going to invoke the prompt and we're going to increase it and you can watch that KB cache fill and you can see this is going to take a little while. So we'll give it a couple minutes while this process we're increasing it. So we can see the KB cache usage and we can look at the metrics. So we have peak usage running and then the KB usage after it's finished. And this will be the math on how you would process that.

再次说明，我们这段代码使用的 worker 位于 foundation.py 中。所以它将作为……的辅助工具。Again the worker that we're using for this code is in the foundation.py. So that's going to be the helper for

冷前缀缓存与热前缀缓存Cold versus warm prefix caching

这段代码。现在我们来看看代码预热与前缀缓存预热的对比。前缀缓存是个很有意思的东西。别把它和提示缓存搞混了，这一点我之前应该提到过。如果你使用某个服务提供商，而且按 token 付费，那么提示缓存尤其重要——如果你有一个持续运行、每五分钟都会执行的应用程序，提示缓存就必不可少，对吧？根据你使用的服务提供商，你最多可以将成本降低约 75%。所以，有一些方法可以让你使用服务提供商的同时节省成本。但我们这里说的前缀缓存，是指为 KB 缓存前缀。这样做之后，我们可以看到它在这里是如何处理的。代码耗时为 220 毫秒，降到了 41 毫秒。this code. Now we're going to look at the warm the code um against the warm prefix caching. Now preface caching is an interesting thing. Um don't confuse it with prompt caching for and that's something I should have mentioned before. If you're using a provider and you're paying by the token, prompt caching, especially if you're do you have a continuous application that's running within every five minutes, prompt caching is a must, right? You can you can decrease the cost by like up to 75% depending on who you're using. So there is some ways you can use a provider and save on cost. But when we're talking about prefix caching, this is caching that prefix for the KB cache. And so when we do this, we can see how this actually is handled here. So the code is going to be 220 milliseconds and it's down to 41 milliseconds.

而这些内容都在向你展示如何在代码中实现这一点。你可以看一下。我想稍微讲快一点，这样奥马尔就能讲到他那部分，希望我们能完成部署。And this all shows you how to do that in the code. So you can go through that. And I'm trying to go a little faster so Omar can get to his part and hopefully we can deploy.

第四模块：密集模型与专家模型Module four: dense models and experts

好。我们回到第四部分，讨论 GPU 上的稠密模型与混合专家模型。说到稠密模型，什么是稠密模型呢？稠密模型加载到 GPU 上时，整个模型都会用于生成 token，对吧？比如说，如果它有 300 亿个参数，或者我们就以现在使用的模型为例。Qwen 3 4B 使用 BF16 时，实际会占用 GPU 上 8 GB 显存。它真正开始生成 token、运行起来时，整个模型都会参与工作，对吧？所以，与使用混合专家模型相比，它会稍慢一些。混合专家模型可能更大，比如占用 30 GB；但它在模型内部实际生成内容时，会选取一组专家，只有这一部分专家会参与生成。All right. So we'll go back and we'll go to the fourth section where we talk about dense model verse the mixture of experts on the GPU. Now when it comes to a dense model, what is a dense model? Well, a disc model when it sits on a GPU, that whole model is used to generate tokens, right? So, if it's 30 billion parameters or let's let's just do the model we're using now. So, the Quinn 3 4B at BF16 8 gigs on the actual GPU. Well, when that actually generates tokens, when it actually is working, that whole entire model is working, right? So it's going to be a little slower than if you use a mixture of experts. Mixture of experts may be a larger model. Maybe it's 30 gigs, but when it actually generates inside of the model, it takes a collection of experts and only that that section of experts does the generation.

所以即使模型有 30 GB，生成时可能只需要 1.5 GB。因此速度会快得多，这就是其中一个好处。虽然混合专家模型更大，但实际上它在应用中的运行效果可能更好，或者运行速度快得多。 这里我们来谈谈内存，以及为什么解码受内存限制。也就是说，在首次生成 Token 后，进入实际生成阶段时，性能会受到内存限制。我们观察到这个 GPU 的速度受限于此，速度不可能再快了，对吧？我们无法通过调整来让它变得更快。不过，有些方法可以帮助我们在规模扩大的情况下提升性能。接下来我们会查看并绘制屋顶线模型，这样就能直观地看到这一点。So even if it's 30 gigs, it may be 1.5 gigs just to do the generation. So it's going to be a lot faster. So that's one of the benefits. Even though it's a larger model when it's a mixture of experts, it's actually a can be performed a lot better within an application or a lot faster within an application. And so here we kind of talk about memory and why decode is memory bound meaning that when we talk about the generation phase after that time to first token when that generation actually happens that's actually going to be bound by the memory. So when we looked at the speed of this GPU that's bound, you can't go any faster than that, right? There's nothing we can do to tweak that to make it go any faster. But there are some things we can do to help this at scale. So we're going to look at our roof line and plot that so we can actually see it.

所以我们要确保模型已经加载。我们有一个 Qwen 3 4B 参数模型，对吧？它会受计算能力限制，因此会碰到屋顶线。我们要看 FLOPS，也要看这个。脊线是每字节 297 FLOPS，对吧？而这里受内存限制。现在我们来看混合专家模型，我们把它当作代码来看，这样就能真正理解它。这个模型有 300 亿个参数，对吧？有 128 个专家，每个 token 只有 8 个处于激活状态。它有 48 层，隐藏维度是 248。现在，混合专家模型中的专家部分就是这些权重，而这些是共享权重。这表示生成时会用到多少权重。相比使用较小的模型，这样会高效得多。实际生成时，它甚至可能更快。So we'll make sure that we have our model loaded. So we had a quint 3 for billion parameter, right? and it's going to be compute bound. So it's going to hit that roof line. So we're going to look at the flops. We're going to look at this. So the ridge line is 297 flops per bite, right? And where this is bound by the memory. Now if we look at the mixture of experts, now we're just going to look at this as code. So we can actually understand it. So at this 30 billion parameter models, right? 128 experts and only eight active per token. That's 48 layers, 248 hidden. And so now the expert mixer expert, this is the weights. This is the share weight. So this is how much it how much is being used when it generates. That's a lot that would be a lot more efficient than using a smaller model. It can actually be faster when it actually does the generation.

所以这里我们看的是密集模型，对吧？它的字节数比密集模型少九倍，对吧？所以解码速度会快一点。这大致就是对比这两者的情况：当你实际在考虑“好吧，我要不要用类似 Mistro 的模型？”或者“我要不要用 Quinn 模型？”时，我们会比较密集模型的尺寸。这里我们要看速率、带宽和 GPU 服务能力。测得 Quinn 4B 的速度大约是每秒 40 个 token，对吧？Quinn 的较小模型 0.6——这不是在 CPU 上完成的，我们只是展示每秒 300 个 token，对吧？不过我们看的是上限。这里实际上是在看带宽，并在这里计算，以得出……And so here we're looking at the the dense model, right? is nine times fewer bytes than a dense model, right? So it decodes a little bit faster. So that's kind of breaking down the two when you're actually trying to figure out, okay, do I want to use something like Mistro, right? Or do I want to use like a Quinn model, we compare the dent sizes. So here we're going to look at the rate, we'll look at the bandwidth, the GPU serve. So measured Quinn 4B is about 40 40 tokens per second, right? The smaller model of the Quinn, the 0.6. Now, this isn't done on the CPU, but we're just showcasing 300 tokens per second, right? But we're looking at the ceiling. This is actually looking at the bandwidth and it's doing the math here to get that the

推理令牌与性能限制Reasoning tokens and performance limits

我们能达到的最大值。我们这块 GPU 的上限是每秒 45 个 token。现在，如果我们在这个模型上启用推理，就可以看看效果。说到这些 Quinn 模型，如果你想构建某种智能体，而模型需要进行推理——也就是说，可能会收到提示词、拥有一些工具，并且需要多次调用工具——那么你就要启用思考模式。这样，当它调用工具并获得结果后，如果还需要再次调用工具，或者需要验证结果才能准确地调用工具，就必须在这些模型上启用思考模式。这只是你可以在模型中调整的一个参数，对吧？不过，关闭思考模式时，它只是生成了一个回答，总共 35 个 token。而现在开启思考模式后，它就得先思考一番。max that we can get. The ceiling that we have on this GPU is going to be 45 tokens per second. Now, if we enable reasoning on that model and we can look at that. Now, when it comes to these Quinn models, if you're wanting to do any type of agent and the model needs to reason, meaning that maybe the prompt comes in, you have tools and it needs to make multiple tool calls. Well, you're going to enable thinking. So now when it makes a tool call, if it gets the results and it needs to make another tool call or verify the results to make the accurate tool call, you're going to have to enable thinking on these models. And that's just a parameter that you can adjust in the model, right? But with thinking off, it just generated a response. It was 35 tokens. But with thinking on now, it had to think about it.

花费了大约 600 个 token。目前，有些模型，比如服务商提供的模型，实际上可以为整个思考过程设置 token 预算。所以，使用服务商提供的模型时，你可以通过这种方式节省成本；或许有些开源模型也能这样做。我不太确定。It cost about 600 tokens. Right now with some of the models like provider models you can actually set a token budget for life of thinking. So that's one of the things you can save on cost with a provider model and maybe some of the open models you can do that too. I'm not sure.

主持人：是的，你可以。Host: Yeah, you can.

嘉宾：是的，你也可以调整思考过程的 token 预算。思考时使用更多 token 并不总是更好，但有时能帮助模型更好地排查问题，并更好地得出你想要的结果。谈到推理时，有很多方面需要考虑，其中一点是显卡本身的显存限制，以及如何将这一限制应用到智能体上。我们会查看所有模块，讨论 KB 缓存、前缀缓存、注意力头，以及量化——我们马上就会讲到这一点。然后是我们之前也讨论过的连续批处理。如果把预填充分离出来，与解码分开，那么就有预填充 GPU 池，而解码受显存限制。所以预填充受计算能力限制，对吧？Guest: Yeah, you can also adjust that token budget on the thinking as well. So more tokens on thinking isn't always good but sometimes it can help the model better troubleshoot and better get to that end result that you're looking for. And so when we talk about inference, there's a lot of things to think about, but one of the things we want to think about is that memory bound limit for the actual card and then how to apply that for the agent. So we look at all of the modules, we talk about that KB cache, the prefix caching, the attention heads as well as quantization which um is going to cover here in a minute and then from there that continuous batching we also discussed that. So if we split the prefill from the decode, you have the prefill GPU pool and the decode which is memory bound. So the prefill compute bound, right?

所以它需要从你的内存传到 GPU，而且需要缓存起来，对吧？在那个阶段（清了清嗓子），你可以做很多优化；但到了解码阶段，实际上你没什么可做的，因为你会受到 GPU 的限制。所以我们只是用代码展示了这一点，而没有实际演示，因为毕竟（哼了一声）这次工作坊的条件有限。话虽如此，我现在要确认一下我们这边都准备好了。So it needs to go from your memory to the GPU and it needs to be cached, right? There's [clears throat] that that phase you can you can do a lot of optimizations, but when it comes to the decode phase, there's nothing you can really do because you're bounded by the GPU. And so we kind of show this in code rather than actually showing it because again we [snorts] only have so much limitation in this workshop. Now with that being said I am going to make sure that we are good to go here.

Omer 的优化研讨会Omer's optimization workshop

嗯，我觉得我不用切换模组。你应该没问题。护甲。接下来交给你了。Um I don't think I need to switch the mod. I think you should be good. Armor. I'm gonna turn it over to you.

主持人：好的。Host: All right.

嘉宾：然后，嗯，我们应该可以开始了。你可以谈谈量化。谢谢。Guest: And then Yeah, we should be good to go. You can talk about quantization. Thank you.

主持人：好的。谢谢你，Dwan。Host: All right. Thank you, Dwan.

嘉宾：是的。Guest: Yep.

主持人：好的。那么，我把它关掉。好。Host: All right. So, let me close that. Sure.

嘉宾：那、那其实是你的。给你。Guest: That That's yours, actually. There you go.

主持人：好的，很棒。那么我想谈谈一些完全不同的优化技术，我认为你可以用它们在 Kubernetes 上部署这些模型。嗯，正如我刚才所说，我的背景是 Kubernetes。我在 Akami 做了很多 Kubernetes 相关的开发工作，也参与了那里的托管 Kubernetes 平台。所以，我们做的很多工作都是为了确保能在 Kubernetes 上很好地使用 GPU，并充分利用它们提供的各种功能。我们也会运用一些我接下来要介绍的技术来部署这些模型。嗯，有几种不同的技术。我们会谈谈量化。我认为这是最重要的一项，特别关键。我们还会谈谈推测解码，这是一种非常酷的技术。Host: Okay. Cool. So, I'll be talking about a whole different um uh optimization techniques I think you can use to deploy these models on Kubernetes. Um, like I said, uh, my background is in Kubernetes. I develop a lot of Kubernetes stuff at Akami and I work on the manage Kubernetes platform there. So, a lot of the stuff that we do is we want to make sure that we're able to use GPUs on Kubernetes really well. Uh, use all the different features that they have. Um, and uh, we deploy a lot of these models using some of these techniques that I'm going to be talking about. Um, so there's uh, different techniques. So, we're going to talk about quantization. I think this one's like the main big one. Um, it's super important. We're going to talk about speculative decoding. Uh, this is a pretty cool technique.

它可以显著提升你的吞吐量。嗯，我们会聊聊如何调整你的 LM、VLM 配置，确保充分发挥性能并满足 SLA。然后我们会实际进行一些调优和评估。我会给大家提供一个框架，帮助你们确定具体需要使用什么配置。It can give you quite a big uh performance boost in your uh throughput. Um, we're going to talk about um like how do you uh you know uh tweak your LM uh VLM config to make sure that you're getting max performance out of it and um you're meeting your SLAs's. And then we're going to actually do some tuning and evaluating. So this I'm going to leave you guys with some sort of framework that you can use to figure out what exact configuration you need to use

模块五：量化选项Module five: quantization choices

针对你的特定工作负载和特定用例。嗯，我来深入讲讲量化。嗯，我们看看。如果时间允许，我们当然也可以进行一个简短的问答环节，不过我想我们有 30 分钟。我会以比较快的节奏讲一遍。量化有很多不同的类型。嗯，你知道，有 BF-16，也就是模型最初采用的格式。这是 Quint 提供给你的全精度模型，除此之外还有许多其他的量化方式。人们还开发了 FP8，这是我们将在本模块中了解的模型之一。嗯，还有 INT4、AWQ、GPQ、GPTQ。这些都是量化模型的不同方式。for your specific workloads and specific use cases. Um so I'll dive into quantization. Um let's see. And if we have time, we can definitely go uh we can have like a quick Q&A as well, but I think we have 30 minutes. I'm going to run through it uh at a good pace. So in quantization um there's a whole lot of different uh there's a lot of different types of quantizations you can do. Um you know there's the BF-16 the the one that the model is going to start off with. That's the full precision model that uh Quint provides you and then there's a lot of other conversations on top of that. People have created FP8 that's one of the models that we're going to look at in this particular module. Um there's INT4. Uh there's AWQ GPQ uh GPTQ. These are different types of uh ways you can quantize the model.

嗯，有 NVFP4，这是 NVIDIA 使用的专有量化技术。它在 Blackwell GPU 等硬件上可以加速运行。这个技术挺不错的。嗯，不过我们来看看，深入了解一下。嗯，希望我们能把这些内容都放进去。好。如你所见，如果使用 BF16 权重，会占用大量内存。嗯，在这个特定用例中，我们有 20 GB 显存，希望尽可能利用这 20 GB 显存来处理更多请求。嗯，所以我们要确保有足够的空间用于 KV cache。原因是，显存中能容纳的 KV cache 越多，就能并行处理越多请求，从而提高吞吐量。这非常重要。嗯，所以我们基本上需要进行量化。Um there's NVFP4 which is Nvidia's proprietary quantization technique that they use. Uh it's hardware accelerated on black hole GPUs and more. Uh that one is pretty cool. Um but let's see. Let's dive into it. Um make this hopefully we can fit that in there. All right. So as you can see, you know, like if you have the BF16 weights, it's going to take a lot of your memory. Uh in this particular use case, we have 20 gigs of VRAM and we sort of want to maximize how many requests we can serve with these 20 gigs. Uh so we want to make sure we have enough space for KB cache. Uh reason being the more requests uh the more KV cache you can have on the memory, the more requests you can process in parallel. Uh thereby increasing your throughput. So that's super important. Uh so that's why we essentially want to do quantization.

我们想缩小模型的尺寸，这样就能容纳更多 KV cache，从而提高吞吐量。这是其中一个原因。我认为另一个重要原因是，你基本上把权重缩减到一半。这样就不必在不同的内存层级之间搬运数据，因此解码速度也会更快。所以我们会看到，速度可能会从基础模型的每秒 40 个 token 提升到每秒 60 个 token 左右，甚至更高。我们来看看具体怎么做。先安装依赖项，然后我会确认模型仍然可用，一切都在正常运行。是的，模型确实在运行，还是同一个 40 亿参数模型，而且它运行在命名空间中。再说一次，正如我所说的，简单概括一下，量化有很多不同的技术可以采用。We want to reduce the size of the model so that way you can fit more KV cache and essentially increase your throughput. Um so that's one reason. The other big reason I think is also because you're shrinking the weights essentially by in half. You don't have to do that memory movement between the different uh memory hierarchies resulting in essentially faster decode as well. So we will see you'll go from you know 40 tokens per second which is the base model will probably go up to like 60 tokens per second if not more. Um so let's look into this. Let's um install the dependencies and then I'm just going to make sure that the model is still available and everything is running. So yeah, we do have the model running uh the same four billion parameter model and it's running in the namespace. Again, as I said, um sort of like a high level overview, there's a whole different sorts of different techniques you can do quantization with.

比如说，Llama CBP 就是一个相当受欢迎的工具，可以用较低精度运行很多这类模型。它使用 GGUF 格式。这个格式挺不错的，可以进行 5 位、3 位、2 位、1 位量化，这很有意思，也很酷。不过它的缺点是，为了获得更高的性能，你会损失很多质量。所以这其中存在取舍。Um for example, like um uh Llama CBP is a pretty popular one for running a lot of these models at a lower precision. Uh that uses GGUF format. Uh that format is pretty cool. you can have um uh you know five bit, 3bit, two bit, one bit quantizations which is pretty interesting um and pretty cool but the the downside of that is you lose a lot of uh you lose quality trying to get a lot of performance out of it. So, there's a trade-off there.

针对实际工作负载检查质量Check quality for the actual workload

嗯，不过总体来说，从 BF-16 切换到 FP8，质量和性能的下降都微乎其微。嗯，而且这样还能提升性能，同时完全保持原有质量。这就是我们想要展示的内容，也是我们所做的假设之一。显然，如果你要针对自己的特定工作负载这样做，具体效果取决于你的用途，我非常鼓励大家进行自己的评估。这样一来，如果你从 BF-16 切换过来，评估结果显示一切正常，工作负载运行良好；但切换到 FP8 后，你可能会遇到一些问题，比如有些东西会出错，有时会产生幻觉，有时会陷入循环。对于非常具体的使用场景来说，可能会出现很多这类情况。因此，拥有自己的评估集并运行自己的评估非常重要，这样才能确保你所做的任何更改都是可行的，同时也不会导致性能下降。Um, but for the most part, going from BF-16 to FP8 is a very negligible drop in quality and performance. Um, and it will result in more performance while keeping the quality exactly the same. Uh, so that's what we want to showcase and that's one of the assumptions we're making. Obviously, like when you're doing this for your particular workload, depending on what you're using it for, I highly encourage people to have their own evals. uh me so that way if you're switching from you know BF-16 your eval are saying okay everything is good everything's running well for our workload you switch to FP8 you might have issues where some things are breaking here sometimes it's hallucinating sometimes it's going in a loop so a lot of those things can happen for very specific use cases and in those scenarios it's very important that you have your own eval set and you're running your own evals to make sure any changes you make um are uh okay to make and also you don't have any degradation in performance

基准负载测试与并发性Baseline load tests and concurrency

嗯，我们再运行一次，确保使用的是 BF16 基础模型，然后测量性能。我们有一个小型负载测试工具包，用来做一些测试。基本上，如你所见，我们会在不同并发数下发送这些请求。每个请求大约包含 256 个 token，输出大约是 128 个 token。这样做是为了模拟你一次能处理多少个请求，我们会以此作为基准，看看基准性能如何。嗯，在并发数为 32 时，整体性能为每秒 714 个 token，相当不错。单并发时，我们的速度是每秒 37.5 个 token。嗯，这个数据需要记下来。Um again we'll run and make sure that we have the BF16 base and then we're going to measure the performance. So uh we have a little load test uh utility package that we use to do some uh testing. Essentially what we're doing if you look at the levels where these are like a at different concurrencies we're making this request. Each request is going to have about 256 tokens and the output is roughly going to be about 128 tokens. So this is sort of trying to simulate how many requests you can process at one time and we're sort of going to use this as a baseline to see what the baseline performance is. Um so for 32 uh we're getting aggregate performance of 714 tokens per second which is pretty good. uh at single concurrency we're getting 37.5 tokens per second. Uh so that's important to uh note down.

另外，首个 token 的时间是 61 毫秒。然后你会注意到，首个 token 的时间会变长，对吧？比如从 32 增加到 64，跃升相当明显。时间从 277 毫秒增加到 1300 毫秒，这是个很大的跃升。你未必愿意接受这种取舍，也就是说，你可能会选择最多只处理 32 个并发请求，而不再继续增加。所以这一点值得注意。嗯，正如我刚才所说，评测非常重要。原因是，量化可能会导致性能下降，也可能会损失一些智能，推理能力可能无法正常发挥，还会出现很多问题。因此，我这里其实是在用一种非常基础的方式模拟一下，你可能需要考虑哪些评测。And also we're getting um uh 61 milliseconds for time to first token. And then you'll notice the time to first token goes up, right? Like the jump between 32 and 64 is quite large. We're going for 277 milliseconds to uh 1300 milliseconds. That's a quite a big jump. and you might not necessarily want to make that trade-off, meaning you might want to potentially stop at only 32 uh concurrent requests that you're processing and not go higher than that. So that is an important thing to notice there. Um again as I was saying um eval are extremely important. Uh reason being again you can have performance degradation when you do quantization and you can lose uh some intelligence reasoning can uh not work as it should there there's going to be a lot of issues. So you want to essentially I'm kind of simulating here like a uh at a very basic level what kind of eval you might want to think about.

嗯，你知道，你需要考虑 JSON 是否生成正确，以及在降低量化精度时 JSON 是否会损坏。比如在 BF16 下，你可能能得到不错的响应；但切换到 FPA 后，函数调用可能就会失败。这是为什么？很可能是因为 JSON 没有正确生成。市面上已经有很多这类评测用例，你可以在开源社区里找到并使用。Hugging Face 上有很多评测；你也可以去 GitHub 上找。不过，针对自己的使用场景，你还是应该设计自己的评测用例。嗯，Um, you know, you want to think about like if the JSON is being generated correctly, if the JSON is being corrupted when you reduce the quantization, you know, at uh BF16, you might have good uh responses, but when you go to FPA, maybe potentially your functions function calls are failing. Why is that? It's probably because it's not generating the JSON correctly. So, there's a lot of these um eval cases that are already out there that you can kind of use in the open source. Hugging face has a lot of evals. You can go on GitHub, you can find a lot of these, but you want to come up with your own for your own use case. Um,

更改 Kubernetes 模型部署Change the Kubernetes model deployment

不过接下来我们要做的是从 BF16 切换到 FP8 模型。这样一来，模型的大小基本上会减半。完成后，我们就能看看性能差异如何、是否值得这么做，以及质量有没有下降。要进行这项操作，你需要进入根目录，然后点击 manifest，里面会看到 vlm.yaml。我来大致介绍一下。这是 Kubernetes 部署中常见的 manifest 配置文件。之前有多少人用过 Kubernetes，或者了解 Kubernetes？好，不少。太好了，那我就快速讲一下，不必介绍所有细节。基本上，我们有一个 PBC，它本质上是我们的缓存，所有模型都存放在里面；此外，还有一个简单的 VLM 部署，其中包含许多不同的参数和标志。but then so next thing what we're going to do is we're going to switch from BF16 to an FP8 model. So we are going to essentially reduce the size of the model in half. And once we do that, we'll kind of get to see um what the performance differences are and are they worth it and are we losing any quality over it. Uh so to do this uh you will need to be in the root and then click on manifest and in there you will see vlm.yaml. So um I'll kind of go over this. So this is a typical manifest uh configuration file that you uh uh have for Kubernetes deployments. Um how many of you have used Kubernetes before or have knowledge of Kubernetes? Okay, quite a few. Okay, great. So this will be quick. I don't have to go over all all the details, but essentially we have um a PBC which is going which is essentially our cache where all of our models are living and we have a simple deployment with VLM with a lot of different arguments and flags here.

嗯，你可以看到我们使用的是 Qwen 34B 参数模型。然后我们会把它切换到 Red Hat 的这个 FP8 版本。我还想快速提一点：你可以看到末尾的 dynamic。这一点相当重要。这意味着这个模型并非所有部分都量化为 FP8。在很多情况下，使用各种不同的量化技术时，你可能只想量化模型的特定部分，而不是其他部分，以确保性能保持不变，也避免出现很多问题。这也是你会看到 dynamic 等各种标记的原因之一。你甚至会看到这样的量化方案：模型的某些注意力头使用 4 位，有些使用 8 位，还有一些则保持全精度。Um so you can see we're using the Quen uh 34 billion parameter model. Um, and then we're going to switch it to this FP8 version from Red Hat. Uh, let me one more quick thing I want to quickly mention is you see the dynamic at the end. Uh, this is pretty important. This means that not everything in this model is quantized to FP8. In a lot of cases, in a lot of different quantization techniques, you might want to quantize the model uh only specific parts of the model and not the others to make sure the performance stays the same and you don't have a lot of uh issues there. So that's one of the reasons why you'll see dynamic, you'll see all kinds of things. You'll even see quantizations where certain heads of the model are going to be 4bit and then some are going to be 8 bit and some are going to be full precision.

而且你知道，大家这么做并从中获得很大的性能提升，这还挺有意思的。他们在确保质量保持不变的同时，获得了性能提升。所以修改好之后，我们会保存，然后快速过一遍，运行并应用。这只是为了确认差异没问题。嗯，我们不会在这里引发任何问题。嗯，应该没问题。然后我们就直接……“应用”在哪儿？就在这里。我们直接点击应用，它就会开始部署。嗯，这需要一点时间。我先把它保存一下。另外，你们还可以打开一个终端标签页。比如，点击加号，然后就会看到一个终端选项，这样你们也能看到命名空间里有什么，以及在这里可以做些什么。And you know, it's it's sort of uh interesting that people are doing that and finding a lot of performance there. They're they're finding the performance gain while making sure the quality stays the same. So once this is uh changed, we're going to save it and then uh we'll just quickly go through and run it and apply it. So this is just making sure that the diff is good. Um we're not causing any issues here. Uh that should be fine. And then we'll just go ahead and where's apply? Right here. We'll just go ahead and apply and that'll roll out. Um it's gonna take uh a little bit of time. Let me save that. But another thing you guys can also do is you can open up a terminal tab as well. Like if you just click on the plus sign and then you'll see a terminal option and then that will also let you see sort of like what's in your nameace and what you can do here.

所以，如果你看到的话，会发现 VLM 端点还在启动。呃，这个 pod 本质上就是在你的命名空间中运行的 Jupyter notebook。嗯，我们看看，这需要一点时间才能启动。对。在它启动的同时，有人有问题吗？我们大概可以在……提一个问题。So if you get you see the VLM endpoint is still coming up. Uh this pod is essentially your Jupyter notebook running within your namespace. Um let's see this will take a little bit of time to come up. Yeah. While this is coming up, does anyone have any questions? We can probably have one question in

Guest：cont. 已经预量化过了。Guest: cont. was pre-quantized already.

主持人：是的，它是预先量化过的。对，你也可以量化自己的模型。有 LLM Compressor。呃，我想它是由 VLM 提供的。LM Compressor。所以，如果你想自己动手，外面有相关工具可以用。嗯，不过，我们目前用的这个模型已经预先量化过了，是从 Hugging Face 下载的。好了，我想应该快启动好了。再说一次，这需要一点时间，主要是因为它要把所有权重从 VRAM 中移出来，再加载新的权重进去。所以这需要一点时间。嗯，还有人有问题吗？不好意思。Host: Yes, it was pre-quantized. Yeah, you can quantize your own models. There is LLM compressor. Uh it's uh it's provided by VLM, I believe. LM compressor. So there there there is tooling out there for you to do this on your own if you would like. Um, but yeah, the one that we're using currently is already pre-quantized that we downloaded off of Hugging Face. All right, I think that should be up. Again, it's taking a little bit of time, mainly because it's taking all the weights out of the VRAMm and loading the new weights in. So, it's going to take a little bit of time to do that. Um, anyone else have any question? I'm sorry.

嘉宾：看看你的、你的 VL，你的 YAML 文件里关于弧线的部分。看，哦，看起来他们在模型上有两个标志。Guest: Look at your your VL your YAML file for your arcs. Look Oh, looked like they had uh two flags on the model.

主持人：哦，他们确实这么做了。哦，提醒得好。谢谢你指出来。我没注意到。好吧，我们再过一遍。重新跑一次。好吧，这大概就是它没能像预期那样迅速出现的原因。哦，我做不了那个。好吧，在这段内容播放期间还有其他问题吗？请说。Host: Oh, they did. Oh, that's a good call out. Thank you for that. Did not see that. All right, we're going to go through it again. Let's rerun that. Okay, that's probably why it didn't come up as quickly as it should. Oh, I can't do that. All right. Any any other questions while this is going through? Go ahead.

主持人：你如何在两者之间做选择？Host: How do you choose between?

嘉宾：呃，FP8 和 int8 该怎么选？呃，这是个好问题。这取决于你的需求，对吧？int8 的精度会比 FP8 略低一些，基本上原因就在这里。呃，两者的区别在于，FP8 是浮点数，所以精度会稍微高一点。呃，而 int8 是整数。所以两者都可以用。呃，另外还有这些选项，你可以根据工作负载和自身的限制来选择。嗯，不过你完全也可以选 int8，应该没问题。嗯，这个在哪里说的？Guest: Uh, how do you choose between FPA and intake? Uh that's a good question. So it depends right like int 8 is going to be slightly less precision than FP8. That's essentially the reason. Uh the the difference is like FP8 is floating point so you get a little bit more precision out of it. Uh and then int8 is integer. So you could use either or. Uh again these are all the other options that you have. You can choose based on your workload and where your constraints are. Um but yeah you could totally go into eight as well and it should be fine. Um, where does it say that?

主持人：很好。Host: Good.

嘉宾：栈。哦，对。我是说，呃，哦，这也是另一个很好的提醒。嗯，这也取决于 GPU。有些 GPU 可能不一定支持它。Guest: Stack. Oh, yeah. I mean, uh, oh, that's the other good good call. Um, it depends on the GPU as well. Some of the GPUs might not necessarily support it.

主机：GPU 支持。Host: GPU support.

嘉宾：是的，所以 GPU 支持很重要。这也是一个很好的提醒。有些 GPU 可能支持它，但不一定能通过 GPU 加速，这意味着你可能无法获得预期的全部性能。比如，我们使用的 GPU 是 RTX 40008，它们支持 FP8，因此我们可以获得完整的精度和性能，而且这些计算都能通过硬件加速。但假设我们改用 FP4，就无法做到这一点，实际上也不会看到太多性能提升，反而会发现性能下降，而不是提升。所以这也是需要牢记的一点。Guest: Yeah. So, GPU support is important. That's another good call out. So some of the um uh GPU might support it but it might not be accelerated by the GPU meaning you might not get the full performance that you would be expecting out of it. Um for example the GPUs that we're using we're using RTX 40008 they support FP8 and we're going to get the full uh precision uh full performance out of it and it's able to hardware accelerate all those calculations. But if we let's just say use FP4, it won't be able to do that and we would actually not see a whole lot of performance gains and we actually see performance go down instead of it going up. So that's another thing to keep in mind.

比较 FP8 性能Compare FP8 performance

好的。它在哪儿？好了，已经成功推出了。我们快速看一下。现在可以看到，它实际上已经在使用 FP8 模型了。接下来我们也快速再做一次负载测试。测试内容完全相同：并发数设为 1、4、8、16 和 32。我想就是这些。测试会完全一样。等测试完成后，我们就可以进行比较，看看性能差异，以及为什么要进行量化。 性能数据出来了。比较前后结果的话，可以看到——我把它放大一点。你可以看到，即使在并发数为 1 时，吞吐量也从每秒 37.5 个 token 提升到了每秒 53.4 个 token。所以这里的性能提升相当明显。然后看峰值吞吐量，在并发数为 64 时，数据从 800 提升到了 40。All right. Where is it? All right. It has successfully rolled out. So we'll uh take a quick look. So now we can see it's actually using the FP8 model now. And then we'll also kind of quickly go through and do another load test. It's going to be the exact same. We're going to do it at uh concurrency 1 uh four 8 and uh 8 16 and 32 I believe over here. It's going to be exactly the same. So and then once that is up we can compare and see the performance differences and why you would want to quantize. Okay. the performance is in. And if we compare it before and after, you can see, let me make this a little bit bigger. Uh, you can see even at concurrency one, you know, we're going for 37.5 tokens per second to 53.4 tokens per second. So, we're seeing considerable jump in performance here. Um, and then at peak throughput, we're seeing uh at 64 concurrency, we're going from 800 to,40.

所以，大概是 1.4、4，呃，42 倍，这个性能提升相当大。不过还是要记住，与 32 和 64 相比，首个 token 的生成时间也明显增加了。然后，你可以再跑一遍评测。我们可以快速过一下，结果大概不会有什么不同。所以，BF16 和 FP8 相比，性能下降并不明显；通常应该开箱即用，不过在某些特定用例中可能不行。但对于我们这里这些简单的评测用例来说，效果不错。我想这个模块就到这里了，接下来我们要讲的是推测解码。So, that's about, you know, 1.4 four uh 42 times uh which is quite a big performance jump but again keep in mind the time to first token is going up considerably as well uh compared to 32 and 64 um but then yeah you can and then again you want to run evals um we can quickly run through this and it probably won't have any differences so yeah again BF16 versus FP8 um there's not a whole lot of uh performance degradation you should spec it's it's going to work out of the box for the most part but uh for specific use cases it might not but for simple eval cases that we had here it's it's working fine I think that is it for this module where we'll jump into next is speculative

模块六：推测解码Module six: speculative decoding

推测解码，我觉得这是个挺有意思的话题。嗯，我们来看看，直接开始吧。好，有多少人之前听说过推测解码？不少人啊，太好了。嗯，对于还不了解推测解码的人，我把这个缩小一点。嗯，简单来说，我们会使用另一个草稿模型，它比我们正在使用的主模型小得多，并让它提前生成多个 token。然后，我们的主模型，也就是这个 40 亿参数的模型，会查看这些生成的 token，并判断它们看起来是否合理。它会验证这些 token，看看是否可行。嗯，从这张示意图中可以看到，草稿模型提前生成了四个 token，然后验证模型，也就是我们的 40 亿参数模型，会说：“哦，只有第一个和第二个 token 有意义，另外两个不行。”于是它就会丢弃后两个 token。生成新 token 时，它基本上每次都会重复这个过程。decoding which I think is a pretty interesting one um let's see let's jump in here all right so how many of you have heard of uh speculative decoding before right quite a few people that is good. Uh for those who don't know what speculative decoding is, let me make this smaller. Um it's essentially we're using another draft model which is going to be a much smaller model than the main model that we're using and it's going to generate multiple tokens ahead of time and then our main model which is the four billion parameter model is going to look at those generated tokens and it's going to either say it's it looks good. That's it's going to verify it and say it's going to look good or it's not going to look good. Uh so in this like diagram you kind of see like um the draft model you know generated four tokens ahead of time and then the verifier model which is our four billion parameter model is going to say oh only the first and second token make sense the other two don't so it's going to discard it and it's essentially going to do this each time it's generating new tokens.

所以它会在那里进入一个循环。呃，对于这个工作坊，我们将使用 Quen 的 60 亿参数模型作为草稿模型，看看它在很多情况下能不能帮到我们。当然，它也可能帮不上忙，甚至降低性能。造成这种情况的原因有很多，我们稍后会讲到。我们时间还够吗？很好。呃，先确认一下这里的目标模型没问题。好，在开始之前，我还想简单介绍一下进行推测解码的各种不同方法。方法有很多，而且每隔几个月就会有新方法出现。呃，草稿模型只是其中一种方法。很多时候，新出现的做法实际上是使用扩散草稿模型，我觉得这真的很酷。So it's going to go in in a loop there. uh and uh for for this uh workshop we're going to be using the 6 billion parameter model uh from Quen as a drafter model and see if that is going to either help us or not help us in a lot of cases uh it can also not help you and reduce the performance um and there's multitude of reasons for that and we'll go into that shortly. How are we doing on time? Good. uh just make sure we are good here target model. Okay, before we get into it, I also want to talk briefly about all of the different ways you can do um speculative decoding. Um there's a lot of different ways and there's more ways coming out uh every every few months. Uh but the you know the draft model is one way. Um, a lot of times, uh, a lot of new, uh, ways of doing it is actually using a diffusion drafter model, which I think is really cool.

呃，我个人就在我的一台家庭实验室机器上用这个。嗯，这能带来巨大的性能提升。它和这个的区别在于，它是一个扩散模型，会一次生成很多 token，呃，而不是像我们使用 LLM 时那样按顺序生成。嗯，然后验证器模型就能验证其中一定数量的 token。嗯，不过，确实有很多不同类型的方法可以用。你可以使用后缀解码。呃，很多时候，模型也会自带一个起草模型。这些就像是与主模型一起训练的原生模型。呃，所以你不必再导入另一个模型来实现这一点。呃，还有 Medusa、Eagle；再说一次，扩散式 DFlash 是一个很重要的方法，最近有点火，而且效果相当不错。Uh, I personally use that for one of my home lab machines. Um, and, uh, that gives you a huge performance. The difference between that and this is because it's a diffusion model, it's going to be generating a lot of tokens uh, at once, uh, rather than doing what we do sequentially with LLMs. Um, and then the verifier model is able to verify X amount of tokens out of that. Um but but yeah there's a there's a lot of different types of uh methods you can use. You can use suffix decoding. Uh there's a lot of times model would come with a built-in drafter model as well. So those are like the native models that are trained along with the main model. Uh so you don't have to uh import another model to do this. Uh there's medusa eagle again like diffusion dlash is a big one that uh has kind of been going viral and it's working pretty well.

所以有一个配置项，用来确定验证器介入并验证之前，模型会生成多长的内容，也就是生成多少个 token。你可以把它设为两个、四个或五个 token。正确设置很重要，因为比如说，你把它设为四个 token，而模型验证器介入后表示“我只接受两个”，那接受率就是 50%，这就很糟糕了。实际上，这会带来负面影响，因为你不仅要运行一个模型，而是要运行两个模型，也就是说，你得花算力运行两个模型，结果接受率却只有 50%，最终会导致吞吐量下降。所以这不是你想要的结果。你得调整一下，多试几种设置，看看自己想提前生成多少个 token。So there's a configuration you set for figuring out how long like how much it generates before the verifier comes in and verifies it. So you can set that you can set it to two tokens, four tokens and five tokens. And it's important that you set it correctly uh because if there are for example let's just say you set it at four and uh the model verifier comes in it's like okay I only accept two that's 50% acceptance rate and that is pretty bad. that's actually going to have a negative impact because not only are you running you're running two models so you're spending compute to run two models and then you're only getting 50% acceptance rate and that's essentially going to result into a decrease like throughput. Uh so that's not something you want. Um so that's something you have to like kind of tweak and play around with and see how many tokens do I want to generate ahead of time.

嗯，所以从技术上来说，你可以生成 15 个 token，但你大概不想这么做。你可能会想生成两个、三个，也许四个。这还取决于你用的是什么类型的——嗯，叫什么来着？嗯，我回去看一下。在哪儿来着？你用的到底是什么类型的方法？比如说，我在家里的实验环境中经常用 deflash。把它设为八时效果相当不错。如果设得超过八，我的接受率实际上会低很多，这会导致性能下降，或者变得很不稳定。嗯，好吧，那接下来我们会……嗯……Um so yeah technically you could generate 15 tokens but you probably don't want to do that. You probably want to do two maybe three maybe four. It also depends what type of um what's it called? Um let me go back. Where is it? What type of method you're using again? Like uh I use deflash uh for a lot of my home labbing. And uh that works pretty well if I set it to eight. And if I set it more than eight, I actually get a lot uh a lot less acceptance rate and that causes my performance to go down or be very flaky. Um but all right so we will next um

主持人：你还有一个问题要问。Host: you got one more question

嘉宾：再问一个问题。嘿，呃，抱歉，呃，这是上一个模块的内容，不过，动态压缩是怎么工作的？我想用一两句话概括一下就行。Guest: one more question. Hey uh sorry uh this is from the last module but uh how does dynamic uh compression work uh in a sentence or two I guess

主持人：嗯，事情是这样的，如果你理解模型中不同的头，呃，因为 LLM 里有不同的层。你看一下 model.json 这个 JSON 文件，就会发现 LLM 由很多不同的部分组成，而且有一整套流水线，会把 token 从一个环节传到另一个环节。很多时候，当你看到动态量化时，它本质上是在说：好，我只会把注意力头量化为 FPA，但其他所有部分都保持原样。Host: um so the way it works is if you do you understand um the different heads of the model uh different because there's different layers within an LLM. So if you look at the model.json JSON, you'll actually see there's a lot of different parts to an LLM and there's a whole pipeline that sort of like pushes the tokens from one thing to another. And a lot of a lot of times when you see a dynamic thing, it essentially is saying, okay, I'm going to only quantize uh the attention heads to FPA, but I'm going to leave everything else as is.

客人：我明白了。Guest: I see.

主持人：所以，这就是其中的原因。如果你还记得前面的模块，你会发现 FPA 版本的大小并不是正好减半。它不是 4 GB，而是大约 5.1 GB。这是因为模型并非每个部分都经过了 FPA 量化。所以这算是其中的动态部分。Host: So, so that's the reasoning. And if you remember from previous modules, you you saw that the FPA version is not exactly half in size. It's not going four gigabytes, it's like 5.1 gigabytes. That's because not every part of the model has been quantized to FPA. So that's like sort of the dynamic part of it.

嘉宾：你看，同样的数学原理适用于所有模型吗？还是要视模型而定？Guest: See, is the same math applicable to all the models or it depends model

主持人：这因模型而异。每个模型的架构都不同。所以从架构角度来说，这取决于你量化模型的哪些部分。呃，有些人会把模型的某些部分量化到 in4，其他部分则保留为 FP8，还有一些部分保留为 BF16。所以……Host: depends model to model. Every model has a different architecture. So architecturally speaking, it depends you know what parts of the model you quantize. uh there's people who quantize you know some parts of the model to uh in4 but leave other parts to FP8 and then leave some parts to like BF16. So

客人：谢谢。Guest: thank you.

主持人：嗯。Host: Yeah.

配置草稿模型Configure the draft model

嘉宾：好。我们再跑一遍。我只是确认一下，我们的基准表现不错。这里用的是 FP8 模型。嗯，然后我们会再快速做一次负载测试，算是再看看在我们为 VLM 中的这个部署实现并添加推测解码之前，它的表现会是什么样。测试运行期间，我先确认一下……让我把这个正确地复制过来。不是那个。哦，没事。你可以直接……这里我们要设置推测配置。这是一个特定的标志，你可以在 VLM 中传入它来设置草稿模型。所以我们将它设为 6 亿参数的模型。然后我们还要设置要生成多少个 token 作为草稿 token。在这个场景中，我们将它设为 4。Guest: All right. We'll run through this again. I'm just confirming that our baseline is good. We're using FP8 model here. Um and then uh we'll do another load test quickly um sort of to get another baseline of um what it would look like before we implement and add speculative decoding uh to this particular deployment in VLM. And then while that is running, I'm just going to make sure uh to uh let me actually copy this correctly. Not that. Oh, never mind. You can just so in here um we are setting the speculative config. This so this is a particular flag you can pass in VLM to set the draft model. So we're setting it to the uh 0.6 billion parameter model. And then we're also uh setting how many tokens do we want to generate as draft tokens. So in this particular scenario we're setting it to four.

这又是需要进行大量测试的事情。嗯，你得试试看：要不要设成 4？要不要设成 2？还是设得更高？这完全取决于接受率，以及你能从中获得多少性能提升。嗯，然后，设置好这个标志后，我们会重新部署。嗯，我们再回到这里。基准值已经有了。接下来我确认一下是否正常。好的，这里的情况看起来不错。然后我们就应用设置并逐步推出。这可能还要再花几分钟。那么，有人有什么问题吗？请说。哦，等一下。Again this is something you have to do a lot of testing on. Uh you have to try out do I want to set it to four? Do I want to set it to two? Do I want to set it higher? It all depends on the acceptance rate and uh how much performance you're getting out of it. Um and then yeah once you set this flag and we will redeploy this. Um we go back in here. We got our baseline. And then let me make sure that it's good. Okay, things look good here. And then we'll go ahead and um apply and roll this out. And this is probably going to take another couple minutes. So, uh does anyone have any any questions? Go ahead. Oh, one second.

主持人：我我注意到你把 GPU 内存利用率设为 7】【。Host: I I noticed that you set the GPU memory utilization to 7

嘉宾：这会让你有很多……嗯，本质上会留出 30% 的显存。Guest: which leaves a lot of your um essentially leaves 30% of the VRAM

主机：未使用。Host: unused.

嘉宾：这个选择有什么特别的逻辑吗？呃，在这个具体场景中，原因可能有很多。呃，有时候你可能会在同一块 GPU 上部署多个实例，并且想确保有足够的 RAM 留给其他用途，也有足够的显存留给其他用途，然后再为这个部署留出足够的显存。但在这个具体场景中，倒也没什么特别的原因。实际上，我们会在后续的一个模块里把它调高。呃，这是其中一个需要调整的参数，对吧？你得确保利用率设置正确。如果设得太低，就会浪费性能；如果设得太高，呃，就会把节点弄挂，那可不好。Guest: Is there any particular logic to that choice? Uh in this particular scenario um there could be a lot of reasons. Uh sometimes you know you might have multiple deployments on the same GPU and you want to make sure that you have enough RAM for something else VRAM for something else and then you keep enough VRAM for this but in this particular scenario not really. We're going to bump it up in a following module actually. Um it's it's one of those knobs right like you have to keep you have to make sure that you're setting the utilization correctly. If you set it too low, you're leaving performance on the table. If you set it too high, um you're going to um kill your node, and that's not good.

主持人：对。所以，这个……Host: Yeah. So, this

嘉宾：好的，它还在显示。这需要多花一点时间。Guest: Okay, it's still coming up. This is taking a little bit more time.

主持人：请继续。从Host: Go ahead. from

客人：抱歉，稍等一下。Guest: sorry one second

主持人：呃，我们看到从 60 32 到 64 并行度的提升，是因为键值缓存一下子撑爆了，还是说……Host: uh the bump that we see from 60 32 to 64 parallelization it's because of the key value cache just blowing out or I mean

嘉宾：对，对，基本上就是键值缓存撑爆了，差不多碰到了它的最大限制，没法再往上了。嗯，原因有很多，这是一方面；另一方面还有连续批处理。很多时候，如果你把配置设在一个导致连续批处理无法达到最佳状态的级别，也会让你看到每秒 token 数很高。嗯，嗯，那里的延迟也会很高。嗯，不过是的，在这个场景下，我们基本上就是碰到了一个上限，嗯，嗯，也就是我们能使用的 VRAM 容量上限。Guest: yeah yeah essentially the key value cache is blowing out and it sort of hit like its max limit and it can't go above that uh so there's a lot of reasons so there's that part and then there's also uh the continuous batching and a lot of times times will happen if if you don't if you set the configuration to a level where the continuous batching is not optimal, it can also cause you to see like high tokens per sec. Uh uh you can see high latency there as well. Um but yeah, in this in this scenario, we're essentially hitting like sort of like a ceiling on us uh uh for us in terms of how much VRAM we can use

变慢而非加速A slowdown instead of a speedup

就是这样，所以我们才会看到这种情况。不过在接下来的几个模块中，我们会看看如何改善这种情况，希望能让它变得更好。呃，好吧。好了，已经部署好了。现在我们已经正确配置了推测解码。接下来我们再快速做一次负载测试。这和我们刚才在上面跑的测试完全一样。请记住，并发数为 1 时，我们的速度是每秒 58.2 个 token，延迟为 42 毫秒。所以这是启用推测解码之前的基线。嗯，不过我在这里想展示的并不是那种能自动带来惊人性能提升的理想情况。你实际上会看到性能大幅下降。现在的速度是每秒 16 个 token，而基线是 58。然后，首个 token 的生成时间似乎也有所增加。there. That's why we're seeing that. But in the following modules, we'll see how we can improve that scenario and hopefully like make it better. Uh, okay. All right, we're rolled out. So, we now have speculative decoding uh set up correctly. So, what we'll do is we'll do a quick load test again. It's the same exact test that we ran up here where, you know, just keep in mind we're getting 58.2 tokens per second for concurrency one and then 42 milliseconds of latency. So this is our sort of baseline before we do speculative decoding. Um but what I want to showcase here is not a happy path where you will automatically get a crazy improvement in performance. You'll actually see the performance went down drastically. You're seeing 16% uh 16 tokens per second whereas the baseline was 58. Um and then even the time to first token is sort of going uh up a little bit as well.

呃，你知道，我们想弄清楚这是为什么会发生。这里有很大的差异，性能也大幅下降：即使看并发数为 16 的情况，吞吐量也从每秒 700 个 token 降到了每秒 240 个 token。性能下降相当明显。对此可以有很多种解释。嗯，其中一个原因，正如我之前提到的，是接受率低。如果接受率很低，验证器模型就会不断拒绝草稿模型生成的 token，那么你就浪费了大量计算资源来生成这些草稿 token，接着验证器又会说：“嘿，我不想采用这些结果，这些工作做得不好。”嗯，这是一个原因。还有一种可能是我们把推测 token 数量设得太高了，也就是说，在当前配置中，我们把它设成了 4。uh and you know we want to figure out like why is this happening? So there's a there's a big difference and there's a big uh decrease in performance like even if you look at concurrency 16 we're going from 700 tokens per second to 240 tokens per second. So that's a considerable decrease in our performance. So the way to interpret this is there's a lot of different reasons. Um one is again as I was talking about earlier it's the low acceptance rate. uh if your acceptance rate is very low and the verifier model is just it's going to keep rejecting those tokens that are generating by draft model, you're wasting a lot of compute to generate those draft tokens and then the verifier is coming in and then saying hey I don't want to take this work like this is bad work. Um, so that's one. Um, it could be because we set the speculative tokens too high, meaning in in our current config, we set it to four.

也许我们需要把它设置好，然后看看这是否真的有助于提高验收标准或接受率。VLM 的调度器也有很多方面需要考虑。具体取决于它如何调度这些请求，这可能会导致部分性能未能充分发挥。因此，这也取决于兼容性等因素。然后，还是同样的 GPU 争用问题：你要用同一块 GPU 来执行这项工作，所以可能会导致资源没有得到合理利用。很多情况下，这种方法通常是可行的，但在我们的场景中却行不通。主要原因是，这个草稿模型并没有针对这个特定模型进行正确训练，所以我们看到的接受率非常非常低。因此，如果你要将这种方法用于自己的模型和部署，就必须确保所用的草稿模型与目标模型兼容。Maybe we need to set it down. Uh, and see if that actually helps in improve the acceptance criteria or acceptance rate. Uh, there's a lot of things with theuler as well with VLM. Uh, depending on how it's scheduling these requests in, uh, that could be leaving some, uh, performance out on the table. So, it also depends on compatibility and things like that. Um and then yeah again same GPU contention you're using the same GPU uh to do this work. So it could be uh resulting in u the resources not being used correctly. Um so in a lot of cases usually this works but in our scenario it didn't work. Uh the main particular reason for that is um this particular drapher model is not trained correctly for this particular model. So that's the reason we're seeing a a very very low acceptance rate. Um so when you are doing this for your own models and for your own deployments, you have to make sure that the model, the draft model that you're using is going to be compatible with the model that you want to use it for.

如果不这么做，你就会遇到我这里展示的这种情况：性能会大幅下降，而且你基本无能为力。从资源角度来说，我可以告诉你，这块 GPU 有足够的余量来支持这一切。但性能下降的原因是模型没有经过正确训练。这一点也要牢记，并进行相应测试。你还要确保推测 token 的数量设置正确。不过，你们有空的时候，我确实建议大家试着调整这个数字，比如从 4 改成 2、6 或 1，看看性能提升或下降了多少，以及为什么会出现这种情况。If you don't do that, you'll run into scenario that I'm showcasing here where like your performance is going to tank and um there's not a whole lot you can do about it. I can tell you from the resource perspective, this GPU has enough headroom to support all of this. But the reason for us seeing this decrease in performance is because the the models are not trained correctly. And uh that is something you want to keep in mind and test for as well. And you want to make sure the speculative tokens uh are set to the right uh number as well. Um but yeah when you in your own time I would definitely implore everyone to play around with that number meaning change this to from four to two to six to one and see uh what kind of performance gain you see what kind of performance decrease you see and why is that happening

回到基准方案，并讨论兼容性。Return to the baseline and discuss compatibility

不过，推测解码的内容就讲到这里了。它实际上帮不了我们解决这种情况，所以我就把那部分注释掉，然后重新部署。这样我们就回到了基线状态。but yeah that that's that is it for speculative decoding it is actually not going to help us in this scenario so I'm just going to go ahead and uh comment that out and then we'll redeploy this. So we are back to our baseline.

主持人：嘿，你有一个问题。Host: Hey, you got one question.

嘉宾：我有个问题。你好。嗯，我知道这可能会因模型架构而有所不同，但根据你的经验，总的来说，如果采用混合精度，比如主模型用 FP8 运行，再把草稿模型量化到 Q2，接受率会大幅下降吗？如果我的内存真的很有限，……Guest: One question. Hey. Um, so I know there's probably some nuance depending on model architecture, but in general from what you've seen, is there a massive hit to acceptance rate if you use like mixed precision essentially like run the main model at FP8 and then quantize the draft model down to Q2? If I'm like really memory constrained,

主持人：这样做通常真的值得吗？呃，或者你知道，总体来说你都见过些什么？比如说，这是否……Host: is it often even worth doing that? Uh, or you know, what have you seen in general? like does

嘉宾：这是个好问题。嗯，很多时候其实没那么重要。嗯，大多数情况下，你要做的是让草稿模型和主模型保持相同的精度。不过假设草稿模型是 BF-16，而主模型是 FP8，这种不匹配可能会导致接受率非常低。所以，关键在于你必须正确训练草稿模型，这样它才能生成正确的 token。在我们这个具体案例中，草稿模型没有经过正确训练。嗯，而且它还被量化到了 FP8。所以，他们训练这个模型时使用的配方可能存在一些问题，结果导致我们的性能遭受了巨大损失。嗯，很多时候，当某个实验室发布一个模型时，也会同时发布一个草稿模型。Guest: that's a good question. Um a lot of times it you it doesn't matter as much. Um most of the time what you want to do is you want to keep the draft model and the main model at the same precision. But let's just say the draft model is at BF-16 and then the main model is FP8. The mismatch can cause the acceptance rate to be very low. So, and then the the main part of it is you have to train the draft model correctly so that way it's able to generate the right tokens. In our particular scenario, the draft model was not trained correctly. Um, and then it was also quantized down to FP8. So, there was probably some issues with that recipe when they trained this model and which resulted in us actually taking a massive performance set. Um, so a lot of times when you when uh a lab releases a model, they release uh a drafter model along with it.

所以你想用的是随模型附带的那些起草模型。还有一些第三方实验室也会开发自己的起草模型。你可以试试市面上的各种选择，看看哪一种适合你。在某些情况下，具体效果取决于使用场景。比如说，如果你用它来编程，起草模型可能完全够用；但如果改用它来写英文，尤其是非常专业的学术文章，它可能就不一定表现得好，你会发现接受率有所下降。所以，这也很大程度上取决于你的使用场景，并不是普遍适用的。So you want to use those drafter models uh that come with the model. Uh there's also thirdparty labs that do their own drafter model. So you can try out a lot of different options that are out there and see which one works for you. And in some cases, depending on the use case, let's just say you're using for coding, the drafter model might work perfectly fine, but if you're switching that to maybe some English writing, very specific academic writing, and it might not necessarily work well, and you're going to see like decrease in acceptance rate. So, it's also uh it also depends a lot on your use case as well. So, it's not universally applicable.

主持人：明白了。好，讲得通。谢谢。Host: Gotcha. Cool. Makes sense. Thanks.

嘉宾：是的。好吧，我们来聊聊推理这部分。不过在继续之前，还有其他问题吗？哦，对了，我可以把 YAML 放上去。嗯，我们之所以有推测解码，大概是因为对于模型来说，验证比解码更快。Guest: Yeah. All right. Let's jump on to the inference side of things. But before you make another jump, any other questions? Oh, yeah. I can put the YAML on there. Um, so the the fact that we have speculative decoding uh presumably is because um for for a model it's faster to verify rather than decode.

主持人：嗯。你能不能简单讲讲它底层是怎么工作的，或者说模型在底层做了什么？给个大致的心智模型也行。嗯，它是怎么验证的，才能比解码更快？你知道，它总不可能是先解码，再和草稿模型生成的内容进行比较吧。所以……Host: Yeah. Can you just give a little bit of mental model for like what it's what a model does underneath the hood um to verify such that it's faster than decoding you know it can't be decoding and then comparing against what the draft model produces. So like

嘉宾：它具体在做什么，才能让速度变快？Guest: what is it doing exactly so so that it's faster.

主持人：是的。所以基本上，典型的 LLM 在解码时，会有一组词汇，它是一个由大量数字组成的巨大数组，表示其中每个项目的置信度分数，对吧？而使用推测解码时，不是一次处理一个 token，而是并行处理很多内容，也就是并行运行模型。再加上连续批处理，你也会并行处理许多不同的请求。所以你做的是同样的事情：对所有内容进行计算，并同时处理许多不同的请求。推测解码也是一样的做法：检查这些推测 token 的置信度分数，确认它们的置信度很高，然后就采用它们。Host: Yeah. So essentially uh when a typical LLM is you know decoding you have a set of vocabulary it's a huge array of bunch of numbers and it's representing um a confidence score for each of those items right so instead of um instead of doing that one token at a time when you run with specular decoding you're running it for a lot of you you're running the model for parallel things. So with continuous batching as well, you're running a lot of different requests in parallel. Uh so you're doing the same uh you're doing the calculation for everything. Uh and you're doing it at once for a lot of different requests. In the same way, you're doing the same thing for speculative decoding. You're checking the confidence score for this uh the speculated tokens and making sure that oh this is a high confidence, so we're going to go for it.

比如这个是 0.95，这个大概是 0.9。然后你接受这个，其他那些，你知道，就把它们拒绝掉。大致的思维模型是：你并行地逐层处理，确保草稿模型输出的每个 token 都有较高的置信度。对，希望这回答了你的问题。哦，还有一个问题。This is you know 0.95 and this one is like 0.9. And then you you accept that and then the other ones you you know uh you reject those. That's sort of roughly the mental model is like you are in parallel running uh through the layers and making sure that each of the tokens that the draft model uh outputs is has a high confidence. Yeah, hopefully that answers your question. Oh, one more question.

客人：你有没有什么例子可以……Guest: Do you have any examples that

主持人：呃，是的，确实如此。这个提醒得很好。嗯，我其实在这里留了几个链接。嗯，我把它们放哪儿了？解读结果。对。关于推测解码，网上有很多不同的文章。只要做得对，性能确实会有相当大的提升。VLM 实际上对此做过一份大型报告，他们发现，如果使用合适的模型、合适的草稿模型，并正确设置各项参数，令牌生成速度大约可以提升到 1.5 倍。嗯，所以，是的，它的表现确实出色。关键是要把它配置好、正确调优，并确保各项条件都准备妥当。嗯，不过就我的个人经验来说，在我的家庭实验室里，我用的是 Quinn 3.6 350 亿参数模型，同时搭配 Dlash 草稿模型，根据具体用例，令牌生成性能大约能提升 2 到 3 倍。Host: uh yes I do. That's a good call out. Um there's actually I left a few links in here. Um where did I leave those? Interpreting the results. Yeah. So there's a lot of different uh writeups on speculative decoding in in general. If you do it right, you do see quite a bit of increase in performance. Uh VLM has actually done a big report on this and they've seen about 1.5x increase in token generation if you set it up correctly with the right model and with the right drafter and the right settings. Um so yeah, this this does excel. It's just a matter of getting it right and tuning it correctly and making sure you have the right um things in place for it. Um but yeah, like from my personal experience, um for my home lab, I used the Quinn 3.6 35 billion model and along with that I use a Dlash um um draft model and I get roughly around two to 3x token generation uh performance improvement uh depending on the use case.

所以它确实能用。关键只是让合适的模型协同工作。So it it does work. It's just a matter of getting the right models to work together.

嘉宾：你如何选择模型大小？Guest: How do you choose the model size

主持人：给 daffer？Host: for the daffer?

嘉宾：嗯，你基本上是想找出一个你能运行的最小模型，它不会占用太多 GPU 算力，因为主模型才是拥有所有知识的模型。那才是你真正关心的、拥有你所需一切的模型。但你不希望草稿模型占用主模型太多资源。所以你希望它足够小，但又不能小到太笨，同时还要足够大，能够真正给你……Guest: Uh you want to you basically want to figure out uh the smallest model that you can run that doesn't take as much GPU power because you the main model is the model that has all the knowledge. That's the model that you really care about that has everything that you need. But you don't want draft model to take a lot of those resources away from the main model. So you want it to be small but not so small that it's dumb but big enough where like it's going to actually give you

主持人：你试过那个。你有没有看到过类似 3B 模型的接受率？Host: you experimented with that. Did you see like acceptance rates like 3B model

嘉宾：3B 太大了。呃，原因是很多模型，比如我刚才提到的 qu 3.6，呃，350 亿参数。它是一个混合专家模型，每个专家大约有 30 亿个参数。所以专家的规模可以在这个范围左右，而且性能也会不错。不过对于草稿模型，你可能希望它少于 10 亿个参数，以确保……我想这也取决于你用的主模型，对吧？比如说，如果你的主模型有 4000 亿个参数，那你大概可以用一个更大的草稿模型，以确保获得更高的接受率。所以基本上，你得找到合适的平衡。对。很好。嗯，那边还有其他问题吗？你开始之前再问一个。Guest: 3B is too big. Uh the reason why is a lot of models like I was talking about qu 3.6 uh 35 billion. So it has it's a mixture of experts model and each expert is about three billion parameters. So the experts can be around that uh and you would get good performance out of it. But for the draft model, you probably want to be less than 1 billion parameter just to make sure that it also depends on the main model you're using I guess right like if you're using if your main model is like 400 billion parameters you could probably get away with using a bigger drafter model to make sure that you get higher acceptance rate. So it's you got to find the right balance essentially. Yeah. Cool. Um, do we have any other questions over there? One more before you jump in.

主持人：嗯，我有个问题。草稿模型是怎么训练的？比如说，是通过蒸馏，还是用其他方法？Host: Yeah, I just had a question. How is the draft model trained? Like is it distillation or something else?

嘉宾：本质上就是蒸馏。呃，你把大模型的各种能力都蒸馏出来，创建一个数据集，然后用它来训练草稿模型。实际上，很多时候有不同的技术可以做到这一点。呃，你还会看到另一种做法：草稿模型和主模型一起训练。呃，这也是一种实现方式，而且很多实验室实际上会在同一个软件包里提供草稿模型。这样你就不必再下载第二个……Guest: It's essentially distillation. Uh, so you take the big model and you know you distill all of its capabilities. You create a data set and then you train the uh drafter model using that. And a lot of times actually what's there's different techniques to do this. Uh the other techniques you'll see is the draft model is trained alongside the main model. Uh so that way that's another way of doing it and a lot of labs will actually give you a drafter model uh inside of the same package. So you don't have to essentially download a second

模块七：饱和度与服务标志Module seven: saturation and serving flags

模型并运行它。嗯，做这件事有几种不同的方法。不过，可能最好的方法是同时进行训练，这样就不会遇到太多问题，而且接受率也能保持在较高水平。好。我们进入第七模块。这里我想聊聊使用 VLM 部署时需要关注的一些不同标志，以及如何尽可能提高饱和度。所谓饱和度，就是尽可能长时间地让 GPU 全力运转。毕竟，无论你是租用还是购买 GPU，都希望物尽其用，让它尽可能多地运行。好，我们先到这里。接下来我想讲一个概念。还记得之前我们看过并发数为 34、32 和 64 的情况吗？model and run it. Um so there's a couple of different ways of doing it. Uh but probably the best way is train alongside it and then that way you don't have a lot of issues and acceptance rate sort of stays high. Cool. Um, we'll jump into module seven. So, here I want to talk about some of the different flags that you uh should care about when deploying using VLM and what and how you want to get the saturation to be as high as possible. Saturation meaning we want to get the GPU to work as hard as possible as long as possible. Uh, and because you want to get your money's worth if you're renting a GPU or if you're buying a GPU, you want to run it uh as much as possible. So uh we will stop here. So uh one of the concepts I want to talk about uh if you remember previously we were looking at the 34 uh 32 concurrency and 64 concurrency.

我们看到延迟急剧上升，原因基本上是：吞吐量在某个时点会趋于平稳，但如果你继续提高并发量，延迟还会持续增加。因此，你要找到一个最佳平衡点。VLM 可以让你调整参数，确保找到这个平衡点；找到之后，就可以为它设置合适的参数。基本上，你要找的不是两者的交点，而是吞吐量即将趋于平稳、同时延迟仍然可以接受的位置。这就是你想达到的状态。我们先进行设置，然后确认各项参数仍然合适。模型服务已经启动。We saw a steep increase in the latency and the reason for that is essentially you the throughput at some point is going to flatten out and but the uh but the latency can keep increasing if you keep increasing the uh concurrency. So you essentially want to find the sweet spot and uh that is something that VLM sort of lets you kind of tweak and make sure that you find that sweet spot and once you do find it you can set the right settings for it. Um but but essentially you want to find the inter not so not the intersection but essentially right before where the throughput gets to like the flat state but the latency is like still acceptable enough. It's sort of like the point you want to get to. Um we will do a setup and then let's make sure our settings are all good still. Serving model is up.

嗯，我再确认一下它是否已经启动。应该是。好。然后这里我们还要再做一次负载测试。这样我们就能了解数据，然后据此找出那个点在哪里。本质上，我们把它称为“拐点”（knee），也就是各项指标开始趋于平缓的地方，同时根据并发量来看，首 Token 时间也是可以接受的。嗯，一旦……哦，我觉得这样应该没问题。所以你们可以看到，随着并发量增加，吞吐量也在缓慢提升。这是可以接受的。嗯，你们也会看到延迟有所上升。这也可以接受，但不会大幅上升。不过随着这里的并发量进一步增加，我们很快应该就会看到明显的跃升。在它运行的同时，大家还有其他问题吗？Um, let me just double check if it's up. Should be. Okay. And then here we're again going to get um another we're going to do another load test. So that way we know the data and then we can based on that figure out where that uh where this point is. Essentially the we call it the knee um where things are flattening but also the concurren based on concurrency uh the time to first token is acceptable. Um once that is Oh, I think that should be fine. So you know the throughput is slowly increasing as concurrency is increasing. That is acceptable. Um and then you also see the latency go up. That is also acceptable but it's not going up drastically. Uh but we should shortly see the big jumps uh with more um concurrency here. But while this is running, any any other questions for anyone?

主持人：请继续。哦，在那边。Host: Go ahead. Oh, over there.

主持人：所以我想再问一下前面提到的推测解码。嗯，我想根据你的经验了解一下，对于像草稿模型这样的模型，你见过它在哪些用例中效果很好？比如在包含工具调用的智能体循环中，或者用于编程？你见过它适用于哪些用例？Host: So my question was again on the previous speculative decoding. Uh just wanted to know based on your experience um with like drafter model uh what kind of use cases you have seen it to work really like for example with agent loops with tool call or with coding like what kind of like use

嘉宾：呃，Agent Loops 很擅长调用工具，也非常擅长，呃，使用浏览器。我用它的一个特定场景就是浏览器操作。我想自动化很多浏览器任务，嗯，因为你知道，很多事情不一定都有 API，所以你得弥补这个缺口。呃，还有，呃，drafting model 能帮你更快地生成更多 token。所以你会看到浏览器代理基本上能更快地四处运行、执行任务。呃，所以从很多方面来说，它确实有帮助。呃，我想说，浏览器操作方面我大概能看到 1.5 倍的提升，而编程方面则接近 3 倍。所以，具体效果取决于你的使用场景，可能会更高或更低之类的。呃，接着问一个相关问题：有没有什么基准测试？比如那种“信我就行”的基准，像 coding fire ship 这种？Guest: uh agent loops it's really good at tool calling it's really good at u browser use is the one particular use case that I use it for I want to automate a lot of browser tasks um because there's still you know you don't necessarily get APIs for everything and so you want to bridge that gap. Uh and uh yeah, the drafting model helps you generate more tokens faster. So you see your browser agent just essentially run around and do things faster. Uh so in a lot of ways it does help. Uh I will say for browser use I probably see around 1.5x increase whereas for coding I see like closer to 3x. So depending on your use case you may see more or less and and things like that. Uh, and a followup on that one. Are there any benchmarks like more trust me bro like coding fire ship?

主持人：确实有一些基准测试。我个人每次看到基准测试都会有点怀疑，因为我从来得不到相同的结果。不知道大家是否认同。嗯，所以总体来说，你最好自己测试，看看什么适合你，因为这很大程度上取决于你的使用场景；接受率会因你的使用场景和用途而有所不同。Host: There there's definitely benchmarks out there. I I'm I'm personally a little skeptical of benchmarks whenever I see them. I never get the same results. I don't know if people agree with that here. Um, so, um, yeah, for the for the most part, like you know, you want to do your own testing and see what works for you because it very much depends on your use case because the acceptance rate is going to be different depending on your use case, what you're using it for.

访客：谢谢。Guest: Thank you.

主持人：嗯，是的。还有其他问题吗？Host: Um, yeah. Any other questions?

嘉宾：试音。试音。怎么样？嗯，像 Codex 或 Claude 这样的大模型会采用这种编程方式吗？他们有没有公开过自己使用这些模式的信息？还是说这更多是针对开源项目，比如你自己训练模型或运行推理时用的？Guest: Mic check. Mic check. How's it going? Um do like big models like codeex or cloud like do they use this type of the coding like have they published that they use these patterns or is this more like for open source if you're training your own stuff or running inference

主持人：我不确定他们会不会这么做，嗯，我想他们会提供某种免费的性能提升。不过，对于开源模型来说尤其如此——所有你想用的开源大模型，比如 Qwen 模型、GLM 模型、Kimi 模型，都有可供使用的投机解码草稿模型。Host: I don't know the question to if they do it or not um I would imagine they do sort of like free performance but uh for for the open source yeah this is particularly for the open source all the open source big models that you want to use all the Quinn models, the GLM models, Kimmy models, they all have specular decoding uh drafter models available that you want to use.

主持人：比如说，他们有没有使用某个主要的起草模型，比如 DeepSseek 或 GLM？Host: Is there like a main drafting model that they use for example for like DeepSseek or GLM?

嘉宾：我记得 DeepSeek 昨天，或者前天，确实推出了一个新东西，叫 DSpark。它有点像是他们自己的一种技术，能以更动态的方式进行投机解码。基本上，他们在研究如何实现这一点。我们这里的配置是静态的，对吧？比如说，我们一次生成四个 token；而用 DeepSeek 的方法，生成数量是动态的，也就是说，它会在运行时自行决定：“我只生成两个 token”，或者“我生成四个 token”或五个 token，具体取决于接受标准、GPU 负载以及各种指标。所以，每个模型、每个实验室也都有自己的实现方式，你可能会想沿用他们的方案，不过也可以自己试验。比如，Quinn 模型方面，Quinn 实验室并没有发布用于草稿生成的 Dlash 模型，不过有第三方实验室发布了这类模型，而且和它们配合得相当不错。Guest: Deepseek actually came out with a new uh thing yesterday I believe or the day before. It's called DSpark. It's like their own technique for doing specular decoding in a more dynamic way. Essentially they're figuring out how so here our configuration is static right like we say we're going to generate four tokens at a time uh but with deepseek's way of doing it they are making it dynamic meaning you know it on on the fly sort of decides oh I'm going to generate only two tokens or I'm going to generate four tokens or five tokens depending on the acceptance criteria depending on the GPU load and all all sorts of metrics. So yeah, every every model uh every lab has like their own way of doing it as well and you probably want to stick to that but you can also experiment. For example, the Quinn models uh Quinn lab doesn't release the Dlash models for drafting but these are released by uh a third party lab but they work quite well with them.

所以你大概得试试看才知道。So you sort of have to test it out and see.

主持人：谢谢。Host: Thank you.

客人：嗯，我觉得这个挂得正好。Guest: Yeah, I think this is hanging right.

从笔记本故障中恢复Recover from a notebook failure

主持人：是的，我会检查一下锅。嗯，这里可能弄死了一簇。呃。Host: Yeah, I would check the pot. Yeah, might have killed a cluster here. Uh

客人：哦，我没法打字。呃Guest: oh, I can't type. Uh

主持人：哦，Host: oh,

访客：这之前运行得很好。Guest: this was working well.

主持人：刷新。Host: Refresh.

嘉宾：我再创建一次这个标签页。好。嗯，呃，出了点问题。它复制了一个 URL，还在新标签页里打开了。看看会发生什么。我打开一个新标签页吧。我不会把那个关掉。你知道吗，在之前所有的测试中都没发生过这种事，今天倒是发生了。至少我可以滚动浏览、看看不同的内容吗？好，至少我能滚动浏览。嗯，你知道，既然集群决定去度假了，我就大致过一遍原本打算讲的内容。嗯，在这里运行这个基准测试，主要是为了展示那个拐点在哪里，也就是说，你会在什么地方看到最大吞吐量。呃，我们回到这张图，你可以看到最大吞吐量，同时确保生成 token 的延迟最低。嗯，基本上，这个限制是在并发数为 32 时达到的。Guest: Let me create the tab again. Okay. Um well, something is wrong. It copied a URL and opened in a new tab. See what happens. Let me open a new tab. I wouldn't close that one. You know, in all the testing that has not happened, it happened today. Can I at least scroll through and go through different things? Okay, at least I can scroll scroll through um you know since uh the cluster decided to take uh a vacation uh I'll sort of go through the content that I was going to cover. Um the main point of running this benchmark here was to kind of show where that knee is meaning where you see the max throughput. Uh if we go back to this diagram where you see the max throughput while making sure you are seeing the lowest uh tokens uh lowest latency for token generation. Um and that that is that essentially limit was at 32 concurrency.

嗯，因为并发数超过 32 之后，你会发现令牌生成基本上都差不多，不会有太大变化。吞吐量也会保持不变，但延迟会大幅上升，而这是我们不希望看到的。所以，这可以说就是你遇到的限制。部署这些东西时，我们需要牢记这一点，并关注相关情况。嗯，如果……Um because anything above 32 concurrency you'll see uh the token generation is going to be kind of the same. It's not going to change much. uh the throughput is going to be the same but but the latency is going to go drastically up and you don't want that. Uh so that's sort of like the limit you are hitting. Um and that is important information that we need to keep in mind and look at when we're deploying these things. Uh and if

主持人：是的，你可以调到 100。去看看电子表格。Host: yeah you can go to 100. Go to look at the spreadsheet

访客：100。Guest: 100.

主持人：嗯，调到 100，然后就从……继续。Host: Yeah, go to 100 and then just continue from

嘉宾：是的。Guest: Yeah.

主持人：七个还是八个。它在哪里？Host: seven or eight. Where is it?

客人：给你。Guest: There you go.

主持人：好吧。嗯，这个能用了。不错。好。那我们就开始吧。Host: All right. Well, this one's working. That's good. All right. Cool. So, we'll jump into

模块八：调优与评估Module eight: tune and evaluate

八。嗯，我也打开一下“谁在做？”eight. Um, let me also open up Who's doing it?

嘉宾：好，那么在模块 A 中，我们会快速聊聊可以调整哪些标志和参数，以确保能够突破我们刚才提到的、并发数达到 32 时的瓶颈。你们会看到这个限制，那么我们该如何突破它？在考虑这个问题时，又有哪些事情需要牢记？这部分我想给大家提供一个思考模型：本质上，你需要进行测量。这也是为什么我在很多不同的环节都会反复进行负载测试——进行这些测试非常重要，这样才能了解性能表现，以及你所做的任何改动会如何影响性能：是让性能变好了，还是变差了？又具体是哪个指标变差了？比如说，如果吞吐量持续上升或保持不变，但延迟却发生了变化，那么其中就一定有对的或不对的地方。Guest: Okay, so for module A, we're going to talk quickly through um what sort of flags and knobs you can tweak to make sure that you can sort of get above that limiting point that we were talking about at 32 concurrency. You're sort of seeing that limit and how do we kind of get beyond that and what things you need to keep in mind when thinking about this. Uh so for this one I'm sort of going to give you guys um a thinking model is essentially you want to measure and that's why with a lot of different cells I was I keep running load tests and it's super important to run those load tests to see what your performance is like and any changes you're making how is that changing the performance is it making better is it making it worse um and in what particular metric it's is it making worse right like if is it if the throughput keeps going up or stays the same but like the latency is changing something is either correct or wrong there.

所以，这是我们必须牢记的一点。嗯，在 VLM 中有不少不同的参数可以调整，让它表现得更好。不过，主要有四个参数，首先还是 GPU 内存利用率。你要根据自己的使用场景，确保正确设置这个值。同样，如果你在同一块 GPU 上运行多个模型，可能会想把这个值设低一些，这是你需要做出的权衡。但如果你只运行一个模型，并且希望它尽可能发挥最佳性能，就应该把这个值提高到 0.9，甚至 0.95。嗯，另一点是要确保最大序列数也设在一个合适的上限。这需要通过测试来确定。嗯，在我们的具体部署中，如果你看一下 VLM.yaml，我们是从 32 个序列开始的，对吧？So, that's something we have to keep in mind. Um, and there's a bunch of different knobs you can sort of like uh tweak in VLM to make it better. But the main four ones are again the GPU memory utilization. You want to make sure that you're setting that correctly depending on your use case. Again, if you're running multiple models on the same GPU, you might want to keep it low and that's a trade-off you're making. But if you're only running a single model and you want to make sure that it's getting the best performance possible, you want to bump that up to 0.9 or point even 0.95. Uh the other thing is you want to make sure that uh the max numbum sequences uh is set to a good limit as well. This is something you have to figure out by testing. Uh in our particular deployment, if you look at the VLM.yaml, um we start with 32 sequences, right?

所以，它的作用是告诉 VLM 一次最多只能处理 32 个请求，不能再多了。如果超过这个数量，性能就会有所下降。有很多不同的标志参数，但这些是你需要记住的关键参数。你还要确保将模型长度设置为正确的值，并确保批处理 token 数也设置正确。如果你想更快地进行预填充，而且 GPU 能够处理，就把这个值设大一些；如果 GPU 性能稍弱，处理不了，就设小一些。不过，关键是一次只更改一个标志参数，不要同时更改多个设置。So what that is doing is it's telling VLM that it can only process 32 requests at a time uh and not more than that. And if you do more than that, you're going to see some performance uh degradation there. Um so there's a lot of different flags, but those are like the key ones that you want to keep in mind. You want to make sure that you're setting the model length to the correct length as well. Uh and then you know you want to make sure you're the batch tokens are also set to the correct length. um you don't want to you want to keep it bigger if you want to do prefill faster if your GPU can handle it. You want to do it smaller if your GPU is a on a little slower side and you can't handle it. Um but yeah, the idea is you want to change one flag at a time. You don't want to change multiple things.

嗯，当你更改内容时，运行一下……运行基准测试，看看性能表现如何。测量一下，然后根据结果判断接下来该做什么。嗯，我们快速过一遍这里的内容。我们大概还剩10分钟。呃，哦，Um and when you change things, run the uh run the benchmarks and see uh what performance you're seeing. See measure it and then make assumptions on what you should do next. Um, kind of go through it quickly here. We have about what 10 minutes left. Uh, oh,

主持人：我觉得我们把 P PVC 弄丢了。我想事情就是这样。我拿到了】【。Host: I think we lost the P PVC. I think that's what happened. I got

嘉宾：哦，它选了另一个。对，我确实得把这个改一下。我想得重新运行。嗯，我还是直接在这里改吧。这个本来应该是按顺序执行的，但另一个 Jupyter notebook 崩了。所以改一下，然后应该就能用了。嗯，就像我之前说的，有几个不同的参数可以调整，看看会发生什么。不过我们会先测一下基准，然后改几个标志位，看看接下来会发生什么。嗯，它说能运行吗？好，太棒了。好了，正在运行。趁这个负载测试还在运行，大家还有什么问题吗？Guest: Oh, it got the different one. Yeah, I need to actually change this. Rerun this, I think. Um, let me actually just change this in here. This was supposed to be like sequential, but other Jupyter notebook died. So, change that and then that should work. Um, again, like I talked about, there's different knobs uh that you want to tweak um and see what happens. But we'll get sort of the measure. We'll measure the baseline and then we'll change a couple flags and see what happens next. Um, does it say work? Okay, cool. There you go. It's working. While this load test is running, any other questions anyone has?

主持人：打扰一下。Host: Excuse me.

嘉宾：我注意到，在优化模型时确实会出现明显的性能下降。比如这里，从 32 到 64，下降幅度很大。呃，这种情况下，你会考虑是不是该扩充硬件规模，还是你有信心还能进一步调优？Guest: I noticed that there's a definite fall-off when we're optimizing the model. Like here, the 32 to 64, huge falloff. Uh is that when you would decide that maybe we should scale the hardware or is that something that you're confident you can still tune it further?

主持人：呃，基本上，这就是上限吗？Host: Uh essentially is this is this the ceiling?

嘉宾：是的，你可以通过计算得出上限。我记得在之前的一些模块里，Dwan 讲过每秒 token 数的上限可能是多少、吞吐量大概是什么水平，以及部署时应该抱有什么样的预期。如果你看到的性能已经达到了这些预期上限，那很可能就是硬件的极限了；要想再提升，可能就得添置更多硬件，或者换更好的硬件。嗯，所以确实存在一定的上限，能调优的空间也有限，但你的目标是尽可能榨取 GPU 的性能。你要不要在继续之前先分享一下幻灯片？快分享一下幻灯片吧。Guest: Yeah, you figure out the ceiling by doing mathematics. I think in some of the previous modules Dwan covered uh like what the max ceiling can look like for tokens per second what the throughput looks like and um you know what sort of expectations you should have for the deployment and if you are like seeing those max expectations that's probably the maximum limit of your hardware and then you probably have to get additional hardware or better hardware to do so. Uh so yeah there there is sort of a limit you know there's only so much you can tune but you're trying to squeeze as much out of the GPU as possible. You want to share the slides before you continue? Share the slides real quick.

工作坊资源与运行时问题Workshop resources and runtime questions

主持人：服务器我可以暂时开几个小时。不过我不会因为你们能使用 GPU 就一直开着它。嗯，我会提供给你们的是：扫描这个二维码，就能获取 PowerPoint 和代码仓库。你们可以把它部署到自己的 AI 云账户中，也可以在笔记本电脑上运行。扫描二维码后会进入一个网站，里面有所有相关资源。如果你们还没有云账户，可以注册一个免费账户，并获得 300 美元的额度。你们需要的东西都在这里了。如果大家有问题，我们两位的联系方式也在这里。奥马尔，我是 Dwan Lightoot。Host: The server I can leave it up for a little bit, couple hours. I'm not going to leave it up just because you do have access to GPUs. Um but what what I am going to give you if you scan this QR code you'll get access to the PowerPoint as well as the repo if you want to spin this up in your own AI cloud account or if you want to run on your laptop you can do that. So if you scan this QR code it'll take you to a site that has all the resources and if you don't have a cloud account you can sign up and you get $300 in credits with a free account. So that's everything you need. And if you all have questions, both of our contact information is here. Omar, I'm Dwan Lightoot.

嘉宾：是的。Guest: Yeah.

主持人：好了，既然如此，大家可以扫一下二维码，然后我让 Omar 做最后的总结。我只是想确保大家都能在有人开始离开之前收到这个。谢谢。Host: And yeah, with that being said, if you all scan the QR code, I'll let Omar finish it up. I just want to make sure you all get this before people started to leave. Thank you.

嘉宾：我们再把它放在那上面几分钟，就能拿到了。Guest: We'll leave it up there for a couple more minutes gets it.

主持人：嗯，请说。私人助理。呃，所以……Host: Yeah, go ahead. personal assistant. Um, so

嘉宾：所以你可以使用 Llama。这取决于你的硬件。我觉得这是我首先会考虑的事情。嗯，Guest: so you can use llamas. It depends on your hardware. I think that's like the first thing I would look at. Um,

主持人：没问题。谢谢。Host: no problem. Thank you.

嘉宾：哦，你两样都有。为了讨论方便，我们就假设你有最好的 GPU。就假设你有，嗯……Guest: Oh, you have both. for the sake of the argument, let's just say you have the best GPU. Let's just say you have um

主持人：是的，是的。你有最新的 Blackwell GPU。嗯，我觉得在测试以及确定你的用例要使用什么模型时，用 Ollama 和 llama.cpp 快速启动模型、试用一下，看看效果如何，是个不错的选择。不过，一旦你确定这个用例肯定要用某个模型，而且它运行得很好，你可能就会想换用 vLLM、SGLang 或其他模型运行时。原因是它们能支持更高的并发量，并提供更好的性能。所以，如果只有一个智能体在运行，我觉得用 Ollama 完全没问题，因为一次只有一个请求。但假设你要运行 20 个智能体，并且希望它们并行运行，那……对。Host: Yeah. Yeah. You have the latest Blackwill GPU. Um you know, I think for testing and doing like figuring out what model you want to use for your use case, I think it's good to use Olama and Llama CPP to quickly spin it up and try it out and see what happens. Uh but once you have a model that you know I'm going to use for sure for this use case and this this is working great you probably want to switch to uh VLM sg lang or one of these other model uh runtimes. Uh reason being they allow you to run at higher concurrencies and give you more performance out of it. So if you have a single agent running I think it's fine to use Olama. It's not a big deal because you only have one request going in at a time. But if you let's just say you're running like 20 agents and you want to run it parallel uh Yeah.

是的，你可以用 VLM 设置投机解码。据我所知，Olma 好像不支持投机解码，不过我也可能记错了。嗯，但如果你想充分发挥硬件的性能，肯定应该这么做。Yeah, you can set up speculative decoding with VLM. I don't think Olma supports speculative decoding as far as I know, but I could be wrong. Um, but yeah, it when you want to get the max performance out of your hardware, you definitely want to do that.

嘉宾：是的， еғирGuest: Yeah,

主持人：我其实不知道有没有扩散式的大语言模型。现在有开源的扩散模型吗？Host: I actually don't know if there's a diffusion uh LLM. Is there an open source diffusion model right now available?

嘉宾：是的，Lana。是的，我是说，我觉得 VLM 支持它。所以我觉得你可以在 VLM 上运行它。呃，如果 Olama 支持的话，你也可以用它。嗯，肯定是可以的。好，我们切回这个模块吧。Guest: Yeah, Lana. Yeah, I mean I think VLM supports it. So I think you you would be able to run it on VLM. Uh Olama, if supports it, you can as well. Um yeah, that's definitely possible. Okay, we'll switch back to the module

提高序列上限Raise the sequence limit

然后大致看一下。嗯，再说一次，就像我刚才说的，在 32 时性能不错，但到了 64，吞吐量基本持平，甚至还有所提升，不过延迟却显著增加了。要解决这个问题，我想谈谈最大序列数（max num sequence）。简单来说，它指的是 VLM 一次能够处理多少个请求，相当于设置这个标志。如果你想并行处理大量请求，就需要适当调高这个值。我记得目前它设为 32。我们可以把它调到 128，然后部署运行，看看性能会有什么变化。好，这里有不少变化。是啊，[哼笑]等它启动后，我们可以再做一次测试。你会看到，我们在 32 时遇到的限制会消失，而且即使在更高的并发级别下，延迟也会重新变得可控。and kind of go through it. Um, again like I was saying, at 32 you're seeing, you know, good performance, but at 64, uh, we're going at, you know, the the throughput is sort of staying the same, if not going up, but we're seeing, um, a significant increase in latency. And to kind of get around it, um, one of the things I want to talk about is the max num sequence. Again, this is sort of, uh, how many requests VLM is able to process at one time. It's sort of setting that flag. If you want to have a lot of uh requests being processed in parallel uh then you want to sort of increase this amount. So currently I believe it's set up at 32. So we can bump this up to 128 and then we can run this deploy it and see how that changes the performance. Okay, it changed a lot of things here. Yeah, [snorts] once this comes up, uh, we can do another test and, um, and you you'll you'll see the the limitations that we're seeing here at 32 is going to go away and you'll actually see the latency be manageable again um, at higher concurrency levels as well.

所以，这有点像你想要进入的那个循环。你要找到瓶颈所在，然后找出瓶颈，弄清楚为什么会这样，以及可以调整哪个标志，呃，确保你能克服这个瓶颈。嗯，这个问题提出来了。那需要花一点时间。嗯，有什么问题吗？那边还有一个。So, this is sort of like the loop you want to go in. You want to find the right flags for wherever the bottleneck is. find the bottleneck, figure out why that is happening, what flag you can tweak uh to make sure that you're kind of overcoming that bottleneck. Um it's come up. That's going to take a little bit of time. Um any questions? One more over there.

主机：流量可能会突增Host: Potentially bursty traffic

嘉宾：嗯，有时候。嗯。Guest: um at times. Um

主持人：哦，对。有没有所谓的突发扩容到无服务器架构这种做法？这属于上层模型路由层的范畴吗？这种模式是怎样的……Host: Oh yeah. Is there such thing as like bursting to serverless? Is that the domain of you know model routing layer on top? Like what is the pattern for

嘉宾：是的。嗯，我是说，自动扩缩容就是一种模式。具体来说，在 LKE 中，你可以进行动态自动扩缩容。这样它就能根据你的需求、根据突发流量进行扩容。比如，它可以扩容到使用一台配备四块 GPU 的节点，而不是一块 GPU 的节点，因为这样可能处理得更好，对吧？所以你可以把很多这类事情自动化，确保做得恰当。另一方面，你也可以构建一些统计模型，帮助你从较长的时间跨度上了解流量模式。然后，集群也可以动态地以某种方式预测流量的突发情况，并据此进行扩缩容。Guest: Yeah. Um I mean autoscaling is one of the patterns like specifically I'll tell you in LKE uh you can have uh dynamic like autoscaling. So it can scale up uh depending on your needs depending on bursty traffic. it can scale up to you know maybe use a node that has four GPUs instead of one because maybe that handles it better right so you can automate a lot of those things uh to make sure that you're doing it well and the other part of it is like you can also build like some statistical models to make sure that you kind of understand your traffic patterns over a long horizon and then the uh the cluster can also dynamically like sort of predict your burstiness in some ways and then can also scale based on that too

主持人：如果你打开浏览器，我可以告诉你它在哪儿。Host: if you got your browser I can show you where it's at

客人：浏览器，你打开浏览器，我把它带给你。Guest: browser you open your browser, I'll take it to you.

主持人：好，这个已经成功推出了。Host: Okay, so this one is successfully rolled out.

嘉宾：嗯，然后我们会快速进行一次负载测试，然后……Guest: Um, and then we'll do a quick load test and

再次比较吞吐量和延迟Compare throughput and latency again

等它运行之后，我们就能看到更多了。好吧。呃，还有其他问题吗？大多数人都走了，不过……once that is run, we can see more. All right. Uh, any other questions? Most folks are gone, but

主持人：对，就像脚 H 一样。Host: yeah, like foot H.

嘉宾：是的。32。上次我们看到的限制就在这里。等我们达到 64 时，应该会看到性能有所提升。好。所以，性能有了非常显著的提升。吞吐量实际上提高了不少。嗯，现在是每秒 1100 个 token。而之前，如果我们看这里的 64，当时是每秒 800 个 token，延迟是 2000 毫秒。但现在，在我们做了这项更改之后，吞吐量达到了每秒 1100 个 token，延迟则是 800 毫秒。所以延迟大幅降低了，吞吐量也大幅提高了。这些就是你可以随着时间推移进行的调优，确保系统运行良好；在系统启动并运行后，你可以针对自己的具体用例进行这些调优。Guest: Yeah. 32. That's where we saw the limit last time. Once we get to 64, we should see uh that it should improve. Okay. So, there was a big drastic improvement. The throughput actually went up quite a bit. Um so, it's at 1100 tokens per second. Whereas previously, if we look at it over here at 64, we were at 800 tokens per second and then the latency was uh 2,000 milliseconds. But now after the change that we made um it is uh 1100 tokens per second and then the latency is uh 800 milliseconds. So it's gone down the latency has gone down quite drastically and then we have also increased the throughput quite drastically. So the these are the sort of the tuning you can do uh over time to make sure that things are running well uh once you have it up and running and you can do this for your particular use case.

也许某个特定用例不会同时收到 64 个请求，但也有可能会。所以，你得根据自己的用例来决定。Maybe you're not going to have uh six uh you know 64 concurrent requests coming in for a particular use case but maybe you are. So um you have to decide based on your use cases

主持人：你的并发数比之前高了一个，是吗？这是你做出的权衡吗？Host: your concurrency one higher than before like was that a trade-off you just did

嘉宾：并发数为 1，我的意思是，它可能会稍微高一点或低一点，不会一直都一样。比如之前是 37，现在是 35。所以运行模型时会有一点偏差，不过应该是在上下两秒、两个 token 的范围内。嗯，我们现在也在运行全精度模型。我们不再运行 FPA 了，因为我们不得不切换 notebook。这也是你看到它从五十几个 token 变成三十几个 token 的另一个原因。Guest: concurrency one I mean it can be a little higher or lower. It's not going to be the same all the time but for example before it was 37 but now it's 35. So it's a little bit of deviation that happens with running models, but it should be plus minus two sec two tokens there. Um we're also running the full precision model now. We're not we're no longer running the FPA because we had to switch the notebook. So that's the other reason why you're seeing it go from like 50 something tokens to 30 something

最后的注意事项与总结Final caveats and wrap up

tokens。呃，这就是我们看到的性能表现。不过，嗯，我要讲的就这些了。这是一个你可以调整的标志。还有各种其他标志，我强烈建议大家都去看看。workshop 里也会介绍这些内容。不过我想我们时间差不多了，我把时间交还给 Dorne。tokens. Uh so that's the performance that we're seeing there. But but yeah, that is all I have. That is one flag you can tweak. There's also all sorts of other flags that I implore everyone to take a look at. The workshop kind of goes through it as well. But I think we're on time and I'll give it back to Dorne.

主持人：是的。感谢大家今天参加我们的课程。如果您还没有扫描二维码，代码、代码仓库和 PowerPoint 演示文稿都可以在这个网站上找到。欢迎免费注册一个 AMA 云账户。我叫 Dwan Lightoot，今天和我的同事 Omar 一起主持。感谢大家今天的参与。课程结束后，欢迎向我们提问。谢谢。Host: Yeah. Thank you all for attending our session today. If you have not scanned the QR code, the code, the repo, the PowerPoints are available on this site. Feel free to start a free AMA cloud account. My name is Dwan Lightoot and I'm joined by my colleague Omar and we thank you all for joining us today. Feel free to ask us questions after this session. Thank you.