Jev 能成为更好的智能体评估器吗？

关键要点Key Takeaways

如今，智能体评估主要有两种形式：基于代码的评估和由 LLM 充当评判者的评估。两者都有各自的局限性：基于代码的评估器只能用于输入固定的一小类问题，而 LLM 评判者可能速度慢、成本高且不可靠。随着 TypeSafe AI 的 Jev 广受欢迎地发布，我们想了解“System One”模型是否可能成为智能体评估器的第三种新形式，以及它可能对智能体工程产生怎样的影响。Today, agent evals come in two flavors: code-based and LLM-as-judge. Both have their own limitations: code-based evaluators can only be used for a narrow set of problems with set inputs, while LLM judges can be slow, expensive, and unreliable. With the popular release of TypeSafe AI’s Jev, we wanted to see whether the “System One” model might be a new third form of agent evaluator, and the impact it could have on agent engineering.

Jev 是什么？What is Jev?

Jev 是 TypeSafe AI 发布的新模型。Jev 实际上并不是传统的 LLM；它不会生成文本。它是 TypeSafe AI 团队称为“System One”的模型：Jev is a new model released by TypeSafe AI. Jev is actually not a traditional LLM; it doesn’t generate text. It’s what the TypeSafe AI team calls a “System One” model:

📖 System One 模型是一类旨在快速做出结构化决策、供软件直接使用的 AI 模型。System One 模型会评估某个状态，并返回带类型的答案和概率。

📖 System One models are a class of AI models built to make fast, structured decisions that software can use directly. A System One model evaluates a state and returns typed answers and probabilities.

据 TypeSafe AI 称，这使 Jev 的速度更快、成本更低；在分类任务上，其推理速度最高可比同类 LLM 快 200 倍，成本则降低至后者的 1/400。According to TypeSafe AI, this makes Jev faster and cheaper than LLMs, up to 200x faster inference and 400x lower cost than comparable LLMs on classification tasks.

如果你想进一步了解如何使用 Jev 构建代理，我们将于 9 月 22 日（星期二）与 TypeSafe AI 团队共同举办一场直播：https://events.langchain.com/webinar/building-a-harness-with-jev/

If you want to learn more about building agents with Jev, we're hosting a livestream with the TypeSafe AI team on Tuesday, Sep 22nd: https://events.langchain.com/webinar/building-a-harness-with-jev/

为什么 Jev 可能是一个优秀的智能体评估器？Why might Jev be a good agent evaluator?

如今的智能体评估要么基于代码，要么采用 LLM-as-a-judge，各自都有一系列优点、局限性和权衡。Agent evals today are either code-based or LLM-as-a-judge, each with its own set of benefits, limitations, and tradeoffs.

基于代码的评估与代码本身一样历史悠久。它成本低、速度快且可靠，但主要缺点在于能力范围较窄。传统函数需要设定确定性输入，因此其评估智能体行为这一随机性世界的能力受到限制。例如，传统函数可以评估智能体是否在首次运行时调用了某个工具，但若要评估智能体随后是否利用工具结果成功回答了用户的问题，就会更加困难。在开放式任务中，同一个工具结果可能有多种有效的使用方式，因此将每个可接受的答案都编码为确定性逻辑，很快就会受到基于代码的评估范围狭窄这一局限的影响。Code-based evaluation has existed for as long as code has. Cheap, quick, and reliable, its main disadvantage is in its narrower abilities. Given a traditional function’s need for set deterministic inputs, its ability to evaluate the stochastic world of agent behavior is limited. For example, while a traditional function could evaluate whether an agent called a tool in its first run, it would have a harder time evaluating if the agent then used the tool result to successfully answer the user’s question. In an open-ended task, there can be several valid ways to use the same tool result, so encoding every acceptable answer as deterministic logic quickly runs into the narrow-scope limitation of code-based evaluation.

这就引出了 LLM-as-a-judge，它利用 LLM 对智能体轨迹中的非结构化输入进行推理并评分。LLM 评判器可以接受问题、轨迹和证据等非结构化输入，然后通过提示词评估响应是否满足了用户的请求。Enter LLM-as-a-judge, which uses an LLM to reason through the unstructured input of an agent’s trace and score it. An LLM judge can accept the question, trace, and evidence as unstructured input, then use a prompt to evaluate whether the response addressed the user’s request.

不过，任何智能体工程师都会证明，LLM 评判器并不是完美的解决方案。它们本质上是非确定性系统，无法作为值得信赖的测试装置的坚实基础。与传统的基于代码的评估相比，它们的运行速度更慢、成本也更高。As any agent engineer will attest though, the LLM judge is not a perfect solution. They are inherently non-deterministic systems, which are not a solid foundation for a trustworthy testing apparatus. They are also slower and more expensive to run than traditional code-based evaluation.

Agent 评估是一项决策任务：给定智能体的状态和行为，分配一个能够提供反馈的分数。Jev 专为这种模式而设计。它针对结构化状态评估类型化问题，并以概率形式返回类型化答案。另一方面，自回归模型通过逐个生成 token 来得出判断。在我们的实验中，这种决策优先的设计与更低的延迟、更低的成本和更低的方差同时出现。Agent evaluation is a decision task: given an agent’s state and behavior, assign a score that provides feedback. Jev is designed for this pattern. It evaluates typed questions against structured state and returns typed answers with probabilities. Autoregressive models, on the other hand, reach a judgment through token-by-token generation. In our experiment, that decision-first design coincided with lower latency, lower cost, and lower variance.

Jev 支持三种类型的问题：Jev supports three types of questions:

• Choice 选择一个选项，并返回概率和置信度
  Choice selects one option and returns probabilities and confidence
• ‍Score 会根据有序评分标准对答案进行评分，并返回概率和置信度。
  ‍Score rates an answer against an ordered rubric and returns probabilities and confidence.‍
• Noul 返回是非判断为真的概率。
  Noul returns the probability that a yes/no judgment is true.
  

多个原子问题可以针对同一状态并行评估。Multiple atomic questions can be evaluated in parallel against the same state.

当智能体行为、检索到的数据或运行间的轨迹上下文发生变化时，比较各个评判器会变得很困难。Deep Agents 和 LangSmith 让我们能够将单次智能体运行捕获为数据集，并在每个模型上重放该运行。Comparing judges is difficult when the agent behavior, retrieved data, or trace context changes between runs. Deep Agents and LangSmith let us capture a single agent run as a dataset and replay it across each model.

使用 Jev 进行评估Evaluation with Jev

为了测试 Jev，我们需要一个评分代理。我们使用开源代理框架 Deep Agents 构建了一个目标代理。随后，我们将测试集定义为 LangSmith 数据集，以便每个评估器都针对相同的问题和预期行为运行。该测试集包含五个天气请求：In order to put Jev to the test, we needed an agent to score. We built a target agent with Deep Agents, our open source agent harness. We then defined a test set as a LangSmith dataset so each evaluator ran against the same questions and expected behavior. The test set consists of five weather requests:

对于数据集中的每个示例，我们都捕获了天气代理的响应，并将完整输出作为固定示例存储在 LangSmith 中。每位评审员都使用两个信号评估这五次捕获的运行：quality（连续评分）和 does_pass（二元判定）。For each example in the dataset, we captured the weather agent’s response and stored the full output as a fixed example in LangSmith. Each judge evaluated the five captured runs with two signals: quality, a continuous score; and does_pass, a binary decision.

为了将正确性与可重复性分开衡量，我们让一名人工审阅者依据同一评分标准为每个固定响应进行标注。以人工审阅者的标注作为基准分数，可以更深入地分析精确性和正确性对评估器整体有效性的影响。To measure correctness separately from repeatability, we had a human reviewer label each fixed response against the same rubric. Using the human reviewers labels as the oracle score enabled a richer analysis on the affects of precision and correctness on overall evaluator effectiveness.

准确性衡量的是与人工标准答案的一致程度。方差衡量的是，在智能体行为完全相同的情况下，评判者能否始终得出相同的判断。较低的方差并不自动意味着更高的准确性：评判者仍可能始终判断错误。但当评判者是准确的时，较低的方差会让这种准确性在生产环境中更加可靠。Accuracy measures agreement with the human oracle. Variance measures whether a judge reaches the same judgment consistently on identical agent behavior. Lower variance does not automatically mean higher accuracy: a judge can still be consistently wrong. But when a judge is accurate, lower variance makes that accuracy more dependable in production.

我们将 Jev 与 GPT-5.6 Luna、GPT-5.6 Terra 和 Claude Sonnet 4.6 进行了比较，计算了 100 次重复实验中各案例的方差以及与人工预言者的一致性。We compared Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6, calculating per-case variance across 100 repetitions and agreement with the human oracle.

评估结果Evaluation Results

准确性Accuracy

以人工审核员的标签作为此次比较的标准答案，我们计算了二元通过/失败判定的准确率。Using the human reviewer’s labels as the oracle for this comparison, we calculated accuracy for the binary pass/fail decision.

对于二元的 does_pass 得分，Jev 在全部 500 次重复决策中都与基准答案一致。Terra 的决策匹配率为 99.8%，Luna 为 96.4%，Claude 为 80.0%。For the binary does_pass score, Jev matched the oracle on all 500 repeated decisions. Terra matched on 99.8% of decisions, Luna on 96.4%, and Claude on 80.0%.

精度Precision

准确率告诉我们评判者是否与人工裁判的判断一致。精确度关注的是：当智能体的行为保持不变时，评判者是否会给出相同的质量分数。我们通过观察每位评判者评分的方差来衡量精确度。Accuracy tells us whether a judge agreed with the human oracle. Precision asks whether it produces the same quality score when the agent behavior is unchanged. We measured precision with the observed variance of each judge’s scores.

Jev 观测到的每案例平均方差最低：0.0000149。Luna 高出 433 倍，Terra 高出 913 倍，Claude 高出 92 倍。Jev had the lowest observed mean per-case variance: 0.0000149. Luna was 433× higher, Terra was 913× higher, and Claude was 92× higher.

这项实验无法告诉我们 Jev 的得分为何波动更小。一种假设是，这些模型针对不同类型的输出进行了优化。TypeSafe 将 Jev 描述为一种经过训练、能够返回校准概率和类型化答案的决策模型，而自回归 LLM 评审器会先生成文本，然后由评估器将该输出映射为分数。这一差异可能使 Jev 更适合这项边界明确的评估任务，但这一结果仅是观察性的，并不能证明其训练目标导致了更低的方差。This experiment cannot tell us why Jev’s scores varied less. One hypothesis is that the models are optimized for different kinds of output. TypeSafe describes Jev as a decision model trained to return calibrated probabilities and typed answers, while an autoregressive LLM judge generates text before the evaluator maps that output into a score. That difference may make Jev a better fit for this bounded evaluation task, but the result is observational, not evidence that its training objective caused the lower variance.

成本与延迟Cost and latency

低成本意味着大规模运行智能体评估可以切实可行。当评估器调用成本高昂时，团队必须在覆盖率和预算之间做出取舍。在本实验中，每次调用的成本为 0.00035 美元，Jev 让这种取舍不再那么严苛。团队可以承担更多重复评判和更频繁的回归检查。对于在线评估而言，这一点更加重要：更低的单次调用成本使团队能够在更大比例的生产轨迹上运行更多评估器，从而产生更密集的反馈信号。Low cost means running agent evaluations at scale can be practical. When evaluator calls are expensive, teams have to decide between coverage and their budget. At $0.00035 per call in this experiment, Jev makes that tradeoff less severe. Teams can afford more repeated judgments and more frequent regression checks. This matters even more for online evaluation, where lower per call cost lets teams run more judges across a larger share of production traces, producing a denser feedback signal.

大规模解锁在线评估Online evals unlocked at scale

对于一个每天生成 10,000 条轨迹的生产级代理而言，观察到的单次调用成本会转化为显著的运营差异。For a production agent that produces 10,000 traces per day, the observed per-call costs translate into a meaningful operating difference.

为了衡量低成本调用是否有用，我们将信号价值定义为二元预言机一致性与二元可重复性的乘积。可重复性是指对同一轨迹进行两次独立调用时返回相同判定的概率。这一指标会奖励既准确又稳定的评判器，同时惩罚始终出错的评判器。To account for whether a low-cost call is useful, we define signal value as binary oracle agreement multiplied by binary repeatability. Repeatability is the chance that two independent calls on the same trace return the same verdict. This rewards judges that are both accurate and stable, while penalizing a judge that is consistently wrong.

像 Jev 这样信号强、成本低的评判器，可能会在在线评估器中释放更高的价值。团队可以在更多生产环境轨迹上生成反馈，更早发现质量变化，并在反馈开始呈现不良趋势时设置警报。A high-signal, low cost judge like Jev could unlock better value in online evaluators. Teams could  generate feedback on more production traces, spot changes in quality sooner, and set alerts when that feedback starts to trend in the wrong direction.

一种新型的智能体评估方式A new type of agent evals

如今，每次智能体评估都伴随着取舍。对更多智能体运行进行评分、评估更多维度，或测试更多变更，都会增加测试成本。这会迫使团队减少评估他们原本希望评估的内容。Today, every agent eval carries a tradeoff. Score more agent runs, evaluate more dimensions, or test more changes, and the cost of your testing grows. That pushes teams to evaluate less that they would like.

在我们的实验中，一次 Jev 评判的成本为 0.00035 美元。除了成本低之外，Jev 还具有高准确率和低方差，这意味着评判结果可靠且信号强。以这样的价格使用高质量评判器，构建者就能针对多个聚焦标准评估每次智能体运行，衡量智能体的每项变更，并在置信度至关重要时重复进行评判。In our experiment, a Jev judgment cost $0.00035. In addition to its low cost, Jev offered high accuracy and low variance, meaning the judge results were reliable and high-signal. A quality judge at that price means builders can evaluate each agent run against several focused criteria, measure every agent change, and repeat judgments when confidence matters.

这很重要，因为构建出色的智能体需要进行大量测试和监控。评估智能体的频率越高，开发周期中获得的有用反馈就越多。This matters because building great agents requires substantial testing and monitoring. The more often you evaluate an agent, the more useful feedback enters the development cycle.

我们仍需观察这一实验中的结果是否适用于其他智能体和生产工作流。此外，低成本可能会放大错误——一个持续出错的评估器可能大规模地产生不良反馈。工程师仍需将人工审核和评判标准对齐纳入其工作流。We still need to see whether the results in this experiment carry over to other agents and production workflows. Additionally, low cost can amplify mistakes - a consistently wrong evaluator can produce bad feedback at scale. Engineers still need to incorporate human review and judge alignment into their workflows.

新型 System One 风格的模型可以让高质量评估变得充足且易于获取，从而加快整个智能体开发生命周期。智能体工程师可以将更多轨迹转化为反馈，更早发现回归问题，并在构建、测试、监控和部署智能体的过程中更快推进。真正的突破不仅在于降低评估成本，更在于为构建可靠的智能体提供更紧密的反馈循环。The new System One style of models could make high quality evaluation abundant. That can speed up the entire agent development lifecycle. Agent engineers can turn more traces into feedback, catch regressions sooner, and move faster as they build, test, monitor, and deploy agents. The unlock is not just cheaper evals, but a tighter feedback loop for building reliable agents.

可复现性Reproducibility

This project’s GitHub repository is available here: https://github.com/danielgshea/jev-as-a-judge

We ran the LLM judges through LangSmith Gateway: GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6. We accessed Jev through langchain-typesafe==0.0.1a2.

为确保可复现性，本次运行使用了 Deep Agents 0.7.15、LangChain OpenAI 1.6.2、LangSmith 0.12.6 和 Tavily Python 0.8.3。我们未为 LLM 评审器设置 temperature、top-p、seed 或 max tokens，因此采用了各提供商的默认值。实验元数据中未提供 Jev 服务的版本信息。For reproducibility, the run used Deep Agents 0.7.15, LangChain OpenAI 1.6.2, LangSmith 0.12.6, and Tavily Python 0.8.3. We did not set temperature, top-p, seed, or max tokens for the LLM judges, so each provider’s defaults applied. The Jev service version was not available in the experiment metadata.

想了解更多？Want to learn more?

If you want to learn more about building agents with Jev, LangChain is hosting a livestream with the TypeSafe AI team on Tuesday, Sep 22nd: https://events.langchain.com/webinar/building-a-harness-with-jev/