GitHub - RayenMlayeh/jevcrawl：专为 AI 代理打造的网站抓取解决方案——比 Crawl4AI 更高效，比 Claude 抓取便宜 10 倍。

为 AI 智能体打造的网络爬虫，寻找答案，而不只是网页。The web crawler for AI agents that finds the answer, not just pages.

一个小型决策模型会挑选每个链接和每段内容，因此你的智能体的 LLM 每次抓取只需花费几美分，就能读取回答其问题所需的信息。A small decision model picks every link and every passage, so your agent's LLM reads only what answers its question at a few cents per crawl.

快速入门 · 基准测试 · 工具 · 工作原理 · 配置Quick start · Benchmarks · Tools · How it works · Configuration

浏览网站的智能体面临两个糟糕的选择：要么读取爬虫抓取的所有内容，答案就会被导航信息和模板文本淹没，还会消耗大量 token；要么让 LLM 挑选每个链接，速度慢，而且每个问题都要花费数美元。jevcrawl 两者都不做：Agents that browse a site have two bad options. They can read everything a crawler fetches, which buries the answer in navigation and boilerplate and burns tokens. Or they can let the LLM pick each link, which is slow and costs dollars per question. jevcrawl does neither:

• 🎯 更常找到答案页面：在 Claude 选择链接时，54% 的任务能找到答案页面，而 Crawl4AI 的目标导向爬取找到答案页面的频率只有它的一半。
  🎯 Finds the answer page more often. 54% of tasks vs 33% when Claude picks the links, and 2x as often as Crawl4AI's goal-directed crawl.
• 💸 每项任务的成本便宜约 8 倍，每个正确答案的成本便宜约 10 倍。爬取决策的成本约为 1 美分。
  💸 About 8x cheaper per task, 10x per correct answer. The crawl decisions cost about a cent.
• ⚡ 速度是由 LLM 驱动的爬取的 2 倍。每次处理四个页面，每个页面发送一个决策请求。
  ⚡ 2x faster than an LLM-driven crawl. Four pages at a time, one decision request per page.
• 📜 逐字引用，并注明来源。每段内容均复制自原页面，并附有 URL、标题路径和字符偏移量。决策模型只负责评分，因此无法凭空编造文本。
  📜 Verbatim, with sources. Every passage is copied from the page with its URL, heading path and character offsets. The decision model only scores, so it cannot invent text.
• 🛡️ 从设计上防范提示注入。页面文本只会被评分，绝不会被遵循。
  🛡️ Safe against prompt injection by design. Page text is only ever scored, never followed.
• 🔌 在 MCP 支持的所有环境中均可使用：Claude Code、Claude Desktop、Cursor 或 CLI。
  🔌 Works everywhere MCP does: Claude Code, Claude Desktop, Cursor, or the CLI.

你需要从 console.typesafe.ai 获取 Jev API 密钥，并安装 uv。You need a Jev API key from console.typesafe.ai and uv.

Claude CodeClaude Code

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx jevcrawl

Claude Desktop、Cursor 以及其他 MCP 客户端（客户端配置文件中的 mcpServers）Claude Desktop, Cursor and other MCP clients (mcpServers in the client's config file)

{
  "mcpServers": {
    "jevcrawl": {
      "command": "uvx",
      "args": ["jevcrawl"],
      "env": { "TYPESAFE_API_KEY": "<your key>" }
    }
  }
}

然后可以这样询问你的智能体：“从 docs.python.org 开始，使用 jevcrawl 查找哪个 Python 版本新增了海象运算符。”使用 uvx jevcrawl doctor 检查设置。能够加载技能的智能体也可以使用 skills/jevcrawl/SKILL.md。Then ask your agent something like "Use jevcrawl to find which Python version added the walrus operator, starting from docs.python.org". Check the setup with uvx jevcrawl doctor. Agents that load skills can also use skills/jevcrawl/SKILL.md.

每个系统都从网站首页开始，拥有相同的 30 页抓取额度，并面对同一个问题。每个系统收集到的内容都会交给同一个 LLM 作答，答案也由同一个评审打分，因此唯一变化的就是爬虫。Every system starts on a site's home page with the same 30-page budget and the same question. What each one collects goes to the same answering LLM, and the same judge grades the answer, so the crawler is the only thing that changes.

Crawl4AI 是最受欢迎的开源 LLM 爬虫之一。它的 BestFirstCrawlingStrategy 配合关键词评分器，可以在不使用 LLM 的情况下，朝着问题相关的方向爬取。于是，我们对比了两者，看它们能否抓取到包含答案的页面。Crawl4AI is one of the most popular open-source crawlers for LLMs. Its BestFirstCrawlingStrategy with a keyword scorer is its way to crawl toward a question without an LLM, so we compared the two on whether they fetch the page that holds the answer.

基准测试代码及复现所有数字的方法：bench/。The benchmark code and how to reproduce every number: bench/.

{"blocks": [{"url": "https://docs.python.org/3/whatsnew/3.8.html",
             "heading_path": "What’s New In Python 3.8 > New Features > Assignment expressions",
             "text": "There is new syntax := that assigns values to variables as part of a larger expression. ...",
             "char_start": 307, "char_end": 580, "relevance": 0.97}],
 "uncertain": [],
 "pages": [{"url": "https://docs.python.org/3/", "title": "3.15.0 Documentation", "lang": "en",
            "text_sha256": "b624ce93..."}],
 "stats": {"pages_fetched": 6, "tokens_raw": 80034, "tokens_returned": 1873, "jev_calls": 11,
           "cache_hits": 0, "jev_input_tokens": 164139, "jev_cost_usd": 0.0069,
           "model": "jev-1.13.0", "degraded": false, "degraded_reason": null,
           "needs_js_urls": [], "stopped_because": "answered", "elapsed_seconds": 8.3}}

精简为一个区块和一页。页面内容的 80,034 个 token 最终变成了 1,873 个回答 token，花费 0.0069 美元，耗时 8.3 秒。Trimmed to one block and one page. 80,034 tokens of pages became 1,873 tokens of answer, for $0.0069, in 8.3 seconds.

• blocks 是相关段落，按相关性从高到低排列；uncertain 是边缘相关的段落，值得在断定某些内容缺失之前先阅读。
  blocks are the relevant passages, best first; uncertain are borderline ones worth reading before concluding something is missing.
• char_start/char_end 是页面文本中的字符索引，其 sha256 哈希值记录在 pages 中，因此之后可以核验引文。
  char_start/char_end index into the page text whose sha256 is in pages, so a quote can be checked later.
• 停止原因可能是已回答、前沿为空、达到最大页数、达到最大秒数、超出预算或身份验证错误。
  stopped_because is answered, frontier_empty, max_pages, max_seconds, budget or auth_error.
• 预期的失败会以 `{"error": "<code>", "message": "<what to do>"}` 的形式返回；工具不会在代理的会话中抛出异常。
  Expected failures come back as {"error": "<code>", "message": "<what to do>"}; the tools never raise into the agent's session.

通过命令行也可以执行相同的操作（uv tool install jevcrawl 或 pip install jevcrawl 会将 jevcrawl 添加到你的 PATH 中）：The same operations work from the command line (uv tool install jevcrawl or pip install jevcrawl puts jevcrawl on your path):

jevcrawl crawl https://docs.python.org/3/ --goal "What does the walrus operator do?" [--json]
jevcrawl scrape https://example.com/help/returns [--goal "..."] [--json]
jevcrawl map https://example.com --goal "..." [--limit 20]
jevcrawl doctor

• Jev负责做出判断，而不是写作。对于每个是/否问题，Jev都会返回一个概率，因此无法生成文本。你得到的所有内容都是从页面复制的。
  Jev decides, it does not write. Jev returns a probability per yes/no question, so it cannot produce text. Everything you get back is copied from the page.
• 页面文本是数据，而非指令。它只会通过问题的数据字段传递；每个问题句子都是由 jevcrawl 编写的固定字符串。注入的“忽略先前指令”最多只能影响评分。
  Page text is data, never instructions. It travels only in data fields of a question; every question sentence is a fixed string written by jevcrawl. An injected "ignore previous instructions" can at worst bias a score.
• 优先级低于下限的链接会被保留，而不是丢弃，因此只要爬取仍有页面预算，就不会提前停止。
  Links below the priority floor are kept in reserve, not dropped, so a crawl never stops early while it still has page budget.
• 失败会导致降级，而不是被隐藏。Jev 请求连续失败三次后，爬取会继续按广度优先方式进行，并将所有区块作为未评分项返回，标记为 uncertain，同时设置 stats.degraded。
  Failures degrade, they do not hide. After three failed Jev requests the crawl continues breadth-first, returns every block unscored in uncertain, and sets stats.degraded.
• 重复运行免费。Jev 的答案会缓存在本地（~/.jevcrawl/cache），因此在相同页面上实现相同目标无需任何费用。
  Repeats are free. Jev's answers are cached locally (~/.jevcrawl/cache), so the same goal on the same pages costs nothing.

其他所有配置（阈值、每页链接数、并发数）在 jevcrawl/config.py 中都有经过实测的默认值，Python 调用方也可以传入自己的 Config。Everything else (thresholds, links per page, concurrency) has a measured default in jevcrawl/config.py, and Python callers can pass their own Config.

Jev 仅对输入 token 收费，jev-1.13.0 的价格为每百万 token 0.042 美元；在基准测试中，平均每次爬取花费 0.011 美元。max_decision_usd 是硬性上限：任何可能使其超限的请求都不会发送，爬取会在预算用尽时停止。每个结果都会报告 jev_cost_usd，以及 tokens_raw（普通爬取会交给你的 LLM 的 token 数）和 tokens_returned。Jev charges for input tokens only, $0.042 per million for jev-1.13.0; in the benchmark a crawl cost $0.011 on average. max_decision_usd is a hard limit: a request that could pass it is never sent, and the crawl stops with budget. Every result reports jev_cost_usd, and tokens_raw (what a plain crawl would hand your LLM) against tokens_returned.

页面文本会发送至 TypeSafe（包括区块、链接标签和 URL）进行评分。API 密钥从环境变量中读取，绝不会写入磁盘或日志。Page text is sent to TypeSafe (blocks, link labels and URLs) to be scored. The API key is read from the environment and never written to disk or logs.

jevcrawl 不会读取 robots.txt，除了每次抓取四个页面外，也不会自行限制速率。你有责任确保抓取的网站、其服务条款以及适用于你的法律得到遵守。它绝不会登录，也不会尝试绕过付费墙或 CAPTCHA，并且不会运行 JavaScript（需要 JavaScript 的页面会列在 stats.needs_js_urls 中）。每次调用都会受到 max_pages、max_seconds、max_decision_usd 和 return_tokens 的限制。jevcrawl does not read robots.txt and does not rate-limit itself beyond fetching four pages at a time. You are responsible for the sites you crawl, their terms of service and the law that applies to you. It never logs in, does not try to get past paywalls or CAPTCHAs, and does not run JavaScript (pages that need it are listed in stats.needs_js_urls). Every call is bounded by max_pages, max_seconds, max_decision_usd and return_tokens.

uv sync
uv run ruff check . && uv run ruff format --check . && uv run mypy --strict jevcrawl
uv run pytest                                    # offline: all HTTP is mocked
TYPESAFE_API_KEY=... uv run pytest -m live -s    # one tiny real request, a fraction of a cent

测试绝不会访问网络或产生费用；凡是会这样做的都标记为 live。README 中的图表由 assets/build.py 根据测量数据绘制。欢迎提交问题和拉取请求；如需报告安全问题，请按照 SECURITY.md 中的说明操作。Tests never touch the network or spend money; anything that does is marked live. The README charts are drawn by assets/build.py from the measured numbers. Issues and pull requests are welcome; please report security problems as described in SECURITY.md.

• 阿里巴巴通义实验室推出的 WebWalkerQA，是一个针对确实需要网页爬取的问题的基准测试。
  WebWalkerQA by Alibaba's Tongyi Lab, for a benchmark of questions that really need crawling.
• Crawl4AI，我们用来进行对比的基准。
  Crawl4AI, the baseline we measure against.
• 模型上下文协议、selectolax 和 httpx。
  Model Context Protocol, selectolax and httpx.

Apache-2.0Apache-2.0