Skip to content

Bounded site research

occam research discovers links (occam_map) then extracts pages (occam_transcode) under URL, page, time, and byte budgets. Discovery and extraction are separate reports. It is not a new MCP tool and not a second crawler.

This recorded folder was assembled with --from-json from the compare-sources golden excerpts (host toolchain ff-occam/1.0.0-rc.2). That is not a GitHub Release v1.0.0 certification.

Folder Seed Stop
nginx-proxy https://nginx.org/en/docs/ budget_pages after two successful extracts; off-site URL named in discovery.outOfScope

Replay:

occam research --from-json docs/examples/site-research/_inputs/nginx-proxy.json \
  --out tmp/research --max-pages 2 --max-urls 8

Live (spawns the local MCP host):

occam research --seed https://nginx.org/en/docs/ \
  --focus "proxy_pass proxy_read_timeout" \
  --max-pages 4 --max-urls 16 --deadline-ms 60000 --max-bytes 1000000 \
  --out tmp/research

--resume continues the same --out folder and restores per-page markdown from pages.json (or excerpts.txt on older folders). Two consecutive resumes keep every previously successful page. A discovery failure during resume updates stop.reason and does not erase the prior report.

--max-bytes counts UTF-8 bytes of kept extracted markdown (output), not HTTP download size. A page that would exceed the remaining allowance is listed under extraction.omitted and is not stored. Remaining --deadline-ms bounds each MCP call. Ctrl+C records stop.reason=cancelled and keeps the last checkpoint.

If a page fails, the folder is still written; do not fill gaps from memory.

Ranking used to order discovered URLs is the A1 spike — it does not change occam_search default order. See capability evaluation.