Skip to content

Chapter 9 — Spending less: discovery before acquisition

Part C — Breadth · Prerequisites: Ch 2, Ch 5 · Next: Chapter 10

Mental model

Three signals that are not one scale.

  • extractability (probe, search rerank) — prediction before full fetch
  • confidence / quality — measurement after fetch
  • Playbook verify.score — authoring gate heuristic (0–100)

Never compare them as interchangeable scores. Discovery tools cheaply answer whether and which URL before paying extraction cost.

Explanation

occam_probe

  • HTTP-only classification: redirects, SPA hints, robots, tier hints, failure prediction.
  • Never escalates to browser—browser-only pages can be mis-predicted.
  • May attach agentHints for next steps.
  • SSRF blocks may appear as network_error—misleading for private URLs.
  • Output is unsigned—not verifiable extract.

occam_map

  • Sources: sitemap, robots, homepage links, bounded hub expansion.
  • Hard cap: 64 links in links[].
  • HTTP-only—JS navigation menus may yield empty/sparse maps.
  • Listing, not crawling—no frontier expansion beyond second level.
  • Registered core tool. Unset OCCAM_SEARCH_PROVIDER uses keyless DuckDuckGo HTML (provider disclosed). off / none fails closed as search_unconfigured.
  • Proxies a named provider (DuckDuckGo default; SearXNG/Brave/Tavily/Donsetch override)—not Occam's index.
  • Optional rerank can fire many live probes (up to ~20) ordering by extractability, not relevance.
  • Results are pointers + hints, not page content.

Task R step 6

  1. Probe docs index—cheap extractability read.
  2. Map with source:"sitemap" to enumerate reference URLs under cap.
  3. If site unknown, search (DuckDuckGo default)—then transcode chosen URLs.

Workflow: discover → acquire (Chapter 5), not discover instead of honesty (Chapter 2).

CHECK

LOCAL / NETWORK

  1. occam_search with no provider configured → must fail closed (typed error), not empty success.
  2. occam_map on a large site → assert links.length ≤ 64.

Common misconception

"Probe tells me whether the page will extract."

Probe predicts from HTTP-only signals. It can be wrong on browser-rendered content. When stakes are high, transcode—or probe plus conservative backend_policy.

Limitations

  • Nothing in discovery PS-3 is signed or hashed.
  • Map/probe never use session storageState tier-1 browser state (Chapter 6).
  • Search relevance is the provider's job; Occam only reranks by extractability.
  • probe.autoRedirect may be registered but not selected—do not rely on it.

Public docs: Search & discover · Choosing a tool · Tools: occam_probe · Tools: occam_map · Tools: occam_search

Next chapter: Chapter 10 — Many sources in one call: digest