Crawler
Promises: every number here is a count. Nothing in this file asks a question.
Invariants
- Bounded three ways at once — page count, depth, and a wall-clock budget
- Sitemap URLs seed the queue when one exists, else the homepage’s internal links
- A second pass stamps inbound degree, because a page cannot know it mid-fetch
- Jitter is not decoration — eight workers backing off identically retry in lockstep
Traps
- SSRF.
assertPublicUrlrejects loopback, private and link-local hosts, so a pasted URL cannot reach an internal service. No DNS step — which is also why it is easy to test. ruleFindingscannot stream. The rule pass is site-wide and needs the finished link graph, so a per-page count at fetch time would be a lie.- Non-HTML is skipped silently. Not an error, and it does not count toward maxPages.