
On this page
Correction — November 11, 2026: The source counts in this post were wrong. Terrabillion tracks 28 sources, as listed on the coverage page — not roughly 500. The crawling approach described below is unchanged.
Every package in the queue starts as a fetched page: a county purchasing site, a state procurement portal, a plan-holder list. Across roughly 500 sources checked daily, the naive approach — fetch everything, constantly — would get us blocked within a week and slow those sites for everyone else. Here is the polite crawling system that keeps the queue fresh without wearing out its welcome.
Every source gets a schedule
Not all bid boards move at the same speed. Ohio's ODNR posts in bursts around bid cycles; small counties update quarterly if that; Kentucky's spring KASTOW wave meant weekly checks on sources we normally visit monthly. Each source carries a check cadence derived from its observed update rhythm: daily for the fast boards, weekly for the steady ones, monthly for the sleepy ones, with automatic promotion when a quiet source suddenly wakes up.
Scheduling matters because waste compounds. Fetching a quarterly county daily burns 90 requests per quarter to catch one update — requests that cost the county bandwidth and cost us nothing but goodwill until the day they cost us access. The scheduler's job is to spend our request budget where the updates are.
Conditional requests do the heavy lifting
Most fetches should transfer nothing. We send conditional requests everywhere — If-Modified-Since and If-None-Match — so unchanged pages answer 304 Not Modified with no body. The large majority of our daily checks resolve this way: a few hundred bytes confirming nothing changed, at negligible cost to either side.
When a page does change, we fetch the full body and diff it against the last seen version before doing anything expensive. Changed boilerplate — a new footer, a rotated banner — gets logged and ignored. Changed listings enter the extraction pipeline. The diff step is what keeps a county webmaster's redesign from becoming fifty phantom new packages.
Backoff is a feature, not an apology
Every source has a failure budget. Timeouts, 500s, and rate-limit responses trigger exponential backoff with jitter: wait, retry, wait longer, retry once more, then stand down and alert a human if the source stays dark. We never hammer a struggling server — a county site returning 500s is having a bad day, and our crawler piling on would make it worse while learning nothing.
Backoff discipline also covers us legally and socially. We respect robots.txt on every source, identify ourselves with a contactable user agent, and keep per-site request rates far below anything a human browsing session would generate. Several county IT departments have emailed us over the years; the conversations go well because the logs show a well-behaved guest. One office even asked for our check schedule so they could plan maintenance around it.
Catching up after downtime
Sources go dark — maintenance windows, expired certificates, the occasional ransomware incident at a county vendor. When a source returns, the pipeline does not just resume; it reconciles. The first successful fetch after an outage gets compared against the full last-known state, and anything posted during the gap enters with its original posting date preserved, flagged as late-caught. Bidders see the true timeline, not our fetching timeline.
Late-caught packages matter disproportionately because they carry the shortest remaining fuses. A package posted Tuesday on a site that was down until Friday has three days less bid time than its calendar suggests. The queue surfaces the catch-up lag explicitly so estimators can triage accordingly.
When polite crawling meets hostile design
Not every source wants to be watched. Some bid boards hide behind search forms with no listable index, CAPTCHAs on every query, or session tokens that expire mid-crawl. Our policy: work with the site's intent, not against it. Search-only boards get queried with our standing keyword set on their cadence. CAPTCHA walls get an email to the webmaster explaining what we do and asking for an accommodation — a surprising number answer with an RSS feed, an email notification list, or at minimum a stable search URL. Session tricks get reported as bugs, because from the accessibility standpoint they usually are.
What we never do is escalate: no CAPTCHA-solving services, no rotating residential proxies to dodge rate limits, no scraping at speeds designed to slip under detection. Circumvention burns the relationship permanently for a temporary data gain, and it teaches every other source to distrust us by association. A source we cannot fetch politely goes on the known-gaps list, labeled publicly, until the office works with us. Our coverage methodology counts those gaps against us honestly — which is exactly the incentive structure that keeps us polite.
What we think at Terrabillion
Crawling is a relationship, not a technique. Every one of our 500 sources is run by people — often one person, often part-time — and our pipeline treats their bandwidth as a shared resource rather than a free good. Polite crawling is slower to build and cheaper to keep: we have never been permanently blocked by a source we monitor, and we intend to keep that record. The queue is only as fresh as its least cooperative source relationship, so we invest in the relationships. Fetch less, catch more, apologize never.
More blog posts to read

Geocoding 40,000 well records without losing our minds
County plats, section-township-range grids, and hand-typed latitudes all disagree. Here is the pipeline that reconciles them into one map.

Making the bid queue fast with virtualized lists
Five thousand rows brought the queue to its knees. Rendering only the visible ones took us from 4fps back to 60.
Orphan-well intel, monthly
Contact sales