
On this page
The same plugging package routinely appears in five places: the county purchasing page, the state procurement portal, a plan-holder list, and two aggregators scraping all of the above. Each copy disagrees slightly — different due times, truncated scopes, stale addenda. Without deduping, the queue would show five listings for one job and shred bidder trust within a week. Here is how the merge pipeline works.
The fingerprint
Every ingested package gets a fingerprint computed from the fields that survive reposting: normalized county and state, well count, approximate centroid, bid due date, and a hash of the scope text's rarest phrases. Posting boilerplate differs across sites, but "Johnny Appleseed 3" and "thirteen wells in Amanda Township" do not. Packages whose fingerprints match above threshold become merge candidates; packages that merely share a county do not.
Fingerprints are deliberately conservative. A false merge — two genuinely different packages fused into one — is far worse than a missed merge, because it deletes a real opportunity from someone's queue. The pipeline would rather show two suspiciously similar listings with a "possible duplicate" flag than fuse them wrongly. Reviewers clear the flags; the flags have never once been the thing a user complained about.
The merge record
When candidates merge, nothing is thrown away. The surviving record keeps every source attached with its own fetch timestamp: the county PDF, the state portal entry, the aggregator copies. Conflicting fields resolve by source rank — the issuing office beats the portal, the portal beats the aggregator — and every overridden value stays visible in the record's history with its provenance.
This matters most for due dates, the most commonly conflicted field. A county amends its due date on its own site; the state portal updates three days later; the aggregator never does. Bidders looking at any single source see one date with full confidence. Bidders looking at the merged record see the current date, the superseded date, and which document changed it. That is the difference between information and a rumor with a URL.
Addenda across sources
Deduping interacts with addenda versioning in the way you would hope: an addendum found on any source attaches to the merged record and propagates to watchers once, not once per source. The pipeline tracks which sources have published which amendments, so the record can also say "amendment 2 seen on county site, not yet on state portal" — a genuinely useful signal during the final week before bids are due, when bidders are checking every source nightly.
Measuring what we catch
We sample merged and unmerged records weekly and hand-verify a fixed count. Current numbers: roughly one package in six arrives from multiple sources, the pipeline merges the large majority automatically, and the reviewer queue clears the flagged remainder within a day. Missed merges surface through user reports — "this package appears twice" — and each report becomes a fingerprint regression case. Duplicate reports per thousand listings fall almost every month, which is the metric that tells us the conservatism is calibrated correctly. The merge candidates themselves arrive via polite board monitoring — and the merged records render through a virtualized list built for exactly this volume.
What we think at Terrabillion
Deduping is where a data product earns or loses the right to summarize. Anyone can scrape bid boards; the queue's job is to present one job once, with every source attached and every conflict visible. That principle runs through everything we build — confidence bands on fields, uncertainty radii on map dots, versioned addenda. Bidders make five-figure decisions from this screen. Showing our work is not a feature; it is the license to operate.
More blog posts to read

Geocoding 40,000 well records without losing our minds
County plats, section-township-range grids, and hand-typed latitudes all disagree. Here is the pipeline that reconciles them into one map.

Making the bid queue fast with virtualized lists
Five thousand rows brought the queue to its knees. Rendering only the visible ones took us from 4fps back to 60.
Orphan-well intel, monthly
Contact sales