Scraping county election results on election night without getting blocked (or getting it wrong)

Primary season means I spend a lot of evenings refreshing county results pages. Here's how our desk does it, after some years and some embarrassing mistakes (mine).

1. Call the county first

Seriously. Many counties publish a JSON or CSV feed behind the pretty results page, and many use the same few results vendors, so once you know one feed format you know a lot of them. Ask the elections office what the official source is and how often it updates. They are almost always helpful and they would much rather you hit the feed than hammer the HTML page.

2. Poll politely

• Every 60 to 120 seconds is plenty. Results come in batches, not per second.
• Send conditional requests with If-None-Match using the ETag. A 304 costs them almost nothing.
• Identify yourself in the User-Agent with a contact email. If something goes wrong, they can tell you instead of blocking you.

3. Store raw snapshots, never overwrite

Fetching and parsing are separate jobs. The fetcher only saves exactly what the server returned, with a timestamp and a hash. If your parser has a bug at 11pm, you can re-run it on the raw data. If the county silently changes a number, you can prove it.

import hashlib, sqlite3, time
from datetime import datetime, timezone
import requests

URL = "https://results.example-county.gov/summary.json"
HEADERS = {"User-Agent": "newsroom-results-bot (data@example.org)"}

db = sqlite3.connect("snapshots.db")
db.execute("""create table if not exists snaps (
  fetched_at text, etag text, sha256 text primary key, body blob)""")

etag = None
while True:
    h = dict(HEADERS)
    if etag:
        h["If-None-Match"] = etag
    r = requests.get(URL, headers=h, timeout=20)
    if r.status_code == 200:
        etag = r.headers.get("ETag")
        digest = hashlib.sha256(r.content).hexdigest()
        db.execute("insert or ignore into snaps values (?,?,?,?)",
                   (datetime.now(timezone.utc).isoformat(), etag, digest, r.content))
        db.commit()
    elif r.status_code != 304:
        print("unexpected status", r.status_code)
    time.sleep(90)

The insert or ignore on the hash means identical bodies (some servers don't do ETags properly) are stored once.

4. Sanity checks before anything goes on the page

• A candidate's votes should not go down between snapshots. Sometimes they do, because of corrections. Flag it, never drop it, and have a human look.
• Precincts reporting can't exceed total precincts.
• Total votes in a contest can't exceed registered voters in that jurisdiction.
• Percentages should sum to roughly 100 per contest. If they don't, your parser split a write-in line wrong.

5. Never call a race from a scraper

Our scraper displays counts. People call races, using a lot more than counts: what's outstanding, where it's from, how mail ballots are processed in that state. A script that prints "WINNER" on a partial count is how you end up as a screenshot in somebody's conspiracy thread.

(If you want to practise, pick a past election in your county and replay the snapshots from the Wayback Machine. Much less stressful than election night.)