Webhook delivery queue

Implementation

Event acceptance and destination jobs commit together. Workers claim a lease, send outside the transaction, and finish only if their lease token is still valid. A restarted service can recover expired claims.

The sender signs each request with HMAC-SHA256, retries transport and selected HTTP failures, and retains terminal attempts for inspection. Operators can pause destinations and replay dead letters.

A local receiver fixture verifies signatures and deduplicates event IDs. One endpoint accepts an event, then closes the connection before sending its first response. The replay exercises the sender's uncertain-delivery case.

Checked online SQLite snapshots, restore into a new directory, and a periodic backup command are included. An executed operations rehearsal covers abrupt process exits, recovery and rejected inputs.

Inputs and assumptions

• Included pump-inspection events are synthetic. The three-worker demonstration uses four local endpoints and a one-request limit per endpoint.
• Python 3.12 and SQLite on one host. The earlier TypeScript/PostgreSQL distributed plan is not included.
• Delivery is at least once. Receivers must deduplicate business effects by event ID; the queue cannot infer whether a lost response followed a commit.

Recorded results

{
  "fixture": "synthetic pump inspection events; fresh local HTTP receiver",
  "events": 13,
  "jobs": 37,
  "workers": 3,
  "per_endpoint_inflight": 1,
  "counts": {
    "dead": 1,
    "delivered": 36
  },
  "attempts": 49,
  "elapsed_seconds": 1.695,
  "outcomes": {
    "delivered": 36,
    "dead": 1,
    "retry": 12
  },
  "receivers": {
    "normal": {
      "requests": 12,
      "business_effects": 12
    },
    "response-loss": {
      "requests": 24,
      "business_effects": 12
    },
    "slow": {
      "requests": 12,
      "business_effects": 12
    }
  },
  "assertions": {
    "all_jobs_terminal": true,
    "response_loss_retried": true,
    "receiver_effects_deduplicated": true,
    "permanent_rejection_dead_lettered": true
  },
  "measurement_note": "Elapsed time includes fixture shutdown; this small local scenario is not a throughput benchmark."
}

• results/summary.json
• results/attempts.csv
• results/jobs.csv
• results/receipts.json
• completion.json
• results/operations-checks.csv
• results/operations-log.txt

Verification

28 of 28 tests passed. Failures: 0. Errors: 0.

28 tests pass for claims, lease recovery, stale completions, retries, signatures, retention, destination restrictions and authorization. The executed response-loss scenario records one receiver effect for each retried event. A separate operations rehearsal passes 15 expected-outcome checks, including process crashes, checked snapshots, restore and a finite backup schedule.

python verify.py
python demo.py
python operations_rehearsal.py

Verification record

Run locally

Python 3.12+ / SQLite via Python standard library

# Terminal 1:
python receiver.py
# Terminal 2:
python app.py --demo
# Open http://127.0.0.1:9080
python verify.py
python demo.py
python build_report.py
python operations_rehearsal.py

Full setup and usage instructions

Source and evidence

• Durable queue and signed sender delivery_queue.py
• HTTP API and worker process app.py
• Response-loss receiver fixture receiver.py
• Sample event examples/event.json
• Executed HTTP attempts results/attempts.csv
• Terminal job states results/jobs.csv
• Receiver deduplication evidence results/receipts.json
• Integration test suite test_queue.py
• Executed operations and recovery report operations-report.html
• Expected and observed operational outcomes completion.json
• Reproducible crash and recovery scenarios operations_rehearsal.py
• Checked SQLite backup and restore CLI operations.py
• Periodic snapshot runner backup_schedule.py
• Deployment, recovery and capacity assumptions OPERATIONS.md

Limits

• Single-host SQLite service with local administrative bearer access. No enterprise identity, high availability, Postgres adapter or production TLS API listener.
• Retries use capped exponential backoff without jitter or Retry-After support. DNS timing follows the OS resolver. A crash before sending consumes an attempt.
• Seven-day terminal payload retention is implemented; job, attempt and audit metadata do not have automatic retention. Purged payloads cannot be replayed.
• The additional operations run sends 80 synthetic jobs through four workers and records 120 HTTP attempts with 80 receiver effects. It is a finite local workload, not an internet capacity or physical power-loss qualification.