Webhook delivery queue

Implementation

Event acceptance and destination jobs commit together. Workers claim a lease, send outside the transaction, and finish only if their lease token is still valid. A restarted service can recover expired claims.

The sender signs each request with HMAC-SHA256, retries transport and selected HTTP failures, and retains terminal attempts for inspection. Operators can pause destinations and replay dead letters.

A local receiver fixture verifies signatures and deduplicates event IDs. One endpoint accepts an event, then closes the connection before sending its first response. The replay exercises the sender's uncertain-delivery case.

Checked online SQLite snapshots, restore into a new directory, and a periodic backup command are included. An executed operations rehearsal covers abrupt process exits, recovery and rejected inputs.

Inputs and assumptions

• Included pump-inspection events are synthetic. The three-worker demonstration uses four local endpoints and a one-request limit per endpoint.
• Python 3.12 and SQLite on one host. The earlier TypeScript/PostgreSQL distributed plan is not included.
• Delivery is at least once. Receivers must deduplicate business effects by event ID; the queue cannot infer whether a lost response followed a commit.

Results

Report figures (2)

Open the full report ↗

Verification

28 tests pass for claims, lease recovery, stale completions, retries, signatures, retention, destination restrictions and authorization. The executed response-loss scenario records one receiver effect for each retried event. A separate operations rehearsal passes 15 expected-outcome checks, including process crashes, checked snapshots, restore and a finite backup schedule.

python verify.py
python demo.py
python operations_rehearsal.py

Project files

The ZIP contains the source, inputs, results, and instructions. These files are also available individually.

• Durable queue and signed senderdelivery_queue.py · 17 KB
• HTTP API and worker processapp.py · 7 KB
• Response-loss receiver fixturereceiver.py · 4 KB
• Sample eventexamples/event.json · 1 KB
• Executed HTTP attemptsresults/attempts.csv · 3 KB
• Terminal job statesresults/jobs.csv · 2 KB
• Receiver deduplication evidenceresults/receipts.json · 3 KB
• Integration test suitetest_queue.py · 11 KB
• Executed operations and recovery reportoperations-report.html · 9 KB
• Expected and observed operational outcomescompletion.json · 8 KB
• Reproducible crash and recovery scenariosoperations_rehearsal.py · 11 KB
• Checked SQLite backup and restore CLIoperations.py · 5 KB
• Periodic snapshot runnerbackup_schedule.py · 1 KB
• Deployment, recovery and capacity assumptionsOPERATIONS.md · 5 KB

Run locally

Python 3.12+ / SQLite via Python standard library

Extract the ZIP and open a terminal in the webhook-delivery-queue folder. The README includes setup and usage details.

# Terminal 1:
python receiver.py
# Terminal 2:
python app.py --demo
# Open http://127.0.0.1:9080
python verify.py
python demo.py
python build_report.py
python operations_rehearsal.py

Limits

• Single-host SQLite service with local administrative bearer access. No enterprise identity, high availability, Postgres adapter or production TLS API listener.
• Retries use capped exponential backoff without jitter or Retry-After support. DNS timing follows the OS resolver. A crash before sending consumes an attempt.
• Seven-day terminal payload retention is implemented; job, attempt and audit metadata do not have automatic retention. Purged payloads cannot be replayed.
• The additional operations run sends 80 synthetic jobs through four workers and records 120 HTTP attempts with 80 receiver effects. It is a finite local workload, not an internet capacity or physical power-loss qualification.