Webhook delivery queue
CJ Ackley | SWE | Project brief

STATUS
Delivery design; retry policy to specify. Implementation and tests are planned.

THE PROBLEM
An application emits events faster than a downstream service can accept them. Sending directly from the request handler couples both services. A delivery queue stores the event first, then tracks each attempt and gives an operator a controlled replay path.

DESIGN QUESTION
What can the sender guarantee when the receiver accepts an event but the response is lost?

SCOPE
- Store incoming events and delivery jobs in PostgreSQL; workers claim expiring leases.
- Use a TypeScript API and worker with signed requests, bounded backoff, and per-endpoint limits.
- Provide an attempt log, paused endpoints, and replay of exhausted jobs.

CONSTRAINTS
Delivery: At least once; receivers must deduplicate stable event IDs
Destinations: Administrator-approved endpoints only
Recovery: Expired leases return interrupted jobs to the queue
Retention: Bounded payload retention; credentials excluded from logs

ENGINEERING DECISIONS (PROPOSED)
Worker leases and duplicate delivery
Commit the event and its delivery jobs together. Claim work in short database transactions and give each claim a lease token. Send outside the transaction; record the result only if the worker still owns that token.

ALTERNATIVE
A separate message broker offers more queue features. A PostgreSQL queue keeps event acceptance and job creation in one transaction for this bounded service.

TRADEOFF
Lease expiry can cause a second send while the first request is still in flight. A token prevents stale workers from overwriting queue state; it cannot undo a request already received. Receivers still need deduplication.

PROPOSED TOOLS AND METHODS
TypeScript, PostgreSQL, HTTP receiver simulator

VERIFICATION SCENARIO (NOT YET RUN)
The receiver commits, then the worker disappears
Let the receiver store an event, drop the HTTP response, and pause the worker before it records an outcome. Expire the lease, start another worker, then resume the old worker with its stale token.

PASS CONDITION
The recovered send keeps the event ID. The receiver records one business effect. The old worker cannot complete the new lease. A separate healthy endpoint continues to make progress.

EVIDENCE TO RETAIN
A fault-controlled receiver, worker lifecycle logs, delivery-attempt rows, and assertions about event IDs and lease ownership.

MEASUREMENT AND LIMITS
Sweep worker count against one healthy and one slow endpoint. Record queue age, attempts per event, and throughput. Use approved HTTPS destinations, disable redirects, and check resolved addresses at connection time to keep deliveries out of private networks.

COMPLETION CRITERIA (PLANNED)
- Event acceptance, delivery, and manual replay run from a clean checkout.
- Crash, lost-response, stale-lease, and destination-validation tests pass.
- An operator can identify and replay an exhausted delivery without editing database rows.

VERIFICATION PLAN
Lost response
Accept the event, drop the response, and verify the retry keeps its ID.

Worker crash
Kill a worker after claim; another worker recovers the expired lease.

Backpressure
A slow receiver must not consume every worker or block other endpoints.

PLANNED DELIVERABLES
- API, worker, schema, and local receiver simulator
- Crash, timeout, signature, and concurrency tests
- Replay UI and an operations guide with failure procedures

SCOPE NOTE
Delivery is at least once. Retention limits how long events can be recovered; receivers must handle duplicate IDs.

NEXT STEP
Define the event envelope, endpoint trust model, and retry budget.
