# Queue operations

This service runs on one host with a loopback operator API and a local SQLite store. Its operations report covers process loss, an older-snapshot restore, real HTTP response loss, a finite concurrent workload, and a theoretical outage longer than the retry budget.

## Start a controlled local instance

Use Python 3.12 or newer. Keep the database on local storage. For a synthetic run, start `python receiver.py`, then:

```sh
umask 077
python app.py --demo --db data/queue.sqlite --workers 3 --port 9080
```

For approved external HTTPS destinations, use the existing settings-file route from README.md with a private signing secret and `WEBHOOK_TOKEN`. The API remains on loopback. Host and Origin checks reject foreign browser origins and DNS-rebinding Host names. Request sockets have a 15-second timeout. A storage exception returns HTTP 503; retry event submission with the original event identity.

The state API does not expose signing secrets. The database does contain them. Restrict the data directory and backups accordingly. The bundled receiver and tokens are public test fixtures.

## Snapshots and restoration

```sh
python operations.py inspect --db data/queue.sqlite
mkdir -p backups
python operations.py backup --db data/queue.sqlite --out backups/queue-001
python operations.py restore --snapshot backups/queue-001 --out recovered-queue
```

The online SQLite backup API copies committed state, including WAL changes, and checks page integrity and foreign keys. A checksum manifest records row counts and project identity. Restore requires a new destination and rejects changed bytes before creating it. It does not replace an active database.

After stopping the former process, start the app with `--db recovered-queue/store.sqlite` and the same approved settings. Keep the former store until reconciliation is complete. Signing keys and destination policies must match the intended environment; changing a config file does not rotate stored endpoint secrets.

`backup_schedule.py` provides periodic snapshots. This finite command exercises it and exits:

```sh
python backup_schedule.py --db data/queue.sqlite --out backups --interval 1 --count 2
```

Omit `--count` for a continuous process and select an interval in seconds. It retains all snapshots, stops on an error, and prints a JSON record for each completed snapshot. Use a supervisor if it must continue after a terminal closes. A separately protected backup copy is required for loss of the host or disk; the local script cannot protect against losing its own storage.

## What queue restoration means for a receiver

An old snapshot can show a job as ready even though the receiver already committed it. The rehearsal snapshots a ready job, sends it, restores the old queue and sends it again. The receiver sees two requests and one business effect because it retains the same event fingerprint.

This is an executed demonstration of at-least-once delivery, not a claim of exactly-once transport. Receiver deduplication must persist beyond the longest possible queue replay and backup-retention horizon. If both queue and receiver revert independently, duplicate effects are possible. A queue snapshot cannot undo an external effect.

## Executed operational checks

Run `python operations_rehearsal.py`.

- A child process persists a lease and exits abruptly. An injected clock advances past expiry. A replacement worker reclaims the job; the former token cannot complete it.
- Four workers send 80 jobs to two real loopback HTTP endpoints. One endpoint closes every first response after its transaction commits. The run records 120 requests and exactly 80 receiver effects.
- Missing tokens, foreign Host and Origin values, and an injected storage exception produce the expected rejection or retryable response.
- An older snapshot is restored and delivered again without duplicating the receiver effect. Identical event submission remains idempotent after restore.
- Changed snapshots and occupied restore directories are rejected. A finite scheduled run produces two checked backups.

`completion.json` records expected and observed outcomes, source hashes and timings. The lease-clock advance is simulated; the SQLite child-process termination, transactions, HTTP connections and restore operations actually run.

## Outage calculation

With four attempts, a two-second request timeout, and backoff delays of 1, 2 and 4 seconds, the request-plus-backoff envelope is:

`4 * 2 + 1 + 2 + 4 = 15 seconds`

DNS resolution, worker scheduling and database lock waits are excluded from that figure. A ten-minute receiver outage exceeds the retry budget, so jobs can reach dead letters before the receiver returns. At an assumed two new jobs per second, that outage creates 1,200 new jobs requiring inspection and replay; the implementation does not promise automatic recovery from an arbitrary outage.

For a distributed deployment, an atomic event/outbox transaction and ownership-token fencing would still be necessary. A shared scheduler store would need transactional row claims, and receiver deduplication would remain necessary even with multiple workers or hosts. The executable implementation here uses SQLite serialization on one host. The distributed design is a theoretical mapping, not an unreported Postgres deployment.