# Webhook delivery queue

A working local delivery service with an operator console, durable jobs, signed HTTP requests, lease recovery, endpoint pause controls, and dead-letter replay. Includes a receiver that deliberately loses a response after accepting an event.

This implementation uses Python and SQLite on one host. The earlier project brief proposed TypeScript/PostgreSQL; that distributed implementation is not included. The transaction, lease, retry, and duplicate-delivery behavior here is executable and tested.

## Run the local demo

Python 3.12 or newer. No packages to install. Open two terminals in this directory.

```sh
python receiver.py
```

```sh
python app.py --demo
```

Open **http://127.0.0.1:9080/**. Enter `demo-admin-token`. The receiver runs at http://127.0.0.1:9081/receipts. Both bind only to loopback.

Submit an event from the console or a third terminal:

```sh
curl -sS http://127.0.0.1:9080/api/events \
  -H 'Authorization: Bearer demo-admin-token' \
  -H 'Content-Type: application/json' \
  --data-binary @examples/event.json
```

The `normal` destination returns 200. `response-loss` saves the event and closes its first response; the retry has the same event ID and the receiver records one business effect. `rejected` returns 422 and goes directly to dead letters. Use the endpoint controls to pause new claims; an already-running request is allowed to finish. Replay creates another delivery generation and preserves the attempt history.

Stop each process with Ctrl+C. Restarting preserves jobs in `queue.sqlite` and receiver deduplication in `receiver.sqlite`. To reset, stop both processes and remove those two databases and their matching `-wal`/`-shm` files. The QA and evidence scripts use temporary databases instead.

## Verify and reproduce evidence

```sh
python verify.py
python demo.py
python build_report.py
```

`verify.py` executes the integration suite and writes `verification.json` plus `results/test-log.txt`. `demo.py` sends 13 synthetic pump-inspection events to four local endpoints with three workers. The checked run produced 37 jobs, 36 deliveries, one permanent rejection, and 49 attempts. The response-loss endpoint received 24 requests for 12 business effects. These are a small reproducible failure scenario, not a throughput benchmark.

Open `report.html` directly. Raw evidence is in `results/summary.json`, `results/attempts.csv`, `results/jobs.csv`, and `results/receipts.json`.

## Delivery contract

- Accepting an event and creating all destination jobs happen in one `BEGIN IMMEDIATE` transaction. The event ID is unique. Repeating identical canonical JSON and destination IDs returns the original acceptance; conflicting content returns 409. Payloads are limited to 64 KiB and fan-out to 32 destination IDs.
- Claiming creates a random ownership token and a lease. HTTP runs after the transaction commits. Completion requires the same token and an unexpired lease. A stale worker cannot complete another worker's claim.
- Lease expiry consumes an attempt, including a crash before sending. A later claim recovers the job. This bounds crash loops but can exhaust a delivery that never reached its receiver; the operator can review and replay it.
- 2xx succeeds. Transport failure, 408, 425, 429, and 5xx retry. Other statuses, including redirects, are terminal. Backoff is `min(300, retry_base * 2^(attempt-1))` seconds. `Retry-After` and jitter are not implemented. Per-endpoint limits prevent one endpoint from occupying every worker.
- Delivery is **at least once**. If the receiver commits and its response is lost, the sender cannot tell whether it committed. The receiver must deduplicate business effects by event ID. A lease cannot undo a request already on the wire.
- Requests carry `X-Event-ID`, `X-Webhook-Timestamp`, and `X-Webhook-Signature`. The signature is `v1=` followed by HMAC-SHA256 over the ASCII timestamp, a dot, and the raw request body. The fixture rejects timestamps more than five minutes from its clock. Clocks must be synchronized.
- Terminal payloads older than seven days are cleared at startup and every minute. `--retention-days N` changes that cutoff. Pending payloads are retained. Event fingerprints remain, so an old ID is still reserved; a purged job cannot be replayed. Job/attempt/audit metadata currently has no automatic retention policy.

## Approved external destinations

Do not expose the demo tokens or receiver. For a controlled single-host trial, copy `examples/settings.json` to a private config file, replace the example host and signing secret, and run:

```sh
python -c "import secrets; print(secrets.token_urlsafe(32))"
# Set WEBHOOK_TOKEN in your shell to the generated value.
python app.py --settings private-settings.json --db private-queue.sqlite
```

Only exact administrator-approved hostnames over HTTPS port 443 are accepted. Every attempt resolves and rejects non-public IP addresses, mixed public/private answers, IPv4-mapped IPv6, 6to4, and Teredo addresses. The connection uses the checked IP directly while TLS verifies the original hostname. Redirects are not followed. Connection/read timeouts and a socket deadline bound HTTP work after DNS resolution; DNS itself follows OS resolver timeout behavior.

The local fixture exception is available only in `--demo`, for the exact `http://127.0.0.1:<receiver-port>` origin. Ordinary configurations cannot use that exception. Existing endpoint settings are loaded once when an endpoint is first created; editing a config does not rotate a stored secret. Use a fresh database for a changed configuration in this local implementation.

The API has no endpoint-registration route. It accepts event data, not arbitrary delivery URLs. Operator routes require one bearer token; they provide local administrative access rather than multi-user role management. The database stores signing secrets in plaintext: restrict access to its directory and configuration. No token or secret is included in the state API or operational error logs.

## API and files

| Route | Operation |
| --- | --- |
| `GET /api/state` | Counts, endpoint configuration without secrets, newest 250 jobs and 500 attempts |
| `POST /api/events` | `{id,type,data,endpoints}` |
| `POST /api/pause` | `{endpoint,paused}` |
| `POST /api/replay` | `{job}` for a dead-letter job |

`delivery_queue.py` owns the durable state and sender. `app.py` runs the API and workers. `receiver.py` is the fault fixture. `static/` contains the dark console. The test suite covers competing claims, restart recovery, expired owners, exhausted retries, pause controls, failed response delivery, signature checks, retention, URL policy, and authorization.

## Deployment limits

SQLite permits one writer at a time; the implementation serializes short writes across worker connections. Use a local disk, not a network filesystem. This package has no high-availability scheduler, Postgres row-lock adapter, TLS API listener, enterprise identity, key rotation workflow, load-test certification, or multi-host failover. It is suitable for inspecting and running the local implementation; those deployment concerns need separate engineering before an internet-facing service.

Technical references: [SQLite isolation](https://www.sqlite.org/isolation.html), [SQLite WAL](https://www.sqlite.org/wal.html), [Python ipaddress](https://docs.python.org/3/library/ipaddress.html), [Python HTTP client](https://docs.python.org/3.12/library/http.client.html).

## Completed operations rehearsal

The operations run sends 80 jobs through four workers, records 120 signed HTTP attempts with 80 receiver effects, recovers a crashed worker lease, and restores an older snapshot after delivery. The second delivery produces no additional receiver effect.

```sh
python operations_rehearsal.py
```

Read `operations-report.html` for expected and observed outcomes, or `completion.json` for source hashes and machine-readable checks. `OPERATIONS.md` contains local startup, online snapshot, restore, periodic backup and explicit recovery assumptions. `operations.py` and `backup_schedule.py` are runnable tools, not pseudocode. The rehearsal uses temporary data and stops all of its services before exiting.