Executed synthetic operations
Webhook queue operations rehearsal
These checks use generated records, disposable local databases and deliberate faults. A passing check means the stated expected outcome occurred, including expected rejection or rollback. It does not certify an internet-facing deployment or hardware power-loss behavior.
Results
| Case | Requirement | Expected | Observed | Result |
|---|---|---|---|---|
| crash.reclaim | Claim survives abrupt exit and a new worker replaces the expired owner | {"exit": 86, "same_job": true, "new_token": true, "attempt": 2} | {"exit": 86, "same_job": true, "new_token": true, "attempt": 2} | PASS |
| crash.fence | Old worker completion rejected; replacement completion accepted | [false, true] | [false, true] | PASS |
| http.delivery | Four workers deliver all 80 accepted jobs | {"delivered": 80} | {"delivered": 80} | PASS |
| http.deduplication | 120 requests cause exactly 80 receiver business effects | {"requests": 120, "effects": 80} | {"requests": 120, "effects": 80} | PASS |
| http.attempts | Attempt history contains the expected first failures and successful retries | {"retry": 40, "delivered": 80} | {"retry": 40, "delivered": 80} | PASS |
| api.authorization | Missing administrator token is rejected | 401 | 401 | PASS |
| api.boundary | Foreign Host and Origin requests are rejected | [403, 403] | [403, 403] | PASS |
| api.storage | Storage failure returns retryable HTTP 503 without accepting an event | 503 | 503 | PASS |
| restore.at-least-once | Restoring an older queue snapshot redelivers without duplicate receiver effect | {"requests": 2, "effects": 1, "status": "delivered"} | {"requests": 2, "effects": 1, "status": "delivered"} | PASS |
| restore.identity | Restored acceptance still recognizes the same event identity | true | true | PASS |
| restore.corruption | Corrupt snapshot rejected before destination creation | true | true | PASS |
| restore.no-overwrite | Existing restore directory remains untouched | true | true | PASS |
| integrity.final | Queue integrity and foreign-key checks pass | {"integrity": "ok", "foreign_key_errors": 0} | {"integrity": "ok", "foreign_key_errors": 0} | PASS |
| theory.retry-budget | Declared four-attempt timeout/backoff calculation totals 15 seconds | 15 | 15 | PASS |
| backup.schedule | Finite scheduler creates two checked snapshots and stops | {"exit": 0, "snapshots": 2, "valid": 2} | {"exit": 0, "snapshots": 2, "valid": 2} | PASS |
Measurements
{
"http_workload": {
"jobs": 80,
"workers": 4,
"http_attempts": 120,
"elapsed_seconds": 0.4144,
"delivered_jobs_per_second": 193.07,
"qualification": "A finite local synthetic workload, not internet latency or sustained capacity."
},
"recovery": {
"snapshot_bytes": 45056,
"snapshot_seconds": 0.003724,
"restore_seconds": 0.002835,
"queue_backup_rpo_seconds": 300,
"receiver_requirement": "Receiver deduplication records must outlive the longest queue replay or backup retention horizon."
},
"theoretical_outage_envelope": {
"attempts": 4,
"per_attempt_seconds": 2,
"retry_delays_seconds": [
1,
2,
4
],
"elapsed_if_every_request_uses_timeout_seconds": 15,
"excluded_from_bound": [
"DNS resolver latency",
"worker scheduling delay",
"SQLite lock wait"
],
"arrival_jobs_per_second": 2,
"outage_seconds": 600,
"generated_jobs": 1200,
"conclusion": "A ten-minute outage exceeds this retry envelope. Jobs become dead letters and need deliberate replay; a queue does not guarantee automatic delivery after arbitrary outages."
}
}Method
- crash.reclaim: Child persists a lease then os._exit. A declared clock advances from 1000 to 1002 seconds, past its one-second lease.
- crash.fence: Call completion with persisted old and new ownership tokens after restart.
- http.delivery: Forty normal and forty first-response-loss events, two in-flight requests per endpoint, real signed loopback HTTP.
- http.deduplication: Receiver stores a fingerprint and one effect per endpoint/event identity. Forty first responses are deliberately closed after commit.
- http.attempts: Count persisted terminal outcomes after every worker has drained.
- api.authorization: Real loopback HTTP without bearer authorization.
- api.boundary: Authenticated requests with foreign Host or Origin.
- api.storage: Inject sqlite3.OperationalError at acceptance, before the transaction starts; error response contains no payload or credential.
- restore.at-least-once: Snapshot a ready job, deliver it, restore the older snapshot into a new directory and deliver again to the same durable receiver.
- restore.identity: Replay the accepted event through the restored domain API.
- restore.corruption: Change stored bytes without changing checksum manifest.
- restore.no-overwrite: Attempt restore into a nonempty directory.
- integrity.final: Check live workload database after all worker threads stop.
- theory.retry-budget: Compute 4 x 2 + (1 + 2 + 4); exclude DNS, scheduling and lock waiting explicitly.
- backup.schedule: Run the supplied CLI in a subprocess with two snapshots and a 10 ms interval. This exercises scheduling without leaving a background process.
Reproduce
python operations_rehearsal.py
See OPERATIONS.md for startup, snapshot, restore and capacity assumptions. The machine-readable record is completion.json.
Boundaries
- The child crash is real process termination, but lease time is deliberately advanced by an injected clock. It is not machine power-loss testing.
- All destinations and events are local synthetic fixtures. No live external destination, DNS outage or public TLS ingress was exercised.
- The restore drill preserves the receiver database. Losing receiver deduplication while replaying a queue can duplicate external effects.
- Snapshots contain endpoint signing secrets. They require restricted storage and encryption at rest where the environment needs it; the checksum is not a signature.
- The theoretical retry calculation excludes unbounded OS DNS timing and scheduler delays. Single-host SQLite provides no automatic failover.