Executed synthetic operations
Usage ledger operations rehearsal
These checks use generated records, disposable local databases and deliberate faults. A passing check means the stated expected outcome occurred, including expected rejection or rollback. It does not certify an internet-facing deployment or hardware power-loss behavior.
Results
| Case | Requirement | Expected | Observed | Result |
|---|---|---|---|---|
| crash.ingest | Abrupt exit before ingest commit leaves every business table empty | {"exit": 86, "rows": 0} | {"exit": 86, "rows": 0} | PASS |
| crash.response | Committed ingest survives process exit and replay creates no extra quantity | {"exit": 87, "duplicate": true, "quantity": "1.125"} | {"exit": 87, "duplicate": true, "quantity": "1.125"} | PASS |
| load.idempotency | Eight concurrent writers turn 800 requests into 400 source events | {"new": 400, "duplicate": 400, "matches": true} | {"new": 400, "duplicate": 400, "matches": true} | PASS |
| load.exact-total | Concurrent exact quantities sum without rounding | "450.000" | "450.000" | PASS |
| load.corrections | Ten corrections append reversals and replacements with the expected net | "447.500" | "447.500" | PASS |
| close.rollback | Crash before close commit leaves period open and no frozen version | {"exit": 86, "status": "open", "closes": 0} | {"exit": 86, "status": "open", "closes": 0} | PASS |
| close.immutability | Late event and correction leave the frozen January digest unchanged | "cfa9e8ea751f8921176b347023e120d2b8d93a7e481e2bcf4c418b56f3046231" | "cfa9e8ea751f8921176b347023e120d2b8d93a7e481e2bcf4c418b56f3046231" | PASS |
| api.isolation | Other-account read and viewer mutation are rejected | [403, 403] | [403, 403] | PASS |
| api.boundary | Foreign Host and Origin are rejected | [403, 403] | [403, 403] | PASS |
| api.storage | Storage failure returns HTTP 503 without inventing success | 503 | 503 | PASS |
| restore.frozen-export | Restored January export matches every frozen row and digest | {"sha256": "cfa9e8ea751f8921176b347023e120d2b8d93a7e481e2bcf4c418b56f3046231", "entries": 420, "identical": true} | {"sha256": "cfa9e8ea751f8921176b347023e120d2b8d93a7e481e2bcf4c418b56f3046231", "entries": 420, "identical": true} | PASS |
| restore.point-in-time | Snapshot correctly excludes the two records received after backup | 0 | 0 | PASS |
| restore.source-replay | Stable upstream identities restore two lost records exactly once | 2 | 2 | PASS |
| restore.carry-forward | Recovered late usage and correction close February at 2.625 minutes | "2.625" | "2.625" | PASS |
| restore.closed-digest | Recovery and carry-forward preserve the January digest | "cfa9e8ea751f8921176b347023e120d2b8d93a7e481e2bcf4c418b56f3046231" | "cfa9e8ea751f8921176b347023e120d2b8d93a7e481e2bcf4c418b56f3046231" | PASS |
| restore.corruption | Modified snapshot is rejected before destination creation | true | true | PASS |
| restore.no-overwrite | Existing restore destination is preserved | true | true | PASS |
| integrity.final | Recovered database passes integrity and foreign-key checks | {"integrity": "ok", "foreign_key_errors": 0} | {"integrity": "ok", "foreign_key_errors": 0} | PASS |
| backup.schedule | Finite scheduler creates two checked snapshots and stops | {"exit": 0, "snapshots": 2, "valid": 2} | {"exit": 0, "snapshots": 2, "valid": 2} | PASS |
Measurements
{
"concurrent_ingestion": {
"writers": 8,
"requests": 800,
"unique_events": 400,
"elapsed_seconds": 0.8952,
"requests_per_second": 893.67,
"qualification": "Finite local synthetic run with repeated connection initialization, not a production rate guarantee."
},
"recovery": {
"snapshot_bytes": 245760,
"snapshot_seconds": 0.004807,
"restore_seconds": 0.006143,
"snapshot_source_events": 410,
"events_after_snapshot": 2,
"upstream_replay_operations": 4,
"new_events_after_replay": 2,
"nominal_backup_interval_seconds": 900,
"theoretical_max_unbacked_events_at_2_per_second": 1800,
"assumption": "At most two events per second, successful backup every 15 minutes. This is a capacity/RPO scenario, not measured customer traffic."
}
}Method
- crash.ingest: Child os._exit after event, entry, posting, totals and audit writes but before SQLite commit.
- crash.response: Child exits after commit. Parent reopens the database and retries the original identity.
- load.idempotency: Separate SQLite connections ingest every synthetic event identity twice.
- load.exact-total: 400 unique events x 1.125 minutes = 450.000 minutes using integer base units.
- load.corrections: 450.000 - 10 x (1.125 - 0.875) = 447.500 minutes.
- close.rollback: Child exits after writing the close, period status and audit row but before commit.
- close.immutability: Post-close input stays pending. Export rebuild uses the stored posting cutoff.
- api.isolation: Real HTTP requests use a demo-scoped operator token and a viewer token.
- api.boundary: Otherwise authorized real loopback requests.
- api.storage: Inject sqlite3.OperationalError before domain ingestion; retry keeps the stable event identity.
- restore.frozen-export: Restore the pre-late-input snapshot into a new directory and rebuild the canonical export.
- restore.point-in-time: This expected loss demonstrates a nonzero backup recovery-point gap.
- restore.source-replay: Replay the retained synthetic input journal twice against the restored snapshot.
- restore.carry-forward: 3.125 late minutes - (1.125 - 0.625) correction = 2.625, posted after the immutable January close.
- restore.closed-digest: Rebuild January again after February approval and close.
- restore.corruption: Change six bytes without changing checksum manifest.
- restore.no-overwrite: Attempt recovery into an occupied folder.
- integrity.final: PRAGMA integrity_check and foreign_key_check after replay, approval and close.
- backup.schedule: Run the supplied CLI in a subprocess with two snapshots and a 10 ms interval. This exercises scheduling without leaving a background process.
Reproduce
python operations_rehearsal.py
See OPERATIONS.md for startup, snapshot, restore and capacity assumptions. The machine-readable record is completion.json.
Boundaries
- The crash drill terminates a process before or after commit. It does not simulate disk firmware, physical power loss or media failure.
- All usage and tokens are synthetic. No invoice, tax, currency or payment conclusion follows from these quantities.
- Point-in-time recovery deliberately loses post-snapshot events until replay. Recovery requires a separately retained source journal with stable identities; the service cannot reconstruct absent source data.
- Snapshot checksum verification detects corruption, not malicious replacement of both database and manifest. Protect or encrypt backups where required.
- The local timing and 15-minute backup scenario are evidence and assumptions respectively, not a service-level agreement or enabled schedule.