# Ledger operations

The local service accepts exact usage quantities, preserves corrections, and closes immutable monthly exports. This operations run exercises lost responses, process crashes, concurrent ingestion, restore from an older snapshot, and replay of source records missing from that snapshot.

## Local service

Use Python 3.12 or newer and a local disk. Keep the source directory separate from runtime data. A synthetic instance starts with:

```sh
umask 077
python app.py --demo --db data/ledger.sqlite --port 9090
```

For private local records, use a restricted token file with the account, role and actor claims described in README.md. Demo tokens are public fixtures. The server binds to loopback, rejects foreign Host and Origin values, and gives request sockets a 15-second timeout. An operation that encounters a storage error returns HTTP 503. Retry with the same source and event ID so the ledger can recognize a prior commit.

## Check and snapshot

```sh
python operations.py inspect --db data/ledger.sqlite
mkdir -p backups
python operations.py backup --db data/ledger.sqlite --out backups/ledger-001
```

The snapshot uses SQLite's online backup API, not a copy of a live database file. It includes committed WAL transactions. The manifest records project identity, row counts, length and SHA-256; integrity and foreign keys are checked before success. Snapshot directories and files receive restricted POSIX permissions.

A finite scheduled run creates two checked snapshots and stops:

```sh
python backup_schedule.py --db data/ledger.sqlite --out backups --interval 1 --count 2
```

Omit `--count` for periodic operation, with an interval in seconds. It retains all snapshots and exits nonzero on an error. Use a supervisor if continuous operation is required, and copy completed snapshots to independent protected storage when host loss is in scope. The snapshots are not encrypted or signed.

## Restore and replay

Stop the service before switching databases. Never restore over the current store.

```sh
python operations.py restore --snapshot backups/ledger-001 --out recovered-ledger
python app.py --tokens private-tokens.json --db recovered-ledger/store.sqlite --port 9090
```

Restore verifies the checksum, schema tables, page integrity, foreign keys and manifest counts before creating a new database. Retain the former store and the upstream source journal. Inspect the restored close list and exported digests before resuming normal ingestion.

Records received after the snapshot are absent from the restored database. Re-submit those records using their original source and event IDs. Replaying a known record is idempotent; using a new ID would create new usage. The service cannot reconstruct events that were lost from both the ledger and upstream source.

The rehearsal writes `results/operations-replay.json` with a synthetic late event and correction. It restores a snapshot that lacks both records, replays that input twice, and verifies that only two pending records exist. After approval into February, the total is 2.625 minutes. January's digest remains identical before and after the recovery.

## Executed checks

Run `python operations_rehearsal.py`. The report is `operations-report.html`; machine-readable outcomes are in `completion.json` and `results/operations-checks.csv`.

- A child exits after ingest writes but before commit. No event, entry, posting, total, period or audit row remains.
- Another child exits after committing. The retried identity returns the original ingestion and does not add quantity.
- Eight writers issue 800 requests for 400 distinct events. The ledger records exactly 450.000 minutes. Ten corrections reduce that to 447.500 by appending reversal and replacement entries.
- A process exits while closing January before commit. The period remains open and no frozen version appears. A later ordinary close succeeds.
- A late event and correction leave the January export unchanged. Account isolation, read-only roles, Host/Origin checks and storage rejection are exercised through HTTP.
- A restored snapshot reproduces every January export row and its digest. Stable source replay recovers post-snapshot records exactly once, then carries their 2.625-minute adjustment into February.
- Changed snapshot bytes and occupied destinations are rejected. A two-iteration scheduler run creates checked snapshots and exits.

These are generated records. They are not measured customer usage or financial results.

## Recovery assumptions

A nominal 15-minute backup interval at two events per second can leave up to 1,800 events after the last completed snapshot, plus arrivals during snapshot processing. That is a recovery-point scenario, not a workload estimate. A separately retained upstream journal closes that gap through idempotent replay. Without it, a backup policy provides a bounded loss window rather than zero data loss.

The measured restore time covers validation and copying on the execution host. Detection, replacement infrastructure, checking private credentials, replaying the source journal and operator approval add to service recovery time.

In a distributed design, ingestion, correction posting and period close must share an account-period lock or equivalent serializable rule. A frozen export must retain its posting cutoff and exact rule version. The current SQLite implementation obtains a database writer lock, which is broader and less concurrent. The run demonstrates that contract on SQLite; it does not claim tested Postgres locks, multi-region failover, pricing, tax or payment behavior.