# Recovery operations

Run the complete package check on Linux with Python 3.10 or newer:

```sh
python verify.py
```

Run only the additional operational drills:

```sh
python operations.py
```

The command creates fresh synthetic databases and files, starts only its own child processes, and removes its temporary root. It uses real SIGKILL signals against those children. It never accepts a production database, remote server or destination restore path. `completion.json` records every expected and observed outcome; `operations-report.html` is the readable report.

## Executed cases

| Case | What was exercised | Acceptance |
| --- | --- | --- |
| Killed after extraction | Restore child stopped without cleanup | Source unchanged, partial owned directory identified, supervised cleanup and retry succeed |
| Killed after database copy | Same failure after database restoration | Same recovery and source-isolation checks |
| Killed after application probes | Successful probes followed by process death | No success report from dead process; retry required |
| Storage exhaustion | ENOSPC raised by the actual archive-copy function | Restore rejected, owned resources removed |
| Parallel verification | Two independent operating-system workers | Separate temporary paths, both valid, sources unchanged |
| Count limits | 1,000-file and 1,001-file generated archives | Documented limit accepted; excess rejected |
| Member damage | Attachment byte flipped, outer archive hash updated | Per-file checksum still rejects damaged bytes |
| Online database snapshot | SQLite backup API with a confirmed concurrent WAL commit | Recovered snapshot passes record/file consistency checks |

A killed process cannot run `finally`. The parent records the exact child-created directory, confirms process death, checks direct containment and its expected prefix, then removes that path. It does not scan or delete every similarly named directory. An external job runner needs the same ownership record; blindly deleting old temporary directories is not part of this tool.

The online snapshot keeps document references and attachment bytes immutable while a writer changes visit status. This tests a usable snapshot contract. If an application replaces attachment bytes in place, the database backup API alone cannot coordinate those external changes.

## PostgreSQL recovery: completed theoretical run

PostgreSQL and a container runtime were unavailable in this execution environment. `operations.py` therefore evaluates the original recovery design as a deterministic state model. Its seven cases contain a healthy path plus a failure at each of six required stages. Only the healthy path reaches acceptance. A database created by the failed run is marked for owned-resource cleanup; a download/checksum failure creates no database.

| Modeled stage | Concrete operation in a PostgreSQL implementation | Required evidence |
| --- | --- | --- |
| Download | Fetch a pinned backup version into a new staging directory | Exact database dump, archive and manifest identifiers |
| Checksum | Verify all local bytes against the selected manifest | Hashes match; failed input cannot enter restoration |
| Create target | Create a distinct disposable database from `template0` | Generated target identifier and ownership record |
| Restore | Restore a custom-format dump with error-stop behavior into that target | Successful restore process status and logs |
| Application probes | Compare declared snapshot, records and file hashes | All application invariants hold after restoration |
| Report and cleanup | Write evidence, remove only owned resources | Success report retained and owned resources removed |

This model tests control-flow decisions. It does not parse a PostgreSQL dump or establish PostgreSQL restore timing. The implemented SQLite runner provides real data, file and process-failure evidence for the same recovery contract.

PostgreSQL dump files represent a database snapshot; external attachment consistency remains the application's responsibility. Plain SQL restores can otherwise continue after an error, so a restore recipe must explicitly stop on errors. PostgreSQL documents restoration from `template0` and using `pg_restore` for non-text archives. See the official [SQL Dump documentation](https://www.postgresql.org/docs/18/backup-dump.html).

## Recovery budget

The six sizing cases are calculations with stated inputs, not performance claims:

`RTO seconds = database MiB / transfer MiB per second + database MiB / restore MiB per second + startup seconds + probe seconds`

The scenarios use 1, 10 and 100 GiB, transfer rates of 25 or 100 MiB/s, an assumed 50 MiB/s restore rate, 30 seconds startup and 60 seconds probes. Modeled RTO ranges from 120.72 to 6,234 seconds. With successful hourly snapshots, the model's worst snapshot-age RPO is one hour. Missed backups invalidate that bound. Compression, operator delay, queueing, failed downloads and retries are excluded and must be supplied as inputs to estimate a specific system.

A useful operational decision is to compare the computed window with a service's declared objective, then measure its actual transfer, restore and probe stages. This package has no arbitrary production RTO acceptance target.

## Results and run procedure

1. Preserve the backup set and its manifest. Run the good fixture first to verify the local environment.
2. Run `verify.py`. A nonzero exit means an expected detection or successful case did not match its assertion; consult `verification.json` and `completion.json`.
3. Interpret an expected corrupt-input rejection as a detection result. It is not evidence that the damaged backup can recover the service.
4. Keep output reports with the backup-set identity and source hashes. Local run times are useful for repeatability, not a substitute for measuring a larger backup.
5. For a forced process death, retain the supervisor's owned-path record. After confirming death, remove that exact disposable path and start a new isolated run.

The manifest is unsigned. A trusted signature or protected distribution channel is needed when authenticity matters. This package verifies consistency, not the identity of whoever created the backup.

## Method references

- [SQLite Online Backup API](https://www.sqlite.org/backup.html): consistent snapshots through the backup API, rather than an ordinary live-file copy.
- [SQLite write-ahead logging](https://www.sqlite.org/wal.html): reader/writer behavior and WAL files.
- [PostgreSQL 18 SQL Dump](https://www.postgresql.org/docs/18/backup-dump.html): dump consistency and restore error handling.

References were checked on 2026-10-27. PostgreSQL statements in this document are a design review of the theoretical run, not claims of PostgreSQL execution.