Executed operations and theoretical recovery
Backup recovery operations
11/11 expected outcomes matched. A rejected damaged input is a successful detection, not a recovered backup.
Acceptance record
| Case | Requirement | Method | Expected | Observed | Check |
|---|---|---|---|---|---|
| kill-files | Recover after hard process termination at files | real subprocess / SIGKILL | partial directory found; source unchanged; supervised cleanup; retry passes | {'exit_signal': 9, 'partial_directory_observed': True, 'source_unchanged': True, 'retry_passed': True, 'unrelated_preserved': True} | Matched |
| kill-restore | Recover after hard process termination at restore | real subprocess / SIGKILL | partial directory found; source unchanged; supervised cleanup; retry passes | {'exit_signal': 9, 'partial_directory_observed': True, 'source_unchanged': True, 'retry_passed': True, 'unrelated_preserved': True} | Matched |
| kill-probes | Recover after hard process termination at probes | real subprocess / SIGKILL | partial directory found; source unchanged; supervised cleanup; retry passes | {'exit_signal': 9, 'partial_directory_observed': True, 'source_unchanged': True, 'retry_passed': True, 'unrelated_preserved': True} | Matched |
| full-volume | Fail closed and clean owned resources on archive-write failure | injected ENOSPC | rejected; cleanup succeeds | {'passed_restore': False, 'cleanup': True, 'error': 'OSError: [Errno 28] synthetic full volume'} | Matched |
| parallel-isolation | Parallel verification does not share a restore target | two operating-system workers | two successful restores; distinct owned directories; source unchanged | {'successes': 2, 'distinct_directories': 2, 'source_unchanged': True} | Matched |
| count-1000 | Enforce documented file-count boundary | generated SQLite and archive fixture | restore_passed=True | {'restore_passed': True, 'seconds': 0.159169, 'input_bytes': 1349468, 'error': None} | Matched |
| count-1001 | Enforce documented file-count boundary | generated SQLite and archive fixture | restore_passed=False | {'restore_passed': False, 'seconds': 0.000587, 'input_bytes': 1349616, 'error': 'CheckError: Invalid file list'} | Matched |
| member-corruption | Reject changed attachment despite updated outer archive checksum | single-byte damage | member digest rejection | CheckError: File digest mismatch: visits/001.txt | Matched |
| online-wal-snapshot | Recover a snapshot produced while writes commit | SQLite backup API plus independent writer | valid snapshot; 300 commits complete; application probes pass | {'writer_errors': [], 'writer_commits': 300, 'restore_passed': True, 'writer_committed_during_copy': True} | Matched |
| postgres-recovery-model | Never accept a backup after an incomplete required stage | theoretical pipeline state model | healthy path accepted; all six injected faults rejected; no production target | {'scenarios': 7, 'accepted': 1, 'failures_rejected': 6} | Matched |
| recovery-budget | Calculate recovery window from declared rates | theoretical serial-stage budget | RTO increases with size and decreases with transfer rate | {'cases': 6, 'range_seconds': [120.72, 6234.0]} | Matched |
Interpretation
The process-kill checks prove that partial disposable restores do not alter the source backup and can be followed by a clean retry. Cleanup after SIGKILL belongs to the supervising job, because the killed Python process cannot execute its cleanup code. The live-WAL case uses SQLite's backup API; copying only a live database file is not substituted for it.
The PostgreSQL model carries a backup through download, checksum verification, isolated restore, application probes, report and owned-resource cleanup. Every injected stage failure prevents acceptance. It is a state-transition model of the original deployment design, not a test of PostgreSQL binaries.
Operating model
{
"live_snapshot_assumption": "Attachment bytes and document references remain immutable during capture; only visit status changes. A real application must supply an equivalent coordinated snapshot contract.",
"postgresql_pipeline_model": [
{
"fault": "none",
"completed": [
"download",
"checksum",
"create_disposable_database",
"restore",
"application_probes",
"report"
],
"accepted": true,
"cleanup_owned_database": true,
"production_target_used": false
},
{
"fault": "download",
"completed": [],
"accepted": false,
"cleanup_owned_database": false,
"production_target_used": false
},
{
"fault": "checksum",
"completed": [
"download"
],
"accepted": false,
"cleanup_owned_database": false,
"production_target_used": false
},
{
"fault": "create_disposable_database",
"completed": [
"download",
"checksum"
],
"accepted": false,
"cleanup_owned_database": false,
"production_target_used": false
},
{
"fault": "restore",
"completed": [
"download",
"checksum",
"create_disposable_database"
],
"accepted": false,
"cleanup_owned_database": true,
"production_target_used": false
},
{
"fault": "application_probes",
"completed": [
"download",
"checksum",
"create_disposable_database",
"restore"
],
"accepted": false,
"cleanup_owned_database": true,
"production_target_used": false
},
{
"fault": "report",
"completed": [
"download",
"checksum",
"create_disposable_database",
"restore",
"application_probes"
],
"accepted": false,
"cleanup_owned_database": true,
"production_target_used": false
}
],
"capacity_model": [
{
"database_GiB": 1,
"network_MiB_s": 25,
"restore_MiB_s_assumed": 50,
"download_s": 40.96,
"restore_s": 20.48,
"fixed_startup_and_probe_s": 90,
"modeled_RTO_s": 151.44,
"backup_interval_s": 3600,
"worst_case_RPO_s": 3600
},
{
"database_GiB": 1,
"network_MiB_s": 100,
"restore_MiB_s_assumed": 50,
"download_s": 10.24,
"restore_s": 20.48,
"fixed_startup_and_probe_s": 90,
"modeled_RTO_s": 120.72,
"backup_interval_s": 3600,
"worst_case_RPO_s": 3600
},
{
"database_GiB": 10,
"network_MiB_s": 25,
"restore_MiB_s_assumed": 50,
"download_s": 409.6,
"restore_s": 204.8,
"fixed_startup_and_probe_s": 90,
"modeled_RTO_s": 704.4000000000001,
"backup_interval_s": 3600,
"worst_case_RPO_s": 3600
},
{
"database_GiB": 10,
"network_MiB_s": 100,
"restore_MiB_s_assumed": 50,
"download_s": 102.4,
"restore_s": 204.8,
"fixed_startup_and_probe_s": 90,
"modeled_RTO_s": 397.20000000000005,
"backup_interval_s": 3600,
"worst_case_RPO_s": 3600
},
{
"database_GiB": 100,
"network_MiB_s": 25,
"restore_MiB_s_assumed": 50,
"download_s": 4096.0,
"restore_s": 2048.0,
"fixed_startup_and_probe_s": 90,
"modeled_RTO_s": 6234.0,
"backup_interval_s": 3600,
"worst_case_RPO_s": 3600
},
{
"database_GiB": 100,
"network_MiB_s": 100,
"restore_MiB_s_assumed": 50,
"download_s": 1024.0,
"restore_s": 2048.0,
"fixed_startup_and_probe_s": 90,
"modeled_RTO_s": 3162.0,
"backup_interval_s": 3600,
"worst_case_RPO_s": 3600
}
],
"capacity_equation": "RTO = database MiB / download MiB/s + database MiB / restore MiB/s + 30 s startup + 60 s probes. RPO bound assumes successful hourly backups; failed or stale backups invalidate that bound. Compression, parallelism, secrets, retries and operator time are excluded."
}Evidence boundaries
- Process termination is real SIGKILL on Linux; it does not remove power or emulate a faulty storage device.
- ENOSPC is an injected write error. No host volume was filled.
- SQLite restores and consistency probes are executed. The PostgreSQL pipeline and remote transfer timing are explicit theoretical models, not pg_restore or network measurements.
- Synthetic records and local timings describe this run only. Backup authenticity needs a trusted signature or protected manifest; hashes alone do not authenticate a backup.
- A killed process cannot run its finally block. The supervisor demonstrates cleanup of only the exact child-owned path after confirming process death.
Machine-readable record · Acceptance CSV · Runbook and PostgreSQL analysis · Reproduction source