Back up and restore

A control plane’s backup has a harder job than most: it must come back with its tamper-evident ledger provably intact. olivares dr is built around that requirement — every bundle records per-tenant chain tips, restore fails non-zero if the restored ledger is not continuity-safe, and dr drill checks backup-to-restore ledger continuity on a synthetic ledger in a scratch directory and prints the measured RTO. It does not check grants, key custody or effects after the restore point, and it does not touch a production data directory.

The bundle is encrypted under a KEK you provide — an Argon2id-derived passphrase (--passphrase-file) or a raw 32-byte key from your KMS (--kek-key-file); exactly one is required. The audit and catalog signing keys travel sealed inside the bundle.

The enrolled DR restore control supports PostgreSQL 15, 16, 17 and 18 in 26.10 (measured on 15.19, 16.15, 17.11 and 18.6). Other majors refuse with DR restore control contract: PostgreSQL major <major> is unsupported (verified contracts: 15, 16, 17, 18), where <major> is the detected server major. A control is never repaired: after an in-place major upgrade, a control that does not match the new major’s catalog and permissions is refused.

Back up

SQLite (single node) — safe while serve is running (the snapshot uses VACUUM INTO; WAL allows the concurrent read):

olivares dr backup \
  --data-dir /var/lib/olivares --engine sqlite \
  --out /backups/olivares-dr-$(date -u +%Y%m%dT%H%M%SZ).drbundle \
  --passphrase-file <your-dr-passphrase-file>

Postgres — a consistent pg_dump --format=custom driven by the same command (--engine postgres --dsn … --admin-dsn …), or hand it a pre-made dump with --snapshot-file. Running the dump directly requires --admin-dsn: pg_dump keeps row_security=off and aborts as the application role against FORCE ROW LEVEL SECURITY tables, so the command refuses up front rather than producing nothing. For near-zero RPO, --pitr-ref produces a keys+manifest companion bundle that pairs with your WAL-archiving PITR setup (deploy/postgres/backup/pitr-setup.md); the wrapper scripts deploy/postgres/backup/pg-dump.sh / pg-restore.sh package the same flow.

Two honesty switches worth knowing:

• The backup refuses to capture a ledger that does not verify at backup time — --allow-unverified exists, is logged, and is not recommended.
• With a pre-made snapshot (--snapshot-file/--pitr-ref) and no --admin-dsn, the backup warns that the captured tenant set may be RLS-limited and incomplete — the dump itself is fine, the manifest’s cross-tenant inventory is what needs the admin role. (Running pg_dump directly is a different case: it is refused outright, see above.)

Scheduling: the Compose stack ships a backup profile, the Helm chart a CronJob; on bare metal, cron the command above. The figures in the table are targets. For the cron tiers, the RPO target is the cron interval. dr drill measures SQLite RTO on a synthetic ledger (restore, boot and verify). It does not measure RPO, and it does not measure a Postgres restore. The measured SQLite RTO values, with their event counts and the host they were taken on, are in docs/DR-RUNBOOK.md; they are not the targets in this table.

Tier	Mechanism	Target RPO	Target RTO
SQLite	dr backup on cron	the cron interval	< 15 min
Postgres logical	pg-dump.sh on cron	the cron interval	< 30 min
Postgres PITR	base backup + WAL archiving	≈ seconds	< 30 min

Mirror bundles offsite and keep the KEK separate from the bundles (3-2-1): a same-host backup is not disaster recovery, and a bundle traveling with its passphrase is not encrypted in any sense that matters.

Drill — before you need it

dr verify checks a bundle without touching your data directory. On SQLite it restores into a scratch directory and verifies ledger continuity, and exits non-zero when that check fails. On Postgres it checks the bundle digest and that the keys decrypt. Full chain verification for a Postgres bundle needs a restore onto a scratch Postgres. Run the check with:

olivares dr verify --in /backups/olivares-dr-<ts>.drbundle \
  --passphrase-file <your-dr-passphrase-file>

dr inspect --in <bundle> prints the manifest (no KEK needed, no secrets shown) — what engine, which tenants, which chain tips. Run the drill on the same cadence as the backup; an unverified backup is a hope, not a control.

Restore

olivares dr restore --in /backups/olivares-dr-<ts>.drbundle \
  --data-dir /var/lib/olivares --engine sqlite \
  --passphrase-file <your-dr-passphrase-file>

The restore sequence is deliberate: signing keys first (fail-closed on overwrite — --force is the explicit override), then the store snapshot, then it boots the restored store and proves ledger continuity, exiting non-zero if the chain is not safe. After any restore, re-verify against your off-box checkpoint pin — a restored older snapshot can pass a naive walk yet fail the off-box comparison (troubleshooting § ledger).

The two keys that decide everything

Key	Rule
The DR KEK (passphrase or raw key)	without it every bundle is noise. Store it in a different system than the bundles; losing both at once is the failure mode
audit-signing.key (in the data dir)	back it up off-box at provisioning — the engine only warns on first boot, there is no enforced escrow, and a lost key makes the ledger permanently unverifiable. Pin the public key off-box too (GET /v1/audit/pubkey)

For KMS-based custody of the signing keys themselves (BYOK envelopes, rotation ceremonies, olivares keys), see the CLI reference; for the failure-mode walkthroughs, the troubleshooting page.