21.6 实战：完成一次隔离恢复演练

这是一次真实运行过的物理恢复，不是伪造的示例输出。

实验完成：

fresh synthetic marker
fresh full pgBackRest backup
post-backup marker
named restore point
post-target marker
forced WAL switch + pgBackRest check
exact-label restore to a fresh isolated path
read-only and promotion phase measurement
positive/negative business boundary
lineage and timeline proof
isolated shutdown
source postflight
14 adversarial counterexamples

安全边界：

local nonproduction sandbox only
production data/traffic forbidden
source cluster not stopped
live /pg/data not touched
no DCS/Patroni membership for restored copy
no TCP listener
archive push disabled
no routing change
no cleanup during acceptance
production approval pending

21.6.1 创建已知业务检查点并执行备份

先读合同

四个入口：

exact target：

pg36-l2-vagrant/pg-test
Pigsty v4.5.0
PostgreSQL 18.6
pgBackRest 2.59.0
pg-test-1 primary
pg-test-2/3 streaming replicas

任何 version/member/role drift 都先停止。

风险分级

action	risk	行为
`capture`	L0	只读采集 source/repository
`verify`	L0	重验已有 evidence
`review`	L0	hash、边界与 secret review
`all`	L0	`verify + review`
`drill:pitr`	L2	marker、full backup、fresh isolated restore
`reset:fixture`	L3	删除一个 run 的 rows 与 retained directory

普通 all 不包含 drill 或 reset。

无授权必须拒绝

PG36_EVIDENCE_DIR=/new/empty/path \
  static/labs/ch21/task.sh drill:pitr

结果：

exit=77
refusing PITR drill: exact target, nonproduction, data, traffic,
and confirmation guards are required

拒绝发生在任何 marker/backup/restore 前。

chapter-19 gate

formal wrapper 先运行：

PG36_CH19_INVENTORY=/absolute/mode-0600/reviewed.yml \
PG36_EVIDENCE_DIR="$run/preflight-ch19" \
  static/labs/ch19/task.sh all

要求：

hosts=4-distinct
pg-meta one primary
pg-test one primary + two streaming replicas
secret values redacted
sandbox_l2=accepted-with-exceptions
production_ch19_gate=pending

private inventory 不复制到 evidence。它用于验证 exact declaration，不应使用公开 placeholder 去做真实部署。

source preflight

在 DDL 前采集：

Patroni exact member set
pg-test-1 only leader/running
pg-test-2/3 replica/streaming
replay lag <= 1 MiB
source in_recovery=false
archive_mode=on
current WAL segment/timeline
recovery-critical settings
sanitized pgBackRest catalog

source system identifier 只保存在 runner 内存用于 equality，evidence 不导出 raw value。

fixture

setup.sql：

CREATE SCHEMA IF NOT EXISTS pg36_ch21;

CREATE TABLE IF NOT EXISTS pg36_ch21.recovery_probe (
    run_id       text        NOT NULL,
    stage        text        NOT NULL
                             CHECK (stage IN ('base','keep','discard')),
    token        text        NOT NULL UNIQUE,
    committed_at timestamptz NOT NULL DEFAULT clock_timestamp(),
    PRIMARY KEY (run_id, stage)
);

每轮 run ID：

run_YYYYMMDDTHHMMSSZ_<8 hex>

路径与 SQL token 只接受这个 allowlist，避免 path/command injection。

三阶段顺序

insert base
  -> take full backup
      -> insert keep
          -> pg_create_restore_point(name)
              -> insert discard
                  -> pg_switch_wal
                      -> pgBackRest check

顺序的逻辑：

stage	文件基础备份	后续 WAL	target 结果
base	已包含或由 backup 所需 WAL 一致化	是	present
keep	backup 后	target 前	present
discard	backup 后	target 后	absent

marker 不是 sleep

我们不靠：

sleep 2

猜测时间边界，而是保存：

run ID
unique token
database commit timestamp
restore-point LSN
target WAL filename

因此验证的是 transaction/WAL 边界。

fresh full

formal command：

sudo -iu postgres pgbackrest \
  --stanza=pg-test \
  --repo=1 \
  --type=full \
  --log-level-console=info \
  backup

runner 在前后读取 JSON catalog，要求恰好新增一份：

type=full
error=false
stanza status code=0

正式结果：

label                 20260729-201041F
backup command        2085.526 ms
logical bytes         36,121,841
repository delta       4,539,288
backup archive        timeline 7

不公开 repository key/cipher pass。

restore point

runner：

SELECT pg_create_restore_point(
  'pg36_ch21_run_20260729T201040Z_961665aa_keep'
);

具体名称由 run 生成，不应手工照抄。evidence 保存：

name
LSN
WAL segment
created_at UTC

强制并检查 archive

SELECT pg_switch_wal();

然后：

sudo -iu postgres \
  pgbackrest --stanza=pg-test --log-level-console=info check

注意没有 --repo=1。正式成功：

check duration          598.471 ms
target WAL              ...000020
repository max WAL      ...000021
target_wal_covered      true

公开结果可以省略完整 WAL 名；完整 secret-safe evidence 保留以审计。

第一次 formal 尝试为何失败

最初 runner 对 check 传了：

--repo=1

pgBackRest 2.59.0 返回：

ERROR [031]: option 'repo' not valid for command 'check'

失败发生在 restore path 创建前：

backup succeeded
markers retained
no isolated directory
no postmaster
source healthy

我们没有删除失败 backup/marker 来伪装“一次通过”，而是修正 command-specific option，再做一轮 fresh formal evidence。

formal 入口

只在 exact local sandbox：

export PG36_CH19_INVENTORY=/absolute/mode-0600/reviewed.yml
export PG36_EVIDENCE_DIR=/absolute/new-empty/ch21-run

export PG36_CH21_TARGET=pg36-l2-vagrant/pg-test
export PG36_CH21_NONPRODUCTION=true
export PG36_CH21_PRODUCTION_DATA=false
export PG36_CH21_PRODUCTION_TRAFFIC=false
export PG36_CH21_CONFIRM=BACKUP_AND_ISOLATED_PITR_CH21

static/labs/ch21/task.sh drill:pitr

evidence dir 非空则拒绝覆盖。

21.6.2 恢复到隔离集群，核对数据库与业务不变量

restore root

formal：

/data/pg36-ch21-restore/
  run_20260729T201040Z_961665aa/
    data/
    socket/
    log/
    spool/
    pg_hba.restore.conf

preflight：

root exact allowlist
root does not exist
not symlink
port 55432 unused
owner postgres
mode 0700

已存在 path 不做 --delta 覆盖，直接拒绝。

exact backup 与 target

formal restore 绑定：

--set=20260729-201041F
--type=name
--target=<this run keep restore point>
--target-action=promote
--target-timeline=latest
--archive-mode=off

这同时固定：

which base
where replay stops
what happens at target
which timeline path
whether recovered fork can archive

文件恢复

结果：

restore_copy_ms = 2758.206
files/bytes      from exact backup catalog
recovery.signal  present
standby.signal   absent

copy 完成不算 PostgreSQL 恢复完成。

隔离启动

runner 用 exact PostgreSQL 18 pg_ctl，覆盖：

listen_addresses=''
port=55432
private Unix socket mode=0700
private HBA
ssl=off
archive_mode=off
primary_conninfo=''
primary_slot_name=''
shared_preload_libraries=''
logging_collector=off
cluster_name=pg36-ch21-restore

live replica：

/pg/data
port 5432
cluster_name=pg-test
Patroni streaming

两者路径与控制面不相交。

recovery-critical maxima

source 观测：

max_connections=500
max_worker_processes=24
max_wal_senders=50
max_prepared_transactions=0
max_locks_per_transaction=500

formal restore 携带这些值。

开发时曾尝试：

max_connections=20

PostgreSQL 在重放前 FATAL 拒绝。修复方法不是降低 source 或编辑控制文件，而是让隔离 runtime 满足 source WAL 记录的要求。

两个 readiness 时刻

第一时刻：

start -> first connection = 962.980 ms
pg_is_in_recovery()       = true
transaction_read_only     = true

第二时刻：

start -> promoted         = 1318.876 ms
pg_is_in_recovery()       = false
transaction_read_only     = false

差：

355.896 ms

所以：

pg_ctl -w returned
  != target-action promotion complete

rollback write probe

promotion 后：

BEGIN;
CREATE TEMPORARY TABLE pg36_ch21_writable_probe(value integer);
INSERT INTO pg36_ch21_writable_probe VALUES (1);
ROLLBACK;

结果：

rollback-write-ok

它证明当前连接可写，不给历史副本增加持久业务事实。

business boundary

exact run：

base     present  true
keep     present  true
discard  present  false
unexpected stages 0

如果只看 row count=2，仍可能误拿其他 run；runner 对 exact token 比较。

lineage

system identifier relation = matches source
raw identifier recorded     = false
source timeline             = 7
restored timeline           = 8
timeline increment          = 1

这是同一物理血缘的一次合法 PITR fork。

isolation proof

运行中：

TCP listener        false
Unix socket         true
socket mode         0700
socket owner        postgres:postgres
postmaster.pid      true
archive_mode        off
Patroni managed     false

仅把 port 改成 55432 不算隔离；若 listen_addresses=0.0.0.0，仍会向网络暴露。

停止

pg_ctl -D <exact-data> -w -t 30 -m fast stop

停止后：

postmaster.pid              false
Unix socket                 false
TCP listener                false
restore directory retained  true

目录保留供审查，未自动删除。

source postflight

恢复后重新执行 chapter-19 gate，并在 drill 内检查 .13 live instance：

pg-test-1 leader/running
pg-test-2 replica/streaming
pg-test-3 replica/streaming
all Patroni timeline 7
source system ID unchanged
pg-test-3 in_recovery=true
pg-test-3 replay paused=false
pg-test-3 port=5432
replica lag=0 bytes

isolated timeline 8 不会加入 source Patroni cluster。

不做什么

no pgsql-pitr destructive replacement
no Patroni reinit
no failover/switchover
no /pg/data write
no service routing
no repository expire
no restore directory deletion

这使实验回答“backup 能否恢复”，而不是把多个高风险动作揉成一个结果。

21.6.3 输出 RPO/RTO 实测、证据链和失败处理 SOP

reference result

restore-run.json：

{
  "backup_command_ms": 2085.525583,
  "restore_copy_ms": 2758.205792,
  "start_to_first_connection_ms": 962.980167,
  "first_connection_in_recovery": true,
  "start_to_promoted_ms": 1318.876208,
  "first_connection_to_promoted_ms": 355.896041,
  "base_present": true,
  "keep_present": true,
  "discard_present": false,
  "source_timeline": 7,
  "restored_timeline": 8,
  "isolated_postmaster_stopped": true,
  "production_ch21_gate": "pending"
}

它不含 credential 或 raw system ID。

这次 RPO 证明了什么

证明：

fresh full ended before keep
keep WAL and named point were archived
restore stopped at named point
post-target discard did not replay

可以说：

named-point selection for this run passed

不能说：

production RPO = 0
any time in 14 days is recoverable
worst-case archive delay is 0
region loss loses no data

因为实验主动 pg_switch_wal() 并 check，没有模拟 source 在 open segment 尚未归档时突然毁坏。

这次 RTO 测量了什么

已测：

backup command
repository check
file restore copy
start to first read-only
start to promotion

未测：

incident detection
human decision/approval
new host provisioning
cold object retrieval
large production bytes
full business validation
application configuration
DNS/proxy/client cutover
unknown transaction reconciliation
backlog clearing

所以：

2.758s + 1.319s \not= RTO_{\text{production}}

甚至简单相加也不完整，因为时钟阶段与执行方式要明确。

四类时钟

建议输出：

T_backup
T_restore_copy
T_recovery_consistent/read_only
T_recovery_promoted
T_database_validation
T_business_validation
T_service_cutover
T_backlog_clear

本章前四项中的三个有正式观测，后四项保持未测。

evidence tree

<run>/
├── preflight-ch19/
├── drill/
│   ├── drill-manifest.json
│   ├── source-before.json
│   ├── fixture.json
│   ├── backup.json
│   ├── recovery.json
│   ├── isolated-shutdown.json
│   ├── source-after.json
│   ├── validation-report.json
│   └── negative-report.json
├── postflight-ch19/
└── review.txt

manifest 记录所有 source input SHA-256。restore-run.json 与 migration-effort.json 是 outcome，不参与输入 hash，但 reviewer 会把 published outcome 与 validation report 对齐。

十四个反例

negative-cases.json 要求拒绝：

反例	policy code
从沙箱声称生产通过	`E_PRODUCTION_CLAIM`
恢复到未审目标	`E_TARGET`
仓库状态失败	`E_REPOSITORY`
backup 标记失败	`E_BACKUP`
target WAL 未覆盖	`E_ARCHIVE`
recovered archive push 开启	`E_ISOLATION`
restore 有 TCP listener	`E_ISOLATION`
只读阶段当 promotion 完成	`E_PROMOTION`
`keep` 缺失	`E_BOUNDARY`
`discard` 存在	`E_BOUNDARY`
system lineage 不同	`E_LINEAGE`
timeline 未前进	`E_TIMELINE`
isolated postmaster 未停止	`E_SHUTDOWN`
source cluster 退化	`E_SOURCE_HEALTH`

formal：

case_count=14
actual_code == expected_code for every case
status=ok

为什么测反例

一个 validator 若只接受正确 evidence，可能只是检查文件存在。让每个关键断言被单独破坏并得到预期错误码，证明 decision rule 真正约束：

positive proof
+ negative falsification

这比“脚本 exit 0”更有审计价值。

read-only 重验

formal 结束后：

PG36_EVIDENCE_DIR=/absolute/completed/ch21-run \
  static/labs/ch21/task.sh all

输出：

status=validation-ok
sandbox_named_pitr=accepted-with-exceptions
counterexamples=14-rejected
status=review-ok
secret_values_exported=0
raw_system_identifiers_exported=0
isolated_postmaster=stopped
restore_directory=retained
production_ch21_gate=pending
mutation=none

它不会重跑 backup/restore。

失败 SOP：backup

backup command fails
  -> preserve console/log exec-id
      -> inspect stanza/repository/archive
          -> check source and disk health
              -> do not select partial label
                  -> retry only after cause and policy review

若已插入 base，它只是 synthetic evidence；不要为“干净”自动删。

失败 SOP：archive gap

target segment > repository max or actual restore requests missing WAL
  -> stop
      -> preserve source pg_wal
          -> inspect spool/archive errors/capacity
              -> recover missing WAL if possible
                  -> choose earlier target only with business approval

不能跳过 WAL 或把“最新可到达点”擅自当成批准 target。

失败 SOP：restore/start

restore fails
  -> retain exact directory/log
      -> ensure no postmaster remains
          -> inspect permission/tablespace/key/WAL
              -> create a new run/path for retry

不要用 --delta 在不清楚内容的目录上反复覆盖。

失败 SOP：target mismatch

base/keep missing or discard present
  -> reject candidate
      -> do not expose service
          -> review target/backup/inclusive/timeline
              -> select a new target with data owner
                  -> restore to another fresh path

PostgreSQL 能启动不降低严重性。

失败 SOP：isolation

若发现：

TCP listener
archive_mode=on
wrong HBA
Patroni/DCS membership
path overlaps /pg/data

立即停止 isolated instance，保留证据，确认 source repository 和 routing 未污染，再调查。不要“验证完再关”。

reset

删除不是 acceptance 的一部分。单独入口要求：

exact target
nonproduction guards
exact run ID
PG36_CH21_RESET_CONFIRM=DELETE_ONE_CH21_SANDBOX_RUN
no postmaster.pid
no Unix socket
path exact allowlist
not symlink

然后只删除一个 retained directory 与对应 run rows。本文 formal 没有执行 reset。

生产 gate

沙箱通过后，生产仍需：

代表性生产规模与 change/WAL rate；
独立 production-class restore infrastructure；
immutable、off-site、独立 credential/key 证明；
任意 time/XID/LSN 与 inclusive 语义；
最旧 retention target；
missing WAL、lost key、repository outage；
extension/tablespace/config/PKI 完整性；
应用与外部系统不变量；
service cutover、client outcome 与 backlog；
region loss 与 break-glass access；
多次样本分布，而不是一次秒数；
生产 change/incident authority。

migration-effort.json 把这些保持为 required_next_evidence。

本章最终判定

sandbox named PITR
  accepted with ten explicit exceptions

PostgreSQL physical lineage
  matched

target boundary
  base + keep, no discard

isolation
  path/process/socket/archive isolated
  shared restore host exception remains

source safety
  healthy before and after

production
  pending

小结

一次可信恢复演练同时具备：

known business boundary
+ fresh or exactly selected backup
+ continuous WAL evidence
+ isolated target
+ two-phase readiness
+ lineage/timeline proof
+ positive and negative invariants
+ source postflight
+ shutdown
+ residual-risk ledger

这才把“我们有备份”升级为“我们曾在明确边界内证明它可以恢复”。

下一章继续服务面：当一个 PostgreSQL 实例真的可写之后，连接池、代理、路由、会话与客户端重试如何决定用户何时恢复。

上一节：用 pgBackRest 交付备份策略 · 返回本章目录 · 下一章：四通八达：服务接入、连接池与路由 · 查看全书目录 · 查看索引中心