# 切换与重建 runbook

LLMS 索引： [llms.txt](/llms.txt)

---

runbook 不是命令收藏。它把每个动作绑定到触发条件、证据、风险、停止线、成功谓词与
复位路径。切换场景尤其要防止“命令执行成功”替代“唯一 authority 已建立”。

建议每一步都使用：

```yaml
step: stable-id
intent: why
preconditions: [...]
command_or_action: exact
risk_class: R0 | R1 | R2 | R3
expected: [...]
evidence: [...]
stop_if: [...]
rollback_or_next_safe_state: [...]
owner: role
```

## 33.6.1 计划切换、故障切换与人工干预入口 {#item-33-6-1}

### 先选路径

```text
healthy leader + maintenance need
  -> planned switchover

leader process demonstrably failed/fenced
  + automatic policy healthy
  -> observe automatic failover

no accepted leader
  + automatic path cannot complete
  + exact candidate/fence/data risk approved
  -> manual failover

replica broken, leader healthy
  -> restart/rewind/reinit replica; do not fail over
```

第 20 章已经完整演练 healthy planned switchover；本章正式 run 是 controlled process
fault 后的 automatic failover，最后才用 planned switchover 复原教学基线。

### R0：只读预检

```bash
pig pt list pg-test -o json
pig pt config show -o json
```

加上 SQL：

```sql
SELECT pg_is_in_recovery(),
       (pg_control_system()).system_identifier,
       (pg_control_checkpoint()).timeline_id;
```

检查：

```text
one primary / expected replicas
pause=false
member tags
ttl / loop_wait / retry_timeout
maximum_lag_on_failover
synchronous and failsafe modes
watchdog mode
sender/receiver and replay gap
DCS member/quorum
service path and client probe
```

R0 也要创建 evidence timestamp，不能把几分钟前的健康状态当动作瞬间事实。

### R1/R2：计划切换

```bash
pig pt switchover --plan
pig pt switchover \
  --leader <current> \
  --candidate <target>
```

版本选项以 `pig pt switchover --help` 为准。执行前：

- current leader 与 candidate 都来自 fresh structured state；
- candidate replay 在窗口内；
- 长事务、DDL、备份和批任务已评估；
- client probe 已开始；
- backout candidate 可用；
- 路由与 connection drain owner 在线。

执行后不能立刻退出：

```text
new timeline
old leader streaming
client writes on new authority
unknown outcome reconciled
archive follows new primary
scheduled jobs no duplicate
```

### R2/R3：故障与手工切换

自动 failover 通常不需要操作者再发一条 failover 命令；重点是确认 fence、候选和服务
收敛。manual path：

```bash
pig pt failover --candidate <exact-member> --plan
```

计划应明确警告 leader 不可用时可能丢失未复制事务。执行前需要独立 reviewer：

```text
old-primary fence
candidate lineage and lag
last ack / unknown manifest
accepted data loss
DCS authority
business owner and incident commander authorization
```

不要用 manual failover 修复 proxy、replica 或 DCS 延迟问题。

### 重建入口

```bash
pig pt reinit <replica> --plan
```

reinit 会删除目标 replica 的 PGDATA，属于 R3。它只应在：

- target 已确认是 replica；
- target 数据无需再取证；
- accepted primary/source 明确；
- rewind 不适用或已失败；
- 带宽、WAL retention、tablespace 和密钥已准备；
- post-validation 与 failure cleanup 已写好；
- 两人复核具体 member。

本章 runner 永不执行 managed reinit。

## 33.6.2 Patroni、DCS、代理和 SQL 证据互证 {#item-33-6-2}

### 证据矩阵

| 问题 | Patroni | DCS | SQL | 服务/客户端 |
|---|---|---|---|---|
| 谁是 primary | REST role | leader lock | recovery=false | write health |
| 旧主是否排除 | member state | lease/failsafe | unavailable/recovery | no accepted route |
| 候选是否最新 | timeline/lag | member/sync state | LSN/receiver | token boundary |
| 切换是否完成 | one leader | lock updated | followers streaming | writes succeed |
| 数据是否丢失 | 不直接回答 | 不直接回答 | token/ledger query | client ack manifest |

任一行中出现冲突，都应保存而不是“修正输出”。例如：

```text
DCS leader = A
Patroni REST A = replica
SQL A recovery = true
SQL B recovery = false
```

这不是运行一个 edit-config 的理由，而是 authority contradiction。先停止写、查 timeline
与动作日志。

### 采集顺序避免自我污染

```text
capture before
start client probe
record action monotonic timestamp
perform one bounded action
capture fence and transition
capture after
reconcile client identities
only then perform cleanup/baseline restore
capture restored
```

如果先 restart/reinit 再取证，就会丢失原 PID、日志、control data 和 timeline 现场。
如果先删除 fixture 再对账，就无法证明 unknown 是否提交。

### Secret-free projection

完整 Patroni、DCS 与 inventory 配置含密码、token、证书路径。证据只投影：

```text
scope / member
DCS kind and endpoint count
config booleans and numeric policy
credential present + hash/length where necessary
REST role/state/timeline
raw log hash + selected event counts
```

本章 journal evidence 只保留行数、整体 SHA-256 和 pattern counts，明确
`raw_log_exported=false`。需要审计原日志时，在受控现场保存，不把它发布到教材。

### monotonic 与 UTC 双时钟

UTC 用于跨主机关联，monotonic 用于本机阶段时长：

```text
started_at UTC
started_monotonic_ns
finished_at UTC
finished_monotonic_ns
```

NTP 校时可能让 wall clock 跳变，不能用两个 UTC 字符串相减替代 monotonic duration。
跨主机 monotonic 不能直接比较，因此 fence/promotion 关键顺序最好由同一 observer
记录，另用多主机 UTC/clock offset 辅助。

### 证据 bundle

```text
requirements + failure model
before phase
fault action
old-primary fence
failed phase + selected candidate
rejoin phase
client event stream + reconciliation
baseline restore action + restored phase
DCS tabletop decision
rewind/basebackup evidence
cleanup
source hashes
positive validation + adversarial mutations
public summary
```

公开摘要
[`failover-run.json`](/labs/ch33/failover-run.json) 不含 inventory、密码、原始日志或
临时 service 文件。

## 33.6.3 集群恢复后重新建立监控与备份健康 {#item-33-6-3}

### HA 恢复不是事故关闭

新主可写后，至少检查：

```text
database authority
replication and slots
client route and unknown outcomes
WAL archive
backup schedule and repository
monitoring/alerts
jobs, CDC, logical replication
capacity and cache
security/audit
```

切换会改变产生 WAL、执行 cron/job、归档、备份和 logical publisher 的节点。只验证
`SELECT 1` 会漏掉一半恢复工作。

### 复制与 slot

```sql
SELECT application_name, state, sync_state, replay_lsn
FROM pg_stat_replication;

SELECT slot_name, active, restart_lsn, wal_status,
       safe_wal_size, invalidation_reason
FROM pg_replication_slots;
```

验收：

- 所有预期 replica streaming；
- 无未知 sender；
- slot owner 与 member/subscriber 对应；
- `pg_wal` retained bytes 有界；
- rebuild member replay 到验证 LSN；
- logical replication origin/subscription 状态正确。

### archive 与 backup

```text
pg_stat_archiver last success/error
pgBackRest stanza status/check
new-primary archive command and credentials
latest backup lineage/timeline
repository capacity and retention
next scheduled backup owner
```

更稳妥的事故关闭门是：在新 topology 上完成一次新的可恢复点，并在计划窗口验证 side
restore；至少不能让 backup/archiving warning 被“主库已恢复”盖掉。

### 服务与应用

- HAProxy 只接受当前 write endpoint；
- PgBouncer 不保留错误 server connection；
- stale DNS/VIP 已收敛；
- application pool 重建；
- old timeline transaction 全部断开或明确结束；
- unknown idempotency token 对账完成；
- scheduler 单实例语义恢复；
- outbox/CDC offset 没有重复或空洞；
- cache/search 等派生系统按权威数据库重建。

### 监控重新设基线

切换会让：

```text
timeline increase
backend PID reset
cumulative stats reset/restart
cache hit ratio 暂时下降
replica lag spike
connection errors spike
checkpoint/archive timing change
```

这些不是都应静默。给事故窗口加 annotation，保留告警作为证据，再针对已解释的瞬态
调整状态；不要批量关闭告警后忘记恢复。

### 关闭条件

```yaml
authority:
  unique_primary: true
  old_primary: streaming_or_decommissioned
data:
  acknowledged_missing: 0
  unknown_unreconciled: 0
  accepted_loss: documented
replication:
  expected_members_streaming: true
  slots_reconciled: true
service:
  write_route_valid: true
  stale_connections_drained: true
recoverability:
  archive_healthy: true
  backup_healthy: true
  restore_followup_scheduled: true
operations:
  alerts_restored: true
  jobs_and_cdc_validated: true
  evidence_bundle_sealed: true
business:
  owner_acceptance: approved | pending
```

技术项全绿但 `owner_acceptance=pending` 时，事故仍不能被描述成“业务已完全恢复”。

---

[上一节：旧主重加入与集群重建](../05/) · [返回本章目录](../) · [下一节：实战：主库故障与 DCS 干扰](../07/) ·
[查看全书目录](/toc/) · [查看索引中心](/indexes/)