# 实战：一次有证据的计划切换

LLMS 索引： [llms.txt](/llms.txt)

---

这是一次真实运行过的实验，不是示例输出。

在第 19 章保留的本地四机沙箱中，本章对 `pg-test` 执行：

```text
pg-test-1 -> pg-test-2 planned switchover
pg-test-2 -> pg-test-1 planned baseline restore
```

同时通过 Pigsty primary service 连续写 synthetic idempotency token，保存
PostgreSQL、Patroni、client 与 chapter-19 pre/postflight evidence。

安全结论先写在前面：

```text
local disposable sandbox only
production data/traffic forbidden
planned transition only
no process/host/network/storage/DCS fault
no reinit
no cluster reset
fixture reset not executed
production approval remains pending
```

## 20.7.1 预检查、切换、客户端观察与数据核对 {#item-20-7-1}

### 风险分级

| action | risk | 远端 mutation |
|---|---:|---|
| `capture` | L0 | 无 |
| `verify` | L0 | 无 |
| `review` | L0 | 无 |
| `all` | L0 | 无 |
| `drill:switchover` | L2 | synthetic fixture + 两次 planned switchover |
| `reset:fixture` | destructive | drop `test.pg36_ch20` schema |

本节只描述如何在指定 local sandbox 重现。生产环境即使也叫 `pg-test`，也
不能因此获得授权。

### 先读合同

- [`lab-contract.md`](/labs/ch20/lab-contract.md)
- [`requirements.json`](/labs/ch20/requirements.json)
- [`failure-model.json`](/labs/ch20/failure-model.json)
- [`ha-adr.md`](/labs/ch20/ha-adr.md)

确认：

```text
target = pg36-l2-vagrant/pg-test
three exact members
initial/final leader = pg-test-1
forward candidate = pg-test-2
production_data_permitted=false
production_traffic_permitted=false
```

### 第 19 章基线仍是前提

正式运行前，wrapper 自动执行：

```bash
PG36_EVIDENCE_DIR="$run/preflight-ch19" \
  static/labs/ch19/task.sh all
```

必须看到：

```text
status=ok
hosts=4-distinct
topology=pg-meta-1-primary+pg-test-1-primary-2-replicas
sandbox_l2=accepted-with-exceptions
production_ch19_gate=pending
mutation=none
```

若第 19 章失败，本章不“顺便修”。先解释 target/host/topology drift。

### 私密输入

需要第 19 章 private inventory，只用于生成临时 libpq service file：

```text
mode=0600
not committed
not copied into evidence
password never printed
```

helper 从 exact `pg-test` user declaration 取 credential，输出：

```text
[pg36-ch20]
host=10.10.10.11
port=5433
dbname=test
user=test
target_session_attrs=read-write
sslmode=prefer
```

此处故意不展示 `password`。临时文件是 mode 0600、拒绝 symlink/overwrite，
wrapper 退出时删除。

`sslmode=prefer` 只因本地 5433 没有 TLS，形成命名例外；生产不可复制。

### synthetic fixture

[`setup.sql`](/labs/ch20/setup.sql) 建立：

```sql
CREATE SCHEMA pg36_ch20;

CREATE TABLE pg36_ch20.write_probe (
    run_id        text        NOT NULL,
    attempt_no    integer     NOT NULL,
    token         text        NOT NULL UNIQUE,
    client_sent_at timestamptz NOT NULL,
    committed_at  timestamptz NOT NULL DEFAULT clock_timestamp(),
    PRIMARY KEY (run_id, attempt_no)
);
```

正式文件还验证 schema owner、column type/nullability、primary/unique key；
如果已存在但结构漂移，拒绝使用。

fixture 只含 synthetic token。清理不是 acceptance 前提，也不会被 drill
自动 drop。

### preflight phase

在任何 DDL 前，底层脚本先 capture `before`：

```text
Patroni exactly one pg-test-1 primary
pg-test-2/3 streaming
one system identifier
SQL roles agree
two sender rows
replay gap <= 1 MiB
dynamic policy exact
pause=false
```

然后创建/验证 fixture、启动 probe，等第一个 acknowledged write，再 warm
up 3 秒并 capture `pre-switch`。

为什么要两次：

```text
before
  证明 mutation 前 baseline

pre-switch
  证明 probe 已运行且 action 紧邻前 topology 仍合格
```

### client probe

运行 24 秒，每 0.2 秒尝试：

```sql
INSERT ... RETURNING
  committed_at,
  pg_current_wal_insert_lsn(),
  current WAL filename timeline,
  backend pid,
  pg_is_in_recovery();
```

每次 token：

```text
<run UUID>:<8-digit attempt>
```

事件立即 append JSONL 并 `fsync` 本地 evidence：

```text
attempt start/end monotonic ns
acknowledged or unknown
SQLSTATE/error class without credential
WAL LSN/timeline for acknowledged event
```

probe 不把异常 token 当成一个新业务动作重试。

### 执行正式动作

读者使用当前 Pigsty 时，可以先：

```bash
pig pt switchover --plan
```

本书 exact v4.5.0 target 的 formal executor 是：

```bash
patronictl -c /etc/patroni/patroni.yml \
  switchover pg-test \
  --leader pg-test-1 \
  --candidate pg-test-2 \
  --force
```

但不要手工绕过本章 guard。完整入口：

```bash
export PG36_CH19_INVENTORY=/absolute/private/path/pg36.yml
export PG36_EVIDENCE_DIR=/absolute/path/to/new-empty/ch20-run
export PG36_CH20_TARGET=pg36-l2-vagrant/pg-test
export PG36_CH20_NONPRODUCTION=true
export PG36_CH20_PRODUCTION_DATA=false
export PG36_CH20_PRODUCTION_TRAFFIC=false
export PG36_CH20_CONFIRM=SWITCH_CH20_PG_TEST_1_TO_2_AND_BACK

static/labs/ch20/task.sh drill:switchover
```

所有 guard 都必须 exact match。output 已非空则拒绝覆盖。

### forward completion

动作完成定义不是 CLI exit：

```text
Patroni member set exact
pg-test-2 sole primary/running
pg-test-1 and pg-test-3 replica/streaming
```

然后 capture `after-forward`，等待 probe 完成并 reconcile token。

### token reconciliation

```sql
SELECT attempt_no, token, committed_at
FROM pg36_ch20.write_probe
WHERE run_id = $1
ORDER BY attempt_no;
```

分类：

```text
acknowledged_missing
unknown_committed
unknown_absent
duplicate_tokens
unreconciled_unknown
```

验收：

```text
acknowledged_missing=0
duplicate_tokens=0
unreconciled_unknown=0
persisted = acknowledged + unknown_committed
```

### 恢复教学角色基线

probe 完成后：

```bash
patronictl ... switchover pg-test \
  --leader pg-test-2 \
  --candidate pg-test-1 \
  --force
```

再等：

```text
pg-test-1 sole primary
pg-test-2/3 streaming
```

capture `restored`。这是第二次 planned switchover，不是 rewind timeline。

### 异常时的保守恢复

若 forward 后中途失败，脚本只在能确认：

```text
pg-test-2 is the sole healthy leader
```

时尝试切回 `pg-test-1`。

若 topology ambiguous：

```text
stop automatic recovery
print inspection requirement
preserve evidence
```

它不会为了“清理实验”猜 leader。

### postflight

角色恢复后重新运行第 19 章 `all`。正式 preflight 与 postflight 都通过。

证据树：

```text
ch20-run/
├── preflight-ch19/
├── drill/
│   ├── phases/
│   │   ├── before.json
│   │   ├── pre-switch.json
│   │   ├── after-forward.json
│   │   └── restored.json
│   ├── forward-action.json
│   ├── restore-action.json
│   ├── client-events.jsonl
│   ├── client-probe.stderr
│   ├── reconciliation.json
│   ├── drill-manifest.json
│   ├── validation-report.json
│   └── negative-report.json
├── postflight-ch19/
└── review.txt
```

目录应放在 private evidence 存储，不进入 Git。

### read-only 重验

```bash
export PG36_EVIDENCE_DIR=/absolute/path/to/ch20-run
static/labs/ch20/task.sh all
```

输出应含：

```text
status=ok
counterexamples=10-rejected
acknowledged_commits=all-present
sandbox_planned_switchover=accepted-with-exceptions
unplanned_failure_drill=not-run
production_ch20_gate=pending
mutation=none
```

`all` 不移动 leader。

### fixture reset

`reset:fixture` 会 `DROP SCHEMA pg36_ch20 CASCADE`，是独立 destructive
action，需要完整 reviewed evidence、drained clients 和另一个 exact token。

正式实验**没有执行 reset**。保留 fixture 供后续查证没有问题；若确实清理，
先读合同，不要把它和 cluster reset 混淆。

## 20.7.2 测量实际 RTO、提交风险与恢复时间 {#item-20-7-2}

### 先纠正这个标题

本次没有注入 failure，所以没有测量 failure-time “实际 RTO”。它测到的是：

```text
planned action command duration
planned action -> Patroni stable duration
sampled client write gap
baseline restore duration
```

把这些都标成 RTO，会把检测、租约、fencing 和故障信息缺失从模型中删掉。

### 四个 monotonic 时钟

```text
A  forward action starts
B  patronictl returns
C  Patroni topology becomes stable
D  first acknowledged client write after C
```

指标：

\[
command = B-A
\]

\[
action\_to\_stable = C-A
\]

\[
conservative\ write\ gap
=D-lastAckBefore(A)
\]

probe 使用 monotonic clock，避免 wall clock adjustment 影响 duration。UTC
timestamp 只用于人类关联日志。

### 正式数据

```text
formal wrapper start        2026-07-29T19:39:33Z
forward start               2026-07-29T19:39:46Z
forward command             2734.741 ms
forward stable              2026-07-29T19:39:52Z
action to stable            5822.678 ms
conservative write gap      6007.080 ms
maximum adjacent ack gap    5207.690 ms
restore command             2851.349 ms
restore action to stable    6018.921 ms
wrapper end                 2026-07-29T19:40:17Z
wrapper elapsed             44 s
```

wrapper elapsed 还包括：

```text
chapter-19 preflight/postflight
four phase captures
24-second client probe
positive/negative validation
review
```

不能和 service interruption 比较。

### 为什么保守 gap 大于 command

```text
command exit       2.735s
topology stable    5.823s
client gap         6.007s
```

差额来自：

```text
Patroni role convergence
old member rejoin
HAProxy health cycle
connection/pool recovery
probe sampling interval
```

这正说明用 CLI 耗时报告 RTO 会偏乐观。

### probe resolution

observed interval：

```text
0.206990 s
```

采样 gap 至少包含一个 probe interval 的不确定性。若业务 QPS、driver
backoff、transaction time 不同，观测会变。

生产 measurement 应：

- 从真实外部 client vantage；
- 按 transaction class；
- 报告 distribution；
- 记录 retry/backoff；
- 分开 read/write；
- 包含 detection；
- 关联 topology/fence；
- 说明 measurement resolution。

### 提交结果

```text
attempts                 120
acknowledged              95
unknown                   25
persisted rows            95
acknowledged missing       0
unknown committed          0
unknown absent            25
duplicate tokens           0
unreconciled unknown       0
```

身份等式：

\[
events=acknowledged+unknown=95+25=120
\]

\[
persisted=acknowledged+unknownCommitted=95+0=95
\]

所有 token list length 也与 count 交叉验证。

### unknown 全 absent 代表什么

本次 transition window 中的 25 个异常尝试，在最终 chosen history 查不到。
它们可能在连接建立前失败，也可能发送但未提交；对应用而言，先统一归为
unknown，再以 token lookup 得到 absent。

它不意味着未来 unknown 都 absent。系统必须支持：

```text
unknown committed -> return existing result, do not duplicate
unknown absent    -> apply business retry policy
```

### acknowledged 全在代表什么

可以严格说：

```text
all 95 writes that returned success in this run exist after forward
switchover and baseline restore
```

不能说：

```text
the asynchronous cluster has zero RPO
```

因为 healthy switchover 会协调 caught-up candidate；没有突然永久丢失
primary WAL tail。

### timeline token

acknowledged event 从 WAL 文件名记录：

```text
00000005
00000006
```

证明 client writes 横跨 forward transition 的两个 timeline。restore 在
probe 结束后执行，所以 client probe 不要求看到 7；restored phase 的 SQL
与 Patroni 证明 timeline 7。

### objective

事前合同：

```text
minimum acknowledged attempts = 30
maximum conservative gap      = 15000 ms
missing acknowledged          = 0
duplicates                    = 0
unreconciled unknown          = 0
```

正式通过：

```text
95 >= 30
6007.080 <= 15000
0 / 0 / 0
```

这个 15 秒目标是教学 sandbox acceptance，不是生产 SLO。

### 建立真正 RTO 需要什么

另行授权的 unplanned drill 要定义：

```text
fault injection time
failure domain
old-primary fence evidence
lease/detection
candidate eligibility
new authority
service route
client transaction
backlog recovery
```

并重复足够次数，报告 percentile 和失败 run。第 33 章再做，不在这里偷换。

## 20.7.3 输出拓扑证据、时间线和改进项 {#item-20-7-3}

### topology evidence

| phase | leader | replicas | Patroni timeline | primary WAL timeline |
|---|---|---|---:|---:|
| before | pg-test-1 | pg-test-2/3 | 5 | 5 |
| pre-switch | pg-test-1 | pg-test-2/3 | 5 | 5 |
| after-forward | pg-test-2 | pg-test-1/3 | 6 | 6 |
| restored | pg-test-1 | pg-test-2/3 | 7 | 7 |

所有 SQL phase 的 system identifier 相同；公开 summary 只记录
`one unchanged identifier`，不把机器/cluster identity 无必要地扩散。

### sender、receiver 与 slot

每个 stable primary：

```text
two async streaming senders
application name and client address exact
replay gap inside 1 MiB lab bound
two active physical slots named for nonleaders
no WAL receiver
```

每个 standby：

```text
one streaming WAL receiver
sender_host=current primary
sender_port=5432
```

这使“old primary rejoined”不是只靠 Patroni label。

### checkpoint timeline 反例

正式结束：

```text
cluster current timeline                7
pg-test-3 checkpoint_timeline_id        3
pg-test-3 Patroni state                 streaming
pg-test-3 WAL receiver upstream         pg-test-1
```

validator 接受 checkpoint `3 <= 7`，不要求相等。若早期实现把
`pg_control_checkpoint().timeline_id` 命名为 `timeline_id` 并强制等于
current，实验会误报。

这是本章最值得保留的“测量修正”：发现反例后改 evidence schema，再重跑
正式实验，而不是改解释去迁就旧字段。

### manifest

```text
schema=pg36-ch20-drill-manifest-v1
run UUID present
mode=planned-switchover-and-planned-baseline-restore
production_approval=false
unplanned_failure_injected=false
secret_values_exported=0
17 source input files hashed
```

`review.py` 对比 current source。source 改过以后，旧 run 不能冒充新实现的
证据。

### 十个反例

| case | expected rejection |
|---|---|
| sandbox 冒充生产 | `E_PRODUCTION_CLAIM` |
| 未评审 candidate | `E_ACTION` |
| 忽略 command failure | `E_ACTION` |
| 外来 system identifier | `E_LINEAGE` |
| 两个 leader | `E_TOPOLOGY` |
| timeline 未推进 | `E_TIMELINE` |
| 旧主未 rejoin | `E_TOPOLOGY` |
| acknowledged token 丢失 | `E_COMMIT_EVIDENCE` |
| unknown 未核对 | `E_COMMIT_EVIDENCE` |
| gap 超过 15 秒 | `E_WRITE_GAP` |

正式 `10/10` 按预期失败。一个 validator 若只在 happy input 上返回绿色，
还没有证明 guard 分支存在。

### accepted decision

```text
sandbox_planned_switchover=accepted-with-exceptions
unplanned_failure_drill=not-run
production_ch20_gate=pending
```

十个 exception ID 与第 19 章/本章合同 exact match。

### 不通过的生产问题

```text
shared hypervisor
single etcd
single backup target
virtual storage unqualified
temporary inventory secret lifecycle
three guests below recommended resource floor
async replication
watchdog off
external probe path without TLS
planned-only drill
```

### 改进项按 gate 排序

#### HA production gate

```text
independent failure-domain placement
production DCS quorum
tested fencing/watchdog
approved sync/degrade policy
unplanned process/host/network/DCS drills
repeatable RTO/RPO distribution
manual failover runbook and authority
```

#### Recovery gate

```text
independent repository
archive continuity
restore + PITR
timeline history
data/business validation
```

第 21 章负责。

#### Client gate

```text
DNS/VIP/HAProxy path
pooling behavior
driver timeout/backoff
session reset
idempotency/reconciliation
read consistency and replica fallback
```

第 22 章负责。

#### Security gate

```text
TLS verify-full or equivalent
credential authority/rotation
least privilege
host key trust
audit
```

第 23 章负责。

### ADR

[`ha-adr.md`](/labs/ch20/ha-adr.md) 记录：

```text
accepted: planned switchover behavior
rejected: automatic failover claim
rejected: command duration as RTO
rejected: no missing tokens as zero RPO
rejected: checkpoint timeline as current standby timeline
rejected: all reruns live mutation
```

这让将来升级时能判断是事实变化，还是 decision boundary 被悄悄扩大。

### 正式运行摘要

[`drill-run.json`](/labs/ch20/drill-run.json) 是 secret-free reference
account：

```text
versions
times
topology relation
client counts/metrics
gates
exceptions/decision
```

它不是完整 evidence 的替代品；完整 bundle 留在 private path。

## 20.7.4 记录把本章迁移到新版本基线的工时 {#item-20-7-4}

### 为什么记录 effort

版本迁移成本不只有：

```text
change version string
```

还包括：

```text
contract review
CLI/API drift
config schema drift
evidence query drift
lab provisioning
safe live run
counterexample maintenance
prose/source verification
```

没有记录，就无法判断下一版是“十分钟 bump”还是“重新认证 HA 行为”。

### 不虚构 human effort

本次可精确测得：

```text
formal wrapper machine elapsed = 44 seconds
```

不能从 wall-clock timestamp 反推出：

```text
human authoring seconds
operator active attention
review effort
```

因此
[`migration-effort.json`](/labs/ch20/migration-effort.json) 明确：

```json
{
  "formal_machine_elapsed_seconds": 44,
  "human_authoring_seconds": null,
  "human_authoring_measurement": "not-instrumented-do-not-infer"
}
```

`null` 比一个看似精确但捏造的数字更专业。

### effort 维度

下次迁移记录：

| category | 例 | measurement |
|---|---|---|
| research | release notes、docs、known issues | active human time |
| source adaptation | API/query/config change | diff + active time |
| environment | build/deploy/converge | machine + operator |
| validation | capture/positive/negative | machine time |
| live drill | guarded transition | exact start/end |
| diagnosis | failed run/root cause | active + elapsed |
| writing | prose/citations/diagrams | active time if instrumented |
| review | technical/safety/editorial | reviewer time |

active 与 elapsed 分开：

```text
machine waits 30 minutes
operator active 3 minutes
```

二者都可能影响排期，但含义不同。

### 新版本迁移顺序

1. 固定目标 release/commit，不用 moving branch；
2. 阅读 PostgreSQL、Patroni、Pigsty release notes；
3. diff inventory、Patroni dynamic/local config、service definitions；
4. 检查 SQL catalog/function 字段变化；
5. 更新 `requirements.json`，不要先改 validator 放宽；
6. 更新 capture schema 与 source allowlist；
7. 用 synthetic evidence 跑 normal/negative；
8. 在新 disposable target 重做第 19 章；
9. 做只读 chapter-20 capture；
10. 获得 L2 authority 后跑 planned drill；
11. 保留失败 candidate run，不覆盖；
12. 更新 reference outcome 与正文；
13. 独立 review；
14. 只在证据支持时改变 decision。

### 兼容性问题清单

```text
PostgreSQL:
  server_version_num, catalog/view/function fields
  WAL/timeline semantics
  pg_rewind prerequisites
  sync commit behavior

Patroni:
  CLI flags/output
  dynamic config
  sync/failsafe/watchdog semantics
  REST health endpoints

Pigsty:
  inventory schema
  default service ports/selectors
  wrapper commands
  monitoring labels
  component versions

client:
  libpq/psycopg behavior
  TLS/service file
  target_session_attrs
```

### baseline result不能机械复制

新版本即使同样输出：

```text
status=ok
```

也要检查：

```text
validator 是否仍在验证同一事实
字段是否变成 null/新语义
默认配置是否改变
错误分支是否仍拒绝
exception 是否增加/消失
```

升级最危险的不是 test fail，而是旧 test 在新语义下无意义地继续 pass。

### 版本迁移 stop conditions

```text
source release not exact
private inventory leaks
unknown config migration
old/new evidence schema mixed
member identity/topology ambiguous
client path differs without contract update
reset/reinit needed but not separately authorized
production target substituted for sandbox
```

遇到即停止，不用“先跑一次看看”跨过 authority。

### 当前 effort 记录的范围

正式文件只承诺：

```text
work stream start timestamp recorded
formal wrapper start/end recorded
machine elapsed=44s
human authoring not instrumented
scope note prevents inference
```

这为以后建立更完整度量留下可比较 schema，同时不美化本次过程。

## 本章最终复核

```text
[x] failure model before operation
[x] exact nonproduction target
[x] chapter-19 pre/postflight
[x] private credential never exported
[x] named leader and candidate
[x] client probe through service
[x] one system identifier
[x] timeline 5 -> 6 -> 7
[x] old primary rejoined
[x] acknowledged/unknown reconciled
[x] ten negative cases rejected
[x] temporary service file removed
[x] production gate remains pending
[ ] unplanned failure
[ ] fencing/watchdog
[ ] DCS quorum failure
[ ] zero RPO
[ ] production RTO/SLO
```

## 小结

本次实验真正证明的是：

> 在 exact Pigsty v4.5.0 / PostgreSQL 18.6 / Patroni 4.1.3 本地沙箱、
> 异步复制、单 etcd、无 watchdog 的记录条件下，一个健康且明确命名的
> candidate 完成计划切换，旧主重新加入，客户端在约 6 秒的采样间隙后
> 恢复写入；95 个已确认 token 全部存在，25 个结果未知 token 全部核对
> 为未提交，最终角色基线恢复。

它明确没有证明：

> 自动故障转移、旧主硬隔离、DCS quorum、零 RPO、生产 RTO、灾难恢复或
> 生产安全。

把两段话一起交付，才是完整的 HA evidence。

## 权威参考

- [Patroni：`patronictl switchover`](https://patroni.readthedocs.io/en/latest/patronictl.html)
- [Pigsty：`pig pt switchover`](https://pigsty.io/docs/pig/pt/)
- [Pigsty：Service/Access](https://pigsty.io/docs/pgsql/service/)
- [PostgreSQL 18：Warm Standby](https://www.postgresql.org/docs/18/warm-standby.html)
- [正式实验合同](/labs/ch20/lab-contract.md)
- [正式运行摘要](/labs/ch20/drill-run.json)
- [迁移 effort 边界](/labs/ch20/migration-effort.json)

---

[上一节：交付并观察 HA 集群](../06/) · [返回本章目录](../) · [下一章：未雨绸缪：备份体系与恢复演练](/backup-recovery/) ·
[查看全书目录](/toc/) · [查看索引中心](/indexes/)