# 实战：实现并演练观察契约

LLMS 索引： [llms.txt](/llms.txt)

---

本节把全章压成一条可重放的证据链：

```text
bind ch24 observation contract
  -> validate signal and diagnostic-pack semantics
      -> compile 18 recording rules
          -> compile 13 alert rules
              -> preserve 7 accepted policies
                  -> quarantine 6 proposed policies
                      -> read live Pigsty/PostgreSQL baseline
                          -> verify target identity
                              -> run isolated VMAlert tests
                                  -> run offline Alertmanager routes
                                      -> test inhibition positive/negative
                                          -> reject 25 counterexamples
                                              -> publish 14-row coverage matrix
                                                  -> keep production gate pending
```

它刻意把两种动作分开：

```text
online baseline
  L0, read-only, no mutation

isolated exercise
  L1-ephemeral, temporary loopback processes and files
  no online rule, no online alert, no real receiver
```

实验合同：

- [边界与运行方式](/labs/ch25/lab-contract.md)
- [要求](/labs/ch25/requirements.json)
- [信号合同](/labs/ch25/signal-contract.json)
- [拓扑](/labs/ch25/topology.mmd)
- [公共运行摘要](/labs/ch25/observability-run.json)

## 25.7.1 为延迟、错误、复制、备份和资源建立规则 {#item-25-7-1}

### 文件布局

```text
static/labs/ch25/
├── requirements.json
├── signal-contract.json
├── coverage-matrix.json
├── diagnostic-pack-contract.json
├── recording-rules.yml
├── alert-rules.yml
├── rule-tests.yml
├── alertmanager-sandbox.yml
├── route-tests.json
├── negative-cases.json
├── topology.mmd
├── lab-contract.md
├── capture.py
├── exercise.py
├── validate.py
├── review.py
├── task.sh
└── observability-run.json
```

职责：

| 文件 | 证明什么 |
|---|---|
| requirements | target、risk、上游、hard rejection、gate |
| signal contract | source/type/label/reset/missing/cost |
| recording rules | SLI window 与诊断聚合 |
| alert rules | accepted/proposed policy |
| rule tests | synthetic arithmetic、pending、firing、recovery |
| AM config | empty sink route 和 inhibition |
| route tests | 八个 route、五个 inhibition 正反例 |
| diagnostic pack | 自动证据字段、limit、权限 |
| coverage | 已有信号、预期缺口、fallback |
| negative cases | validator 不是只会通过 |
| capture | live L0 baseline |
| exercise | remote `/tmp` isolated test |
| review | file mode、secret、claim、count |
| public summary | allowlist 结果，不含 private evidence |

### 上游 binding

第 24 章四份输入按 hash 绑定：

```text
observation-contract.json
alert-candidates.json
slo-policy.json
governance-run.json
```

并要求：

```text
run_id              34909737-527a-460c-927c-d9d71c93aa13
production_ch24_gate pending
```

如果上游改了 target、alert policy 或 missing semantics，本章 capture 不能继续
假装基于旧合同。

### 记录规则：可用性

五个窗口：

```text
rate5m
rate30m
rate1h
rate6h
rate3d
```

表达式：

```promql
sum by (service, operation_class, environment) (
  rate(pg36_shop_request_outcomes_total{
    eligible="true",
    outcome!="good"
  }[5m])
)
/
sum by (service, operation_class, environment) (
  rate(pg36_shop_request_outcomes_total{
    eligible="true"
  }[5m])
)
```

注意三个设计：

1. 分子是 bad eligible event，不是 instance；
2. 分母没有任意 `clamp_min(...,1)` 改变低流量比率；
3. 缺失由独立 freshness/metamonitoring 处理，不补健康零。

instrumentation 必须预初始化 outcome category，或者用能区分“零 bad”与“bad
series 未产生”的设计。

### 记录规则：延迟

```promql
1 -
sum by (service, operation_class, environment) (
  rate(pg36_shop_request_duration_seconds_bucket{
    eligible="true",
    le="0.25"
  }[5m])
)
/
sum by (service, operation_class, environment) (
  rate(pg36_shop_request_duration_seconds_count{
    eligible="true"
  }[5m])
)
```

产生 5m/1h bad ratio。

规则可计算，但 alert governance 尚未接受：

```yaml
route: test
severity: candidate
governance_status: proposed-not-accepted
```

### 记录规则：新鲜度

```promql
sum by (service, read_path, environment) (
  rate(pg36_shop_commit_visibility_probes_total{
    eligible="true",
    within_bound="false"
  }[5m])
)
/
sum by (service, read_path, environment) (
  rate(pg36_shop_commit_visibility_probes_total{
    eligible="true"
  }[5m])
)
```

它不读取 `pg_lag`，因为用户 SLI 必须携带 commit token。

### 记录规则：PostgreSQL 诊断

```text
pg36:replica_replay_distance_bytes:max
pg36:archive_failures:increase15m
pg36:longest_transaction_seconds:max
pg36:table_freeze_age:max
pg36:dead_tuples:sum
pg36:exporter_unavailable:max
```

这些是原因/容量输入，不自动 page。

### 记录规则：metamonitoring

```text
pg36:vmalert_rule_errors:sum
pg36:vmalert_missed_iterations:increase15m
pg36:notification_failures:increase15m
```

一个重要限制：notification failure 为零仍不能证明 receiver receipt。还需要
外部 canary。

### accepted 七条

validator 逐字段与第 24 章比对：

| alert | route | `for` |
|---|---|---:|
| availability fast | page | 2m |
| availability slow | page | 5m |
| availability budget | ticket | 15m |
| freshness fast | page | 2m |
| correctness mismatch | page | 0m |
| capacity horizon | ticket | 1h |
| monitoring path broken | page | 5m |

比较：

- route；
- severity；
- objective；
- owner；
- runbook；
- first safe action；
- governance status；
- `for`；
- multiwindow token。

任意漂移使 lint 失败。

### proposed 六条

| alert | 用途 | 为什么不直接上线 |
|---|---|---|
| latency fast burn | 用户延迟 | ch24 尚未接受 page policy |
| restore evidence stale | recovery control | route/severity 待接受 |
| archive failure active | recovery risk | threshold/pgBackRest 联动待评审 |
| transaction age | maintenance/capacity | workload-specific policy |
| freeze age | wraparound horizon | 应从配置/rate 生成 |
| SLI missing | observation path | expected traffic/fallback 待实现 |

统一：

```yaml
route: test
severity: candidate
governance_status: proposed-not-accepted
```

validator 会拒绝 candidate 进入 page/ticket sink。

### archive candidate 不比较历史总数

```promql
pg36:archive_failures:increase15m > 0
and on (cls, ins, ip)
(time() - pg_archiver_finish_time) > 900
```

同时要求：

- 15 分钟出现新失败；
- 成功推进已经停滞。

仍需 runbook 复核 pgBackRest 和 restore evidence。

### transaction candidate

```promql
pg36:longest_transaction_seconds:max > 900
```

只走 test，因为：

- batch 可能合法；
- idle in transaction 与 active 不同；
- cancel 权限与 unknown outcome；
- 阈值依赖 workload；
- user impact 可能不存在。

first action 是找 owner/queryid 与安全性，不是 terminate。

### freeze candidate

```promql
pg36:table_freeze_age:max > 1000000000
```

固定值仅用于 synthetic test。生产应使用：

```text
current age
current autovacuum_freeze_max_age
table override
consumption rate
vacuum throughput
blocker
safety margin
```

### capacity accepted 规则仍需 reviewed input

```promql
pg36_shop_capacity_horizon_days < 14
and on (service, environment)
pg36_shop_capacity_forecast_reviewed == 1
```

accepted 是告警合同，未表示 forecast exporter 已实现。coverage 仍标记应用层
缺口。

### rules 与 groups

三组 recording 和四组 alert 分开。原因是 VMAlert 的 recording result remote
write 是异步的；同 group 依赖上一条结果可能读不到本轮。

25 个反例之一会把 alert 塞回 recording group，确认 validator 拒绝：

```text
recording group contains an alert and creates rule chaining
```

### 当前 live baseline

`capture.py` 只读：

```text
VictoriaMetrics   health/API/query
VictoriaLogs      health/version
VictoriaTraces    health/version
VMAlert           rules/alerts/self metrics
Alertmanager      health/alert count/self metrics
pg_exporter       raw HELP/name/label inventory
PgBouncer exporter response
Patroni           selected identity/state
PostgreSQL        bounded read-only SQL
```

不会读取 Alertmanager config，因为可能有 receiver secret；只保存 count 与
`configuration_exported=false`。

### PostgreSQL target identity

工作站配置：

```text
10.10.10.10 -> local port 2222
10.10.10.11 -> local port 2200
...
```

现场发现多个转发落到元节点。正式路径改为：

```text
ssh ProxyJump meta
  target HostName=10.10.10.11 inside sandbox
```

并验证：

```text
hostname            pg-test-1
cluster_name        pg-test
Patroni             pg-test/pg-test-1
pg_is_in_recovery   false
replication rows    2
```

这一检查会拒绝“查询成功但目标错误”。

### 实际 PG observation settings

正式快照记录：

```text
track_activities              on
track_counts                  on
track_io_timing               on
track_wal_io_timing           off
stats_fetch_consistency       cache

pg_stat_statements.track      all
track_planning                off
track_utility                 off

logging_collector             on
log_destination               csvlog
log_file_mode                 0640

auto_explain.log_min_duration 1000ms
auto_explain.log_analyze      on
auto_explain.log_timing       on
auto_explain.sample_rate      1
```

两个待评审：

```text
auto_explain overhead reviewed  false
log group access reviewed       false
```

实验不修改配置。

## 25.7.2 注入可控症状，验证触发、路由、抑制和恢复 {#item-25-7-2}

### 只做静态 lint

```bash
static/labs/ch25/task.sh lint
```

输出：

```text
recording_rules=18
alert_rules=13
accepted_alerts=7
proposed_alerts=6
counterexamples=25-rejected
live_deployment=false
production_ch25_gate=pending
```

不访问远端。

### 正式完整运行

先创建一个新的私密路径；`task.sh` 也会拒绝覆盖非空目录：

```bash
export PG36_EVIDENCE_DIR=/absolute/private/new-empty/ch25-run
static/labs/ch25/task.sh all
```

动作：

```text
capture
  L0 live read-only

exercise
  L1 remote temp / loopback only

verify
  positive + negative + source hash + live evidence

review
  private mode + secret scan + claim boundary
```

### capture 的 SQL 安全边界

```text
statement_timeout=5s
lock_timeout=500ms
default_transaction_read_only=on
BEGIN READ ONLY
```

query allowlist 不含：

```text
query text
bind
client address
EXPLAIN ANALYZE
reset
vacuum/checkpoint
cancel/terminate
DDL/DML
```

即便如此，`pg_locks` 会看到采集查询自身的 relation lock；这是 observer
effect，证据不会假装完全无影响。

### isolated workspace

`exercise.py`：

1. 在元节点执行 `mktemp -d /tmp/pg36-ch25.XXXXXXXX`；
2. 验证路径符合严格 regex；
3. 只上传四个无 secret 文件；
4. 执行测试；
5. `rm -rf` 只允许匹配该 prefix；
6. 断言目录已经不存在；
7. 输出 `remote_cleanup=ok`。

如果 temp path 不匹配，宁可失败也不清理未知目录。

### 规则语法

```text
vmalert -rule=recording-rules.yml
        -rule=alert-rules.yml
        -dryRun
```

证明 parser/规则结构可接受，不执行 live query。

### 合成时序

`vmalert-tool`：

```text
isolated VictoriaMetrics
simulated periodic ingestion
recording/alert evaluation
expected sample/alert comparison
```

availability recording 输入：

```text
good +100/min
bad  +2/min
```

验证结果：

$$
\frac{2}{102}=0.019607843...
$$

### fast pending/firing/recovery

输入：

```text
1h bad ratio  0.02 for 4 samples, then 0
5m bad ratio  0.02 for 4 samples, then 0
evaluation    1m
for           2m
```

期望：

```text
1m  no firing alert
3m  exact alert labels/annotations
7m  no alert
```

这验证 expression、`for`、label identity 和 recovery。

### 其他合成用例

```text
availability slow burn
availability budget ticket
freshness fast burn
correctness immediate
reviewed capacity ticket
monitoring path page
latency proposed route
expected traffic but SLI missing
```

correctness `for=0m`，capacity 需要 reviewed gauge，missing 不补零。

### Alertmanager 配置检查

```text
amtool check-config
```

发现：

```text
global config
route
2 inhibition rules
7 receivers
0 templates
```

所有 receiver 只有 `name`，没有 integration。

### 八个 route

| 用例 | 期望 sink |
|---|---|
| availability page | `pg36-shop-page-sink` |
| correctness page | `pg36-shop-page-sink` |
| service ticket | `pg36-shop-ticket-sink` |
| platform ticket | `pg36-platform-ticket-sink` |
| metamonitoring page | `pg36-platform-page-sink` |
| diagnostic | `pg36-null-sink` |
| proposed | `pg36-proposed-rule-sink` |
| unknown | `pg36-default-sink` |

使用：

```text
amtool config routes test --config.file=...
  --verify.receivers=<expected>
  <labels...>
```

离线解析，不连接 live Alertmanager。

### 五个 inhibition

| source → target | 结果 |
|---|---:|
| fast → same service/objective slow | inhibit |
| meta → derived missing | inhibit |
| meta → independent user symptom | do not |
| meta → correctness | do not |
| fast → other service slow | do not |

validator 同时解析 config 和测试表；添加 broad correctness inhibition 会被
反例拒绝。

### 25 个对抗性变体

类别：

```text
scope
  production data/approval promoted

online mutation
  live deploy/notification/Alertmanager

semantics
  missing healthy, timing disabled = zero

privacy
  tenant label, query text, public evidence

claims
  app metric/rules/pager claimed live

query safety
  EXPLAIN ANALYZE

alert governance
  accepted route/for drift, candidate page

time
  slow rule loses second window

component semantics
  archive historical counter

engine
  alert chained in recording group

routing
  real webhook, correctness inhibition, real receiver name
```

validator 不是检查“文件存在”，而是逐个 mutation 后要求出现预期 rejection。

### 正式运行结果

```text
run_id      b876731e-5741-40e3-a03e-17cf7d20881b
captured    2026-07-29T22:48:21Z
target      pg36-l2-vagrant/pg-test
live risk   L0
exercise    L1-ephemeral
```

组件：

```text
Pigsty            v4.5.0
PostgreSQL        18.6
pg_exporter       v1.4.0
VictoriaMetrics   v1.148.0
VictoriaLogs      v1.52.0
VictoriaTraces    v0.9.4
Alertmanager      0.33.1
```

在线：

```text
VM series                  44,842
label-value pairs          387,724
VMAlert groups             17
live alert/recording       50 / 698
live rule errors           0
live current alerts        0
application SLI series     0
```

PostgreSQL：

```text
primary                 pg-test-1
async replicas          2
observed WAL gap        0 / 0 bytes
pg_stat_statements rows 194
pgss calls              122,303
query text exported     false
archive failed_count    21
last success after fail true
```

演练：

```text
18 recording rules       pass
13 alert rules           pass
8 routes                 pass
5 inhibition cases       pass
25 counterexamples       rejected
remote cleanup           pass
secret scan              pass
```

结论：

```text
live chapter deployment  false
real receiver            false
production_ch25_gate     pending
```

### evidence

private bundle：

```text
observability-evidence.json
isolated-exercise.txt
validation-report.json
negative-report.json
review.txt
```

全部：

```text
directory 0700
files     0600
```

公共摘要只包含 allowlist，不包含 source body、SQL text、log body 或 receiver
config。

### 重验

```bash
export PG36_EVIDENCE_DIR=/absolute/private/existing/ch25-run
static/labs/ch25/task.sh verify
static/labs/ch25/task.sh review
```

如果 source file 在 capture 后改变，hash 校验失败。必须新跑，而不是手工改
report。

### 没有 reset action

本章：

- 在线没有 mutation；
- remote temp 自动清理；
- isolated process 退出；
- 没有 database fixture；
- 没有 live rule。

所以没有数据库 reset。不要添加一个“为了形式完整”的 destructive reset。

## 25.7.3 输出告警覆盖表、盲区与 ch31 的诊断入口 {#item-25-7-3}

### 14 行覆盖矩阵

[`coverage-matrix.json`](/labs/ch25/coverage-matrix.json) 包含：

```text
5 user/control signals
6 PostgreSQL/platform observation areas
logs
traces
routing
```

每行：

```text
question
source
live expected/status
rule status
fallback
owner
blind spot
```

### 五个应用缺口

| coverage | live | fallback |
|---|---:|---|
| availability event | absent | independent place-order probe |
| latency histogram | absent | end-to-end duration probe |
| commit freshness | absent | unique-token probe |
| correctness reconciliation | absent | signed reconciliation |
| restore evidence age | absent | immutable restore manifest |

它们是 `expected-gap`，因为教学沙箱没有对应应用。

不得由：

```text
pg_up
HAProxy up
pg_lag
backup success
zero DB errors
```

代填。

### 已有 PostgreSQL/platform 覆盖

```text
exporter reachability
activity/transaction/wait/lock
I/O/WAL/checkpoint/archive/replication
maintenance/freeze/object
pg_stat_statements aggregate/reset
VMAlert/Alertmanager self monitoring
VictoriaLogs endpoint
VictoriaTraces endpoint
```

“已有”仍带盲区：

- one sample misses transient wait；
- PG I/O 不区分 OS cache/device；
- dead tuple 不是 exact bloat；
- queryid collision；
- endpoint 不证明 body/parse；
- backend 不证明 application spans；
- notification failure 0 不证明 receipt。

### 配置观察到的待评审项

#### `auto_explain`

```text
log_analyze   on
log_timing    on
sample_rate   1
threshold     1s
nested        on
parameter max -1
```

生产前：

- 代表性 load test；
- short statement overhead；
- log volume；
- parameter leakage；
- `log_timing=off` 方案；
- sampling；
- rollback；
- owner approval。

本章不直接说“必须关闭”，也不说“当前安全”。

#### log `0640`

可能为 Vector collector group 提供 read。生产前：

- actual owner/group；
- group membership；
- directory mode；
- rotation；
- central storage ACL；
- retention；
- redaction；
- access audit。

### alert coverage 不等于 implementation

七条 accepted alert 有规则，但 app source 不存在：

```text
policy coverage   yes
synthetic test    yes
live data         no
live deploy       no
real route        no
```

因此不能说“availability monitoring complete”。

### proposed queue

需要治理拍板：

1. latency page route/severity；
2. restore evidence stale 的 change gate；
3. archive risk 的 threshold 与 pgBackRest 证据；
4. long transaction workload class；
5. freeze horizon based on config/rate；
6. expected traffic/missing source；
7. real receipt canary。

每项完成后：

- 更新 ch24 governance；
- hash binding 失效；
- 重跑 ch25；
- canary deploy；
- 验证 route/receipt；
- 保留 rollback。

### 生产上线前的门禁

```text
service instrumentation
  event schema/version/cardinality

source validation
  producer/zero/missing/reset

rule validation
  synthetic + shadow/live query

governance
  owner/runbook/first action/severity

routing
  real receiver secret outside Git

delivery
  receipt canary

cost
  exporter/query/storage/log overhead

privacy
  labels/body/parameters/retention/access

deployment
  staged canary, rollback, change event

operations
  drill and evidence
```

完成之前：

```text
production_ch25_gate=pending
```

### ch31 诊断入口

本章输出按 symptom 路由给第 31 章：

#### 慢查询

```text
user latency windows
entry/pool wait
activity wait
pg_stat_statements reset + top queryid
I/O/temp/WAL
release/plan event
host evidence
```

#### 锁与死锁

```text
user outcome
current blocker graph
lock wait/deadlock log aggregate
transaction age/xmin
application owner
safe cancellation boundary
```

#### 连接与池

```text
edge queue
HAProxy/PgBouncer client/server/wait
PG active/idle/idle-in-xact
connection config change
max connection headroom
```

#### 复制与新鲜度

```text
commit token probe
read path
Patroni role/timeline
sender/receiver position
slot/WAL retention
route change
```

#### 磁盘与容量

```text
filesystem horizon
PG I/O bytes/time
device latency/queue
WAL/archive
object growth
rewrite/backup headroom
```

#### vacuum 与冻结

```text
old xact/xmin
table estimates
progress
freeze age/config/rate
slot/feedback
host I/O
```

### 诊断入口的统一字段

```text
run id
captured at
service/environment
target identity
symptom windows/count
topology
changes
PostgreSQL reset and aggregates
platform freshness
known gaps
hypotheses/falsifiers
risk/authority
```

第 31 章不需要从“打开所有 dashboard”重新开始。

### 完成标准

本章不是以“31 条规则存在”完成，而是：

```text
semantics are explicit
accepted and proposed are separated
live and isolated are separated
missing is not healthy
rules are tested
routes are tested without delivery
inhibition has negative tests
native PG evidence is bounded
privacy and cost are explicit
blind spots are published
production gate remains honest
```

### 本节验收

你应当能复述：

1. 18 条 recording rule 分哪三组；
2. 13 条 alert 中哪七条 accepted、哪六条 proposed；
3. archive rule 为什么同时看增量与 last success；
4. target identity 为什么需要 ProxyJump 后三源校验；
5. capture 与 exercise 的风险/权限边界；
6. synthetic time series 如何验证 pending/firing/recovery；
7. route test 为什么使用空 receiver；
8. inhibition 为什么必须有 must-not-inhibit；
9. 25 个反例覆盖哪些失败模式；
10. private evidence 与 public summary 如何分离；
11. 五个 application gap 为什么不能用 PG component metric 替代；
12. `auto_explain` 与 `0640` 为什么是待评审而非自动修复；
13. 生产门禁还缺哪些步骤；
14. 第 31 章如何使用本章诊断包。

---

[上一节：从告警到诊断包](../06/) · [返回本章目录](../) · [下一章：胸有成竹：容量规划与压测基线](/capacity-benchmarking/) ·
[查看全书目录](/toc/) · [查看索引中心](/indexes/)