# 实战：把 `pg36_shop` 纳入服务治理

LLMS 索引： [llms.txt](/llms.txt)

---

本节把全章压成一条可重放的 L0 证明：

```text
machine-check service card
  -> calculate SLO and control objectives
      -> bind observation sources and missing semantics
          -> accept actionable alerts
              -> reject actionless page
                  -> bind four SOPs to upstream drills
                      -> enforce high-risk authority
                          -> enforce evidence retention/redaction
                              -> rerun current ch19 deployment gate
                                  -> hash-bind ch20–ch23 summaries
                                      -> reject 20 adversarial mutations
```

实验不会：

- 创建或修改 PostgreSQL 对象；
- 改 PgBouncer/HAProxy；
- 切换 leader；
- 运行 restore；
- 修改 role/credential/HBA/TLS；
- 部署 VMAlert rule；
- 发送真实 page；
- 删除或重置数据。

`all` 在本章表示“采集只读 gate 并验证全部合同”，不是“执行所有被引用的
高风险演练”。

实验合同：

- [实验权限与边界](/labs/ch24/lab-contract.md)
- [要求](/labs/ch24/requirements.json)
- [治理 ADR](/labs/ch24/governance-adr.md)
- [依赖图](/labs/ch24/dependency-map.mmd)

## 24.6.1 发布服务卡、SLO、责任人与升级路径 {#item-24-6-1}

### 目标环境

```text
service             pg36_shop
target              pg36-l2-vagrant/pg-test
environment         l2-sandbox
data                synthetic teaching data
production traffic  false
Pigsty              v4.5.0
PostgreSQL major    18
host count          4
PG member count     4 across pg-meta and pg-test
```

服务卡明确：

```text
catalog_status       teaching-reference
production_tier      false
production_slo       false
```

因此 validator 会拒绝任何把它改成 production SLO 的 mutation。

### 文件布局

```text
static/labs/ch24/
├── requirements.json
├── service-card.json
├── slo-policy.json
├── observation-contract.json
├── alert-candidates.json
├── sop-catalog.json
├── change-policy.json
├── evidence-retention.json
├── governance-adr.md
├── dependency-map.mmd
├── lab-contract.md
├── negative-cases.json
├── build_evidence.py
├── validate.py
├── review.py
├── task.sh
└── governance-run.json
```

核心合同八份，其他文件负责解释、验证和保存公开摘要。

### 服务卡

[`service-card.json`](/labs/ch24/service-card.json) 记录：

```text
customer journeys   place-order, read-order
owner functions     service, data, platform, security/privacy
dependencies        6
health layers       6
escalation           SEV-1, SEV-2, ticket
known gaps          6
```

四个 owner：

| function | role id | accountable |
|---|---|---|
| service | `shop-service-owner` | journey、SLO、release、业务决定 |
| data | `shop-data-owner` | 语义、保留、质量、隐私分类 |
| platform | `database-platform-owner` | PG/Pigsty、容量、恢复、执行 |
| security/privacy | `security-privacy-duty` | access、例外、披露、隐私 |

这些是 role contracts，没有写个人名字。`route://...` 也只是教学 route id，
不声称有真实 pager。

### 依赖图

```text
user
  -> pg36_shop application boundary
      -> HAProxy / PgBouncer
          -> PostgreSQL pg-test
              -> Patroni / etcd
              -> pgBackRest / WAL repository

all observable through
  VictoriaMetrics / VictoriaLogs / VMAlert / Alertmanager / Grafana
```

Mermaid 源文件：
[`dependency-map.mmd`](/labs/ch24/dependency-map.mmd)。

每项依赖带 failure semantics。例如：

```text
victoria-observability fails
  -> service state becomes unknown
  -> do not interpret missing telemetry as healthy
```

### 六层健康

```text
process
endpoint
authentication
transaction
correctness
durability_and_recovery
```

所有层的 `sufficient_for_service_health=false`，服务卡另有：

```json
"service_health_requires_all_layers_and_user_contract": true
```

validator 会把第一层改成 `true`，确认“process alive 即服务健康”被拒绝。

### SLO policy

[`slo-policy.json`](/labs/ch24/slo-policy.json) 定义：

| ID | kind | target |
|---|---|---:|
| `SLO-AVAILABILITY` | ratio | 99.9% / rolling 28d |
| `SLO-LATENCY` | ratio | 99% under 250 ms / rolling 28d |
| `SLO-FRESHNESS` | ratio | 99% under 5 s / rolling 28d |
| `CTRL-CORRECTNESS` | control | zero unexplained mismatch |
| `CTRL-RESTORE-READINESS` | control | passing isolated restore ≤ 90d |

算术验证：

```text
window seconds              2,419,200
availability target         0.999
sample eligible events      10,000,000
sample error budget events  10,000
equivalent time budget      2,419.2 s / 40.32 min
```

Python validator 使用容差重新计算，而不是信任 JSON 中的答案。

### exclusion

三个默认排除：

- admission 前的语法错误；
- 正确拒绝的未授权身份；
- admission 前客户端断开且没有 server outcome。

明确：

```text
planned_maintenance_excluded  false
retroactive_exclusion_allowed false
```

一个新例外必须记录 id、时间、operation、independent approver、reason 和
before/after count。

### error-budget state

```text
healthy      > 50%
watch        > 25% and <= 50%
constrained  > 0%  and <= 25%
exhausted    <= 0%
```

每个状态都有 action。validator 把 exhausted actions 清空，确认“没有后果的
预算”被拒绝。

### 只做合同 lint

不连接任何 sandbox：

```bash
static/labs/ch24/task.sh lint
```

输出：

```text
status=validation-ok
schema=pg36-ch24-validation-report-v1
production_ch24_gate=pending
status=validation-ok
schema=pg36-ch24-negative-report-v1
counterexamples=20-rejected
production_ch24_gate=pending
status=lint-ok
counterexamples=20-rejected
mutation=none
production_ch24_gate=pending
```

`lint` 使用私密临时目录保存报告，并在结束时删除；不产生 repository 文件。

## 24.6.2 为备份、切换、权限和发布建立 SOP {#item-24-6-2}

### 四份 SOP

[`sop-catalog.json`](/labs/ch24/sop-catalog.json)：

| ID | capability | risk | owner / approver |
|---|---|---:|---|
| `SOP-BACKUP-RESTORE` | backup and restore | L3 | platform / data |
| `SOP-ROLE-CHANGE` | switch/failover | L3 | platform / service |
| `SOP-ACCESS-ROTATION` | access/credential | L2 | security / platform |
| `SOP-SCHEMA-RELEASE` | schema release | L2 | service / platform |

owner 与 approver 不同。每份包含：

- trigger；
- exact target；
- blast radius；
- prerequisites；
- procedure phases；
- stop conditions；
- verification；
- rollback/roll-forward；
- evidence；
- review interval。

### 恢复 SOP

输入：

```text
source cluster
backup repository
recovery objective
target time/name
isolated destination
application invariants
space/WAL/authority
```

停止：

- identity/target 模糊；
- destination 可接 production traffic；
- WAL/integrity 失败；
- 验收未事先固定。

通过：

- target marker 存在；
- post-target marker 不存在；
- database identity 正确；
- recovered server 已停止；
- timing 只称 sandbox observation。

validator 会把 verification 替换为：

```text
backup command exited zero
```

并拒绝。

### role change SOP

先区分：

```text
planned switchover
unplanned failover
```

共同停止线：

- 多写主可能；
- DCS/fencing unknown；
- candidate 不符合 durability policy；
- unknown writes 不能核对。

验证：

```text
one leader
timeline progression
acknowledged rows present
unknown classified
pool routes current leader
```

rollback 文案刻意写：

```text
prefer safe roll-forward to one fenced leader;
a planned return is a new change
```

timeline 已推进后，不能把“切回”当撤销历史。

### access rotation SOP

步骤：

```text
new secret version out-of-band
  -> new authentication test
      -> bounded client/pool migration
          -> old new-session auth rejected
              -> revoke login/credential
                  -> explicit existing-session decision
                      -> verify final role and secret-free evidence
```

若 suspected disclosure，不能 rollback 到旧 secret，只能再次向前轮换。

### schema release SOP

```text
expand schema
  -> old/new app compatible
      -> bounded idempotent backfill
          -> constraint/reconciliation
              -> feature switch
                  -> contract old representation later
```

停止：

- lock 超时；
- WAL/lag/disk/budget 越界；
- old app 提前不兼容；
- backfill 不可恢复或不幂等。

### 上游实验绑定

三份 SOP 引用第 20、21、23 章；第 22 章作为入口/池证据也在本章 upstream
manifest 中绑定。

| 章 | run id | SHA-256 | production |
|---|---|---|---|
| ch20 | `475e9b47-bc35-4687-87da-012f1d5ea455` | `602e932b...9d9a5d` | pending |
| ch21 | `run_20260729T201040Z_961665aa` | `c0b3589d...a212ca` | pending |
| ch22 | `87a63891-d6bf-46b3-bb65-d70a8d7bac3a` | `444a6521...e6a2` | pending |
| ch23 | `64b857a6-8d8f-46e2-9462-3f097a95a69f` | `e1548658...cb59` | pending |

完整 digest 见
[`governance-run.json`](/labs/ch24/governance-run.json)。

绑定证明：

```text
this governance contract references these exact public summaries
```

不证明：

```text
raw private evidence is embedded
the runs are fresh today
the actions were repeated in chapter 24
the sandbox results are production performance
```

### 变更政策

[`change-policy.json`](/labs/ch24/change-policy.json) 固定 L0–L3。

L2/L3：

```text
requester_may_approve                 false
approver_may_execute                  false
requester_approver_executor_distinct  true
shared_accounts_allowed               false
independent_approver_qualified        true
exact_target_and_blast_radius         true
machine_preconditions                 true
confirmation_binds target/action/time true
```

break-glass：

```text
skip normal wait          allowed
skip target/evidence      forbidden
time-bounded identity     required
after-action review       required
credential rotation       required
```

### 正式 L0 运行

需要第 19 章 private mode-0600 inventory：

```bash
export PG36_CH19_INVENTORY=/absolute/private/baseline.yml
export PG36_EVIDENCE_DIR=/absolute/private/new-empty/ch24-run

static/labs/ch24/task.sh all
```

runner：

```text
1 reject nonempty evidence directory
2 run ch19 task.sh all
3 build governance-evidence.json
4 hash current source files
5 bind ch20–ch23 summaries
6 positive validation
7 negative mutation validation
8 review permissions and secret patterns
```

没有 reset action。

### 当前第 19 章 gate

正式运行观测：

```text
captured        2026-07-29T21:57:40Z
Pigsty          v4.5.0
PostgreSQL      major 18
hosts           4
PG members      4
sandbox L2      accepted-with-exceptions
exceptions      6
production      pending
mutation        none
```

六项例外：

- shared hypervisor；
- single etcd；
- single backup target；
- virtual storage；
- sandbox inventory secret handling；
- lab resource floor。

再次通过 gate 没有消除这些例外。

### 正式 evidence

```text
run id       34909737-527a-460c-927c-d9d71c93aa13
captured     2026-07-29T21:57:40.539Z
mode         read-only-contract-and-live-baseline-binding
mutation     none
```

private evidence 保存：

- ch19 current capture；
- governance evidence；
- positive report；
- negative report；
- review output。

公开仓库只保存
[`governance-run.json`](/labs/ch24/governance-run.json)。

## 24.6.3 输出观察契约，并拒绝没有动作的告警候选 {#item-24-6-3}

### 观察契约

[`observation-contract.json`](/labs/ch24/observation-contract.json) 为五个目标绑定：

```text
source
metric type
good/total selector
dimensions
recording rule
query template
missing semantics
fallback
```

Pigsty component labels：

```text
cls / ins / ip
```

application labels：

```text
service / operation_class / environment
```

禁止：

```text
customer_id / tenant_id / order_id / raw_sql / error_message
```

### component telemetry

四组：

| ID | sources | purpose |
|---|---|---|
| `PG-COMPONENT-ACTIVITY` | activity/database/io | concurrency/error/I/O diagnosis |
| `PG-COMPONENT-REPLICATION` | replication/receiver/LSN | replay and WAL risk |
| `PG-COMPONENT-ARCHIVE` | archiver/pgBackRest | recovery-path risk |
| `PIGSTY-CORRELATION` | Victoria stack | correlate PG/pool/LB/host/log |

它们不替代 user SLI。

### accepted alerts

[`alert-candidates.json`](/labs/ch24/alert-candidates.json)：

| alert | class | route |
|---|---|---|
| `PG36ShopAvailabilityFastBurn` | symptom | page SEV-1 |
| `PG36ShopAvailabilitySlowBurn` | symptom | page SEV-2 |
| `PG36ShopAvailabilityBudgetTicket` | symptom | ticket |
| `PG36ShopCorrectnessMismatch` | integrity | page SEV-1 |
| `PG36ShopFreshnessFastBurn` | symptom | page SEV-2 |
| `PG36ShopCapacityHorizon` | capacity | ticket |
| `PG36MonitoringPathBroken` | metamonitoring | page SEV-2 |

其中：

```text
symptom/integrity pages  4
metamonitoring pages     1
tickets                  2
```

所有 accepted item 都有：

- user impact；
- owner function；
- route；
- runbook；
- first safe action；
- verification；
- dashboard；
- missing semantics；
- silence policy；
- test id；
- review expiry。

### rejected alert

```text
PostgresInstanceDownWithoutAction
```

字段：

```text
proposed_route     page
decision           rejected
user_impact        null
owner              null
runbook             null
first_safe_action  null
```

replacement：

```text
correlate endpoint and user-event symptoms;
retain instance state on topology dashboard
```

这不是漏填；它是一个特意保留的反例。若有人将 `decision` 改为 accepted，
validator 失败。

### 三个 diagnostic-only causes

```text
PG36ReplicaReplayDistance
PG36PoolQueueDepth
PG36HostCpuHigh
```

都不 page。它们进入 dashboard，只有 correlated user/integrity/durability
合同触发后才帮助定位。

### metamonitoring

要求：

- application series freshness；
- VMAlert evaluation failures；
- Alertmanager notification canary；
- VictoriaMetrics query/ingestion；
- external blackbox probe。

`missing_is_healthy=false`。

### 二十个反例

| ID | mutation | 被哪类不变量拒绝 |
|---|---|---|
| N01 | 宣称 production SLO | scope |
| N02 | 删除 platform owner | ownership |
| N03 | process 即服务健康 | health chain |
| N04 | SLO target = 100% | ratio target |
| N05 | 排除 planned maintenance | exclusion |
| N06 | instance health = user SLI | measurement |
| N07 | missing = healthy | missing semantics |
| N08 | exhausted 无 action | budget policy |
| N09 | availability series 缺失算健康 | observation |
| N10 | 允许 unbounded labels | cardinality/privacy |
| N11 | page 无 first action | actionability |
| N12 | cause 直接 page | alert class |
| N13 | capacity 直接 page | route |
| N14 | 接受 actionless instance page | rejection |
| N15 | backup exit 0 即 restore | recovery evidence |
| N16 | 一人申请批准执行 | authority |
| N17 | break-glass 跳过 target/evidence | emergency boundary |
| N18 | access evidence 允许 secret | privacy |
| N19 | 伪造 ch21 run id | upstream identity |
| N20 | 沙箱切换称 production proof | claim boundary |

正式结果：

```text
case_count      20
rejected_count  20
failure_count   0
```

### review 输出

```text
status=review-ok
run_id=34909737-527a-460c-927c-d9d71c93aa13
service_card=complete
objectives=3-ratio+2-control
accepted_alerts=7
actionless_alerts=1-rejected
sops=4
counterexamples=20-rejected
ch19_live_gate=accepted-with-exceptions
upstream_runs=4-bound-by-hash
mutation=none
production_ch24_gate=pending
secret_material=absent
```

### 为什么 production 仍 pending

机器验证通过，不等于生产批准。仍缺：

1. 真实 service/data/platform/security owner；
2. 真实 on-call schedule 与 notification test；
3. 真实用户 traffic 的 eligible/good classifier；
4. 生产 failure domains；
5. 生产容量和 workload；
6. 多次切换/恢复的分布，而非一次观察；
7. 真实 RTO/RPO 谈判；
8. 法律/隐私 retention authority；
9. production secret/audit 路径；
10. 第 25 章真实 instrumentation/rules/dashboards；
11. 生产等价环境中的重复演练；
12. 正式 exception 与 expiry。

所以本章结论：

```text
governance contract       pass
machine validation        pass
adversarial validation    pass
current sandbox baseline  accepted-with-exceptions
upstream identity binding pass
real alert delivery       not run
production SLO evidence   absent
production approval       pending
```

### 重验已有 evidence

```bash
export PG36_EVIDENCE_DIR=/absolute/private/existing/ch24-run
static/labs/ch24/task.sh verify
static/labs/ch24/task.sh review
```

修改任何受 hash 保护的 policy/script 后，旧 evidence 验证会失败。应：

```text
retain old evidence with old source
create new empty evidence directory
run current contract again
compare decisions explicitly
```

不能覆盖旧 evidence 让历史“自动符合”新政策。

### 本章验收

- [x] service card 有 4 owner、6 dependencies、6 health layers；
- [x] 3 个 ratio SLO 和 2 个 control objective 可计算；
- [x] planned maintenance 未被隐藏；
- [x] missing telemetry 不算健康；
- [x] error budget 真实影响 change policy；
- [x] 四类 SOP 有停止线与验证；
- [x] L2/L3 权力分离；
- [x] break-glass 仍需 target/evidence；
- [x] 五个 observation source 有 query/missing/fallback；
- [x] 7 个 accepted alerts 可行动；
- [x] cause、capacity 与 symptom 分开；
- [x] actionless page 明确拒绝；
- [x] 六类 evidence 不允许 secret；
- [x] ch19 live gate 重新只读通过；
- [x] ch20–23 run 按 id/hash 绑定；
- [x] 20 个 adversarial mutations 全拒绝；
- [x] production gap 与 pending gate 保留。

第 25 章的任务不是重新发明这些语义，而是把它们实现为真实的应用 metric、
Pigsty recording/alert rules、dashboard、notification route 和 rule tests。

---

[上一节：证据、审计与合规](../05/) · [返回本章目录](../) · [下一章：望闻问切：监控体系与可观测诊断](/observability/) ·
[查看全书目录](/toc/) · [查看索引中心](/indexes/)