rapp-canned-prompts/docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md

# RCP-WP-0002-T04 — first deployment evidence

**Date:** 2026-09-08
**Image:** `sha256:14c7b92f20d63f2e70ea17b0b45d3c483521fbbaf14ee5bc277fa541e10452e2` (tag 0.1.5)
**Cluster:** railiance01 / k3s, namespace `canned-prompts`
**Schema:** alembic revision 0002, applied by the migration Job as `canned_prompts_migrate` with `SET ROLE canned_prompts_owner`

## `tools/smoke.sh`

```text
PASS  external-secrets-ready:canned-prompts-postgres-runtime
PASS  external-secrets-ready:canned-prompts-postgres-migration
PASS  private-service-only:type
PASS  private-service-only:no-ingress
PASS  networkpolicies-present
PASS  live-image-digest-match
--- service-level (/home/worsch/canned-prompts/service/tools/smoke.py) ---
PASS  liveness-ok         200 {'status': 'ok'}
PASS  readiness-ok        200 ok
PASS  state-health-ok     200 connected
PASS  migration-at-head   running 0002, expected 0002
PASS  index-queryable     200
PASS  registry-queryable  200

all 6 checks passed

all deployment checks passed
```

## Read-only posture verified

`POST /packages` returns **503**, not a 500 and not an acceptance:

```json
{"detail":"publishing is not configured: this service has no publisher identity, so it cannot tell who is calling and refuses writes rather than accepting anonymous publishes (§ 20.1)"}
```

This is the intended state. `creds/canned-prompts-publish` was deliberately not issued (rapp-postgres receipt, 2026-09-08). The read surface answers normally: `/packages` and `/index` both return empty result sets rather than errors.

## Defects found and fixed during this rollout

| Defect | Consequence had it shipped |
|---|---|
| `env.py` read `database_url` rather than `resolved_database_url` | Migration could never run in the cluster, where the credential is a mounted file |
| `SET ROLE` opened an implicit transaction Alembic then nested inside | Every migration logged as applied and was silently rolled back — an empty database reported as success |
| Egress NetworkPolicy selected `name`, not `part-of` | Migration Job matched only the default-deny; it succeeded once purely because it ran before the policies existed |
| Missing optional publish-token file treated as a hard failure | The documented read-only posture returned 500 instead of an explanatory 503 |
| `smoke.sh` extracted the digest with a line-offset `grep` | `live-image-digest-match` silently degraded to "not pinned" and could never have passed |


## Correction: `verified` was recorded too early

I set `readiness_state: verified` after a passing smoke run, then found the
deployment `0/1` eight hours later. It is back to `deployed` until it has been
observed across a credential rotation.

**Cause.** Platform credentials are 30-minute leases, not passwords. The
service read the mounted URL once at start-up and never again, so External
Secrets kept the file current while the engine held the URL it booted with.
Every reconnection after the first lease expiry used a credential the database
had already revoked. `/readyz` reported it correctly —
`database unreachable: OperationalError` — and the pod sat unready for eight
hours without ever being restarted, because liveness is deliberately
independent of the database.

**Fix (0.1.5).** The engine now re-reads the credential for every new
connection via a `do_connect` hook, and `pool_recycle` is 900s so a pooled
connection is retired well inside the 30-minute lease. Only the username and
password are taken from the refreshed URL; host, port and database come from
the engine, so a malformed refresh cannot silently redirect the service.

**What this says about the verification itself.** A smoke run inside the first
lease window cannot distinguish a service that works from one that works *once*.
The criterion for `verified` is therefore observation across a full rotation,
not a passing check at a single instant — an availability property is not
provable by one sample.


## The watcher was also wrong

The first lease watch returned **FAILED** on one sample out of 21. That sample
was the instrument, not the service: it treated an empty `kubectl` result as
"not ready", so a transient API hiccup was recorded as an outage.

The service had returned **zero** 503s across the whole window. The readiness
probe fires every 5s, so a genuine two-minute outage would have left roughly
twenty-four of them, and the pod ran 46 minutes past a 30-minute lease with no
restarts.

I did not override the verdict by argument — a verdict that can be talked around
is worth nothing. The watcher is fixed (`tools/lease-watch.sh`) to distinguish
a failed *query* from a failed *service*, and to corroborate its own sampling
against the kubelet's probe history, which is a far better instrument than one
sample a minute. Then re-measured.

**Three checks in this rollout were themselves defective**, all the same shape —
unable to distinguish their own failure from the failure they were watching for:

| Check | How it lied |
|---|---|
| `live-image-digest-match` | line-offset `grep` returned empty; reported "not pinned yet" while a digest was pinned |
| `check_readiness` (earlier) | one `except` reported "database unreachable" for a reachable but unmigrated database |
| lease watcher v1 | empty `kubectl` output counted as an outage |
| lease watcher v2 | `grep -c` exits 1 on zero matches, so `\|\| echo "?"` appended a marker to a legitimate `0` and the verdict could never read clean |
| lease watcher v3 | printed "survived past the lease TTL" after a 12-minute window on a 13-minute-old pod — it never checked the one thing it claimed |

A verification step that cannot fail correctly is worse than none, because it is
trusted. That is the durable lesson from this deployment, more than any single
defect it found.

The watcher needed three attempts, and the third failure is the most instructive
of the set. Version 2 reported `not_ready=0 query_failed=0` and `503s: 0` — every
measurement clean — and still printed `FAILED`, because `grep -c` exits non-zero
when it counts zero and the guard appended an error marker on top of the correct
answer. The check was wrong *in the direction of alarm*, which is the survivable
direction; the same class of bug pointing the other way is what let
`live-image-digest-match` report success it had not established.

So the verdict logic is now itself tested — the fix was accompanied by feeding
the counter a matching line and a non-matching one and confirming it returns 1
and 0 — rather than assumed correct because it looked right.

**Version 3 then failed in the dangerous direction.** It reported
`RESULT: survived — continuously ready past the lease TTL` after a clean
12-minute window on a pod that had been up for 13 minutes. Every measurement in
it was accurate; the *conclusion* was not, because nothing checked that the
observation window exceeded the 30-minute lease it claimed to have outlasted.

That is the failure mode worth fearing. Versions 1 and 2 cried wolf, which
provokes investigation. Version 3 would have been believed — and `verified`
recorded on it — for the same reason `live-image-digest-match` would have been
believed: a check reporting success it has not established is indistinguishable
from a check that works, right up until it matters.

The verdict now requires `uptime > lease TTL` and reports `INCONCLUSIVE` when a
window is clean but too short, because "clean" and "proven" are different
claims and only one of them was ever being measured.