uptime 36m > lease TTL 30m, 21 samples, zero not-ready, zero query failures,
zero kubelet readiness 503s, no restarts. The verdict now names the uptime it
checked, so the claim can be audited rather than taken.
T05 done upstream in canned-prompts 0.2.0 / migration 0003. DR-3 had already
resolved the identity question as app-local accounts with OIDC demand-gated;
research found that rather than my judgement supplying it.
Left open deliberately: the NetworkPolicy ingress rule still admits any
namespace. Publisher identity now gates writes so it is no longer the only
control, and it should narrow once the legitimate callers are known — recorded
rather than tightened on a guess.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM
Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
Version 3 reported 'survived past the lease TTL' after a clean 12-minute window
on a 13-minute-old pod. Every measurement in it was accurate; the conclusion
was not, because nothing checked that the observation window had actually
exceeded the 30-minute lease it claimed to have outlasted.
Fifth defect in this instrument, and the first to err toward reassurance.
Versions 1 and 2 cried wolf, which provokes investigation. This one would have
been believed, and readiness_state: verified recorded on it — the same way
live-image-digest-match would have been believed. A check reporting success it
has not established is indistinguishable from one that works, until it matters.
The verdict now requires uptime > lease TTL and reports INCONCLUSIVE when a
window is clean but too short. 'Clean' and 'proven' are different claims and
only one of them was being measured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM
Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
Continuously ready across a full credential lease rotation: 11 samples, zero
not-ready, zero query failures, zero kubelet readiness 503s with a probe every
5 seconds, no restarts, pod uptime well past the 30-minute lease TTL.
Recorded from a working instrument on the third attempt. The first two verdicts
were FAILED and both were the watcher's own defects — an empty kubectl result
counted as an outage, then grep -c's exit status poisoning a clean count. The
verdict was not talked around; the instrument was fixed and the measurement
repeated.
Evidence: docs/evidence/RCP-WP-0002-T04-lease-rotation-2026-09-08.log
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM
Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
I recorded readiness_state: verified after a passing smoke run and found the
deployment 0/1 eight hours later.
Platform credentials are 30-minute leases, not passwords. The service read the
mounted URL once at start-up and never again, so External Secrets kept the file
current while the engine held the URL it booted with, and every reconnection
after the first lease expiry used a credential the database had already
revoked.
/readyz reported it accurately — "database unreachable: OperationalError" — and
the pod sat unready for eight hours without being restarted, because liveness
is deliberately independent of the database. That separation behaved exactly as
designed: the process was alive and could not serve, and it said so.
Fixed in 0.1.5: the engine re-reads the credential for every new connection and
pool_recycle is 900s, well inside the lease. Only username and password come
from the refreshed URL; host, port and database come from the engine, so a
malformed refresh cannot silently redirect the service.
readiness_state back to deployed. The bar for verified is now observation
across a full lease rotation, because a smoke run inside the first window
cannot distinguish a service that works from one that works once — an
availability property is not provable by a single sample.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM
Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
RCP-WP-0002 T02-T04 done, readiness_state verified with evidence attached
rather than ahead of it.
rapp-postgres provisioned canned_prompts on platform-pg-2 and sent a
database-owner receipt with 12 checks proven live. creds/canned-prompts-publish
was deliberately not issued, so the service runs read-only and POST /packages
returns 503 explaining why — the intended posture, not a gap.
Four defects surfaced that only a real rollout could expose, two of them
silent:
- SET ROLE opened an implicit transaction that Alembic nested inside rather
than owning, so every revision logged as applied and was rolled back.
Alembic reported success against an empty database.
- The egress NetworkPolicy selected app.kubernetes.io/name, which the
migration Job does not carry. The Job matched only the default-deny and
succeeded exactly once, because it ran before the policies existed; the next
migration would have failed on DNS. Now selects part-of, with ingress split
into its own policy so the Job is never reachable.
- env.py read database_url rather than resolved_database_url, so the migration
could never run where the credential is a mounted file.
- live-image-digest-match extracted the pin with a line-offset grep, which
returned empty once comments were added above `version:`. The check degraded
to reporting "not pinned yet" while a digest was pinned — it could not have
passed for any pin. Now parsed as YAML. A verification step that cannot fail
is worth less than none, because it is trusted.
Evidence: docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM
Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
RCP-WP-0002-T01 done. Published
forgejo.coulomb.social/coulomb/canned-prompts:0.1.0 and pinned
sha256:e0ded3c7fe25... in declarations/rapp.yaml and both manifests.
Pinned by digest rather than tag: a tag can be moved, and
live-image-digest-match would then pass against something that is no longer
what this repo reviewed. Verified after the push by fetching the manifest back
by digest rather than trusting the push output.
readiness_state draft -> declared. Not deployed, so not `deployed`.
T02 requested rather than performed. rapp-postgres now carries
consumers/canned-prompts.yaml against its documented PostgresConsumer shape,
and its agent has the request. `make provision-consumers`, which mints the
OpenBao credentials, was deliberately not run: credential issuance belongs to
that repo's operator, and running it from the consuming side would take a
decision that is not this repo's, however available the script is.
T03 and T04 move to wait behind it.
Also corrected a check of my own: I briefly read the cluster list as lacking
platform-pg-2 and suspected the manifests targeted a host that does not exist.
That was my own truncated output. platform-pg-2 is present and healthy, and the
postgres-client label matches what sbom-nexus actually carries in the cluster
rather than only what its repo says.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM
Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
Registers the repo with State Hub (agents / practice, prefix RCP-WP) and fills
in the rapp shape.
declarations/rapp.yaml declares a manifest-managed platform service owned by
canned-prompts, bound to rail-kubernetes and reef-railiance, with rollout,
smoke and rollback contracts.
The image pin says `pending-publication` rather than carrying a placeholder
digest. The image builds and was verified locally (canned-prompts
CANP-WP-0006-T06) but has never been pushed, so no registry digest exists. A
placeholder shaped like a real digest would be worse than a sentinel: it could
be mistaken for something deployable.
manifests/ follows the rapp-sbom-nexus shape: namespace labelled for the
postgres client, external secrets from OpenBao, a migration Job, and the
runtime Deployment with a ClusterIP-only Service, dedicated ServiceAccount and
default-deny plus runtime NetworkPolicies.
Three choices worth stating. Credentials arrive as mounted files, never env
vars — an env var holding a password is visible in kubectl describe, in crash
dumps, and to anything that can read /proc. Migrations run as a Job rather than
at start-up, so a schema rollback stays separate from a code rollback and
replicas do not race. Liveness points at /healthz, which checks only that the
process is up: pointing it at a database-dependent path would restart every
replica during a database blip.
Egress is PostgreSQL and DNS only. A package arrives by publish; the registry
never reaches out, so it is given no path to.
tools/smoke.sh checks what only the cluster can answer and calls
canned-prompts' service/tools/smoke.py for health and migration head, rather
than holding a second opinion about whether the service is healthy.
RCP-WP-0002 carries the two operator actions that block a first rollout —
publishing the image and provisioning database roles — and records
per-publisher identity as a decision belonging upstream, which this repo must
not paper over with cluster configuration implying finer control than exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM
Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
