First deployment: verified on railiance01

RCP-WP-0002 T02-T04 done, readiness_state verified with evidence attached
rather than ahead of it.

rapp-postgres provisioned canned_prompts on platform-pg-2 and sent a
database-owner receipt with 12 checks proven live. creds/canned-prompts-publish
was deliberately not issued, so the service runs read-only and POST /packages
returns 503 explaining why — the intended posture, not a gap.

Four defects surfaced that only a real rollout could expose, two of them
silent:

- SET ROLE opened an implicit transaction that Alembic nested inside rather
  than owning, so every revision logged as applied and was rolled back.
  Alembic reported success against an empty database.
- The egress NetworkPolicy selected app.kubernetes.io/name, which the
  migration Job does not carry. The Job matched only the default-deny and
  succeeded exactly once, because it ran before the policies existed; the next
  migration would have failed on DNS. Now selects part-of, with ingress split
  into its own policy so the Job is never reachable.
- env.py read database_url rather than resolved_database_url, so the migration
  could never run where the credential is a mounted file.
- live-image-digest-match extracted the pin with a line-offset grep, which
  returned empty once comments were added above `version:`. The check degraded
  to reporting "not pinned yet" while a digest was pinned — it could not have
  passed for any pin. Now parsed as YAML. A verification step that cannot fail
  is worth less than none, because it is trusted.

Evidence: docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502

SCOPE.md

@@ -27,15 +27,17 @@ Railiance, without moving product ownership into an operations repository.
 
 ## Current State
 
-**Declared.** The image is published and pinned by digest
+**Verified and running** on railiance01 as of 2026-09-08. Image
-(`sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123`), the manifests parse, and
+`sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf`
-`readiness_state` is `declared`. Nothing is deployed yet.
+(tag 0.1.4), schema at alembic `0002`, all deployment and service-level smoke
+checks passing. Evidence in `docs/evidence/`.
 
-One thing blocks a first rollout: the database and its roles do not exist.
+The service is **read-only by design**: `creds/canned-prompts-publish` was
-`rapp-postgres` has the request as `consumers/canned-prompts.yaml`; minting the
+deliberately not issued, so `POST /packages` returns 503 explaining why. That is
-OpenBao credentials is its operator's step, not this repo's.
+the intended posture until per-publisher identity exists upstream — not a
+misconfiguration. Do not "fix" it by minting a credential.
 
-`workplans/RCP-WP-0002-first-deployment.md` carries the remaining sequence.
+`workplans/RCP-WP-0002-first-deployment.md` records how it got there.
 
 ## Verification
 

declarations/rapp.yaml

@@ -4,7 +4,7 @@ rapp_id: rapp-canned-prompts
 repo: rapp-canned-prompts
 ownership_repo: canned-prompts
 contract_version: 1.0.0
-readiness_state: declared
+readiness_state: verified
 workload_identity:
   name: canned-prompts
   package_type: manifest-managed-platform-service
@@ -34,11 +34,11 @@ composition:
   upstream_components:
     - name: canned-prompts
       source: forgejo.coulomb.social/coulomb/canned-prompts
-      # Published 2026-09-07 from canned-prompts service/Dockerfile, tag 0.1.0.
+      # Published 2026-09-07 from canned-prompts service/Dockerfile, tag 0.1.4.
       # Pinned by digest rather than tag: a tag can be moved, and
       # live-image-digest-match would then pass against something that is no
       # longer what this repo reviewed.
-      version: sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123
+      version: sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf
 rollout_contract:
   default_mode: kubectl-server-side-apply
 smoke_contract:

docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md

@@ -0,0 +1,48 @@
+# RCP-WP-0002-T04 — first deployment evidence
+
+**Date:** 2026-09-08
+**Image:** `sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf` (tag 0.1.4)
+**Cluster:** railiance01 / k3s, namespace `canned-prompts`
+**Schema:** alembic revision 0002, applied by the migration Job as `canned_prompts_migrate` with `SET ROLE canned_prompts_owner`
+
+## `tools/smoke.sh`
+
+```text
+PASS  external-secrets-ready:canned-prompts-postgres-runtime
+PASS  external-secrets-ready:canned-prompts-postgres-migration
+PASS  private-service-only:type
+PASS  private-service-only:no-ingress
+PASS  networkpolicies-present
+PASS  live-image-digest-match
+--- service-level (/home/worsch/canned-prompts/service/tools/smoke.py) ---
+PASS  liveness-ok         200 {'status': 'ok'}
+PASS  readiness-ok        200 ok
+PASS  state-health-ok     200 connected
+PASS  migration-at-head   running 0002, expected 0002
+PASS  index-queryable     200
+PASS  registry-queryable  200
+
+all 6 checks passed
+
+all deployment checks passed
+```
+
+## Read-only posture verified
+
+`POST /packages` returns **503**, not a 500 and not an acceptance:
+
+```json
+{"detail":"publishing is not configured: this service has no publisher identity, so it cannot tell who is calling and refuses writes rather than accepting anonymous publishes (§ 20.1)"}
+```
+
+This is the intended state. `creds/canned-prompts-publish` was deliberately not issued (rapp-postgres receipt, 2026-09-08). The read surface answers normally: `/packages` and `/index` both return empty result sets rather than errors.
+
+## Defects found and fixed during this rollout
+
+| Defect | Consequence had it shipped |
+|---|---|
+| `env.py` read `database_url` rather than `resolved_database_url` | Migration could never run in the cluster, where the credential is a mounted file |
+| `SET ROLE` opened an implicit transaction Alembic then nested inside | Every migration logged as applied and was silently rolled back — an empty database reported as success |
+| Egress NetworkPolicy selected `name`, not `part-of` | Migration Job matched only the default-deny; it succeeded once purely because it ran before the policies existed |
+| Missing optional publish-token file treated as a hard failure | The documented read-only posture returned 500 instead of an explanatory 503 |
+| `smoke.sh` extracted the digest with a line-offset `grep` | `live-image-digest-match` silently degraded to "not pinned" and could never have passed |

manifests/database-secrets.yaml

@@ -69,22 +69,19 @@ spec:
     - extract:
         key: creds/canned-prompts-migration
 ---
-# The publish token. Absent, the service is read-only — which is the correct
+# The publish token is deliberately NOT declared as an ExternalSecret.
-# posture until per-publisher identity exists (canned-prompts service/auth.py).
+#
-apiVersion: external-secrets.io/v1
+# rapp-postgres confirmed on 2026-09-08 that `creds/canned-prompts-publish`
-kind: ExternalSecret
+# does not exist and is not being created, at our request. An ExternalSecret
-metadata:
+# pointing at it would sit permanently unresolved — indistinguishable from a
-  name: canned-prompts-publish-token
+# broken deployment, and an invitation for the next operator to "fix" it by
-  namespace: canned-prompts
+# minting a credential nobody decided to issue.
-spec:
+#
-  refreshInterval: 15m
+# Without the secret the service runs READ-ONLY, which is the intended posture
-  secretStoreRef:
+# until per-publisher identity exists upstream (RCP-WP-0002-T05). The runtime
-    kind: ClusterSecretStore
+# Deployment mounts the secret `optional: true`, so it starts cleanly without it
-    name: openbao-canned-prompts-database
+# and picks the token up if one is ever added.
-  target:
+#
-    name: canned-prompts-publish-token
+# To enable publishing later: have rapp-postgres issue
-    creationPolicy: Owner
+# `creds/canned-prompts-publish`, add the ExternalSecret here, and revisit the
-    deletionPolicy: Retain
+# NetworkPolicy ingress rule, which currently admits any namespace.
-  dataFrom:
-    - extract:
-        key: creds/canned-prompts-publish

manifests/migration.yaml

@@ -17,6 +17,10 @@ spec:
     metadata:
       labels:
         app.kubernetes.io/name: canned-prompts-migration
+        # Carries part-of so the egress NetworkPolicy selects this Job too.
+        # Without it the Job matches only the default-deny and cannot reach
+        # PostgreSQL or DNS.
+        app.kubernetes.io/part-of: canned-prompts
     spec:
       automountServiceAccountToken: false
       restartPolicy: Never
@@ -29,7 +33,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: migrate
-          image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123
+          image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf
           command: ["alembic"]
           args: ["upgrade", "head"]
           workingDir: /app
@@ -37,6 +41,12 @@ spec:
             # The migration role owns the schema; the runtime role does not.
             - name: CANNED_PROMPTS_DATABASE_URL_FILE
               value: /var/run/secrets/postgres-migration/url
+            # Authenticate as the leased migration login, then SET ROLE to the
+            # durable owner before creating anything. Leases are revoked; an
+            # object owned by a dead login has to be normalized afterwards.
+            # Required by the rapp-postgres database-owner boundary.
+            - name: CANNED_PROMPTS_MIGRATION_ROLE
+              value: canned_prompts_owner
           securityContext:
             allowPrivilegeEscalation: false
             capabilities:

manifests/runtime.yaml

@@ -27,7 +27,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: canned-prompts
-          image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123
+          image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf
           imagePullPolicy: IfNotPresent
           ports:
             - name: http
@@ -134,19 +134,36 @@ spec:
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
-  name: canned-prompts-runtime
+  name: canned-prompts-ingress
   namespace: canned-prompts
 spec:
+  # Ingress is for the serving pods only. The migration Job serves nothing and
+  # must not be reachable.
   podSelector:
     matchLabels:
       app.kubernetes.io/name: canned-prompts
-  policyTypes: [Ingress, Egress]
+  policyTypes: [Ingress]
   ingress:
     - from:
         - namespaceSelector: {}
       ports:
         - protocol: TCP
           port: 8000
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: canned-prompts-egress
+  namespace: canned-prompts
+spec:
+  # Selects on part-of, not name, so it covers the migration Job as well as the
+  # serving pods. Selecting only `name: canned-prompts` left the Job matched by
+  # nothing but the default-deny — it succeeded once purely because it ran
+  # before these policies existed, and the next migration would have failed.
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/part-of: canned-prompts
+  policyTypes: [Egress]
   egress:
     # PostgreSQL and DNS only. The service fetches nothing: a package arrives
     # by publish, never by the registry reaching out, so it needs no egress to

tools/smoke.sh

@@ -11,7 +11,15 @@
 set -euo pipefail
 
 NS=${NS:-canned-prompts}
-EXPECT_DIGEST=${EXPECT_DIGEST:-$(grep -A1 'name: canned-prompts$' declarations/rapp.yaml | grep 'version:' | awk '{print $2}' || true)}
+# Parsed, not grepped. A line-offset grep silently returned empty once comments
+# were added above `version:`, and the check then degraded to "not pinned yet"
+# instead of failing — it could not have passed for any pin.
+EXPECT_DIGEST=${EXPECT_DIGEST:-$(python3 -c "
+import yaml, sys
+d = yaml.safe_load(open('declarations/rapp.yaml'))
+c = [u for u in d['composition']['upstream_components'] if u['name'] == 'canned-prompts']
+print(c[0]['version'] if c else '')
+" 2>/dev/null || true)}
 FAILED=0
 
 check() { # name, condition-output

workplans/RCP-WP-0002-first-deployment.md

@@ -4,7 +4,7 @@ type: workplan
 title: "First deployment of canned-prompts on Railiance"
 domain: agents
 repo: rapp-canned-prompts
-status: active
+status: finished
 owner: codex
 topic_slug: practice
 created: "2026-09-06"
@@ -62,7 +62,7 @@ output. `readiness_state` moved `draft` → `declared`.
 
 ```task
 id: RCP-WP-0002-T02
-status: wait
+status: done
 priority: high
 state_hub_task_id: "61f5cbd7-8fcf-5156-87da-57239ae55d8f"
 ```
@@ -91,13 +91,19 @@ OpenBao credentials — was deliberately **not** run: credential issuance is
 `rapp-postgres`' to perform, and running it from the consuming side would take a
 decision that is not this repo's, however available the script happens to be.
 
-Blocked here until those credentials exist. T03 and T04 are blocked behind it.
+**Received 2026-09-08.** `rapp-postgres` provisioned and sent a database-owner
+receipt with 12 checks proven live: runtime DDL denied (SQLSTATE 42501),
+statement timeouts and `search_path` as declared, and both logins refused
+CONNECT on `sbom_nexus` — the cell is shared, so that last one matters.
+
+`creds/canned-prompts-publish` was **not** created, as asked. The service runs
+read-only, which is the intended posture rather than a gap.
 
 ## Apply and migrate
 
 ```task
 id: RCP-WP-0002-T03
-status: wait
+status: done
 priority: high
 state_hub_task_id: "0b8d206e-bd28-5950-abf7-d824015c09a4"
 ```
@@ -120,11 +126,28 @@ The runtime deliberately does not migrate at start-up. Migrations as a Job keep
 a schema rollback separate from a code rollback and stop replicas racing each
 other.
 
+**Done 2026-09-08, after four defects that only a real rollout could expose.**
+Each is recorded in
+`docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md`; the two worth
+repeating here failed *silently*:
+
+- `SET ROLE` opened an implicit transaction that Alembic then nested inside
+  rather than owning, so every revision logged as applied and was rolled back.
+  Alembic reported success against an empty database.
+- The egress NetworkPolicy selected `app.kubernetes.io/name`, which the
+  migration Job does not carry. The Job matched only the default-deny and
+  succeeded exactly once — because it ran before the policies existed. The next
+  migration would have failed with a DNS error. Now selects `part-of`, and
+  ingress is a separate policy so the Job is never reachable.
+
+`rapp-postgres` asked to be told when the first revision landed so they can
+re-run ownership reconciliation. Notified.
+
 ## Verify and record evidence
 
 ```task
 id: RCP-WP-0002-T04
-status: wait
+status: done
 priority: high
 state_hub_task_id: "6d9eb97c-57e2-5b74-b6eb-455076713417"
 ```
@@ -138,6 +161,19 @@ second opinion about whether the service is healthy.
 Record the output as evidence, then move `readiness_state` to `deployed`, and to
 `verified` only with that evidence attached.
 
+**Done 2026-09-08.** All six deployment checks and all six service-level checks
+pass; evidence at
+`docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md`.
+`readiness_state` is `verified`, with the evidence attached rather than ahead of
+it.
+
+**The check that could never have passed.** `live-image-digest-match` read the
+pin with a line-offset `grep`, which returned empty once comments were added
+above `version:` — and the check then degraded to reporting "not pinned yet"
+instead of failing. It reported that while a digest *was* pinned. Now parsed as
+YAML. A verification step that cannot fail is worth less than none, because it
+is trusted.
+
 ## Decide per-publisher identity
 
 ```task