From 4d5af7698d095f8cb9d87016aafaa3c24cc3650a Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 8 Sep 2026 08:56:02 +0200 Subject: [PATCH] First deployment: verified on railiance01 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RCP-WP-0002 T02-T04 done, readiness_state verified with evidence attached rather than ahead of it. rapp-postgres provisioned canned_prompts on platform-pg-2 and sent a database-owner receipt with 12 checks proven live. creds/canned-prompts-publish was deliberately not issued, so the service runs read-only and POST /packages returns 503 explaining why — the intended posture, not a gap. Four defects surfaced that only a real rollout could expose, two of them silent: - SET ROLE opened an implicit transaction that Alembic nested inside rather than owning, so every revision logged as applied and was rolled back. Alembic reported success against an empty database. - The egress NetworkPolicy selected app.kubernetes.io/name, which the migration Job does not carry. The Job matched only the default-deny and succeeded exactly once, because it ran before the policies existed; the next migration would have failed on DNS. Now selects part-of, with ingress split into its own policy so the Job is never reachable. - env.py read database_url rather than resolved_database_url, so the migration could never run where the credential is a mounted file. - live-image-digest-match extracted the pin with a line-offset grep, which returned empty once comments were added above `version:`. The check degraded to reporting "not pinned yet" while a digest was pinned — it could not have passed for any pin. Now parsed as YAML. A verification step that cannot fail is worth less than none, because it is trusted. Evidence: docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM Assistant: claude-code Assistant-Model: opus Assistant-Process: 388925@bnt-lap001 Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502 --- SCOPE.md | 16 ++++--- declarations/rapp.yaml | 6 +-- ...WP-0002-T04-first-deployment-2026-09-08.md | 48 +++++++++++++++++++ manifests/database-secrets.yaml | 35 +++++++------- manifests/migration.yaml | 12 ++++- manifests/runtime.yaml | 23 +++++++-- tools/smoke.sh | 10 +++- workplans/RCP-WP-0002-first-deployment.md | 46 ++++++++++++++++-- 8 files changed, 157 insertions(+), 39 deletions(-) create mode 100644 docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md diff --git a/SCOPE.md b/SCOPE.md index 80f2648..54c69a2 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -27,15 +27,17 @@ Railiance, without moving product ownership into an operations repository. ## Current State -**Declared.** The image is published and pinned by digest -(`sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123`), the manifests parse, and -`readiness_state` is `declared`. Nothing is deployed yet. +**Verified and running** on railiance01 as of 2026-09-08. Image +`sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf` +(tag 0.1.4), schema at alembic `0002`, all deployment and service-level smoke +checks passing. Evidence in `docs/evidence/`. -One thing blocks a first rollout: the database and its roles do not exist. -`rapp-postgres` has the request as `consumers/canned-prompts.yaml`; minting the -OpenBao credentials is its operator's step, not this repo's. +The service is **read-only by design**: `creds/canned-prompts-publish` was +deliberately not issued, so `POST /packages` returns 503 explaining why. That is +the intended posture until per-publisher identity exists upstream — not a +misconfiguration. Do not "fix" it by minting a credential. -`workplans/RCP-WP-0002-first-deployment.md` carries the remaining sequence. +`workplans/RCP-WP-0002-first-deployment.md` records how it got there. ## Verification diff --git a/declarations/rapp.yaml b/declarations/rapp.yaml index 2c7433a..5d55cf6 100644 --- a/declarations/rapp.yaml +++ b/declarations/rapp.yaml @@ -4,7 +4,7 @@ rapp_id: rapp-canned-prompts repo: rapp-canned-prompts ownership_repo: canned-prompts contract_version: 1.0.0 -readiness_state: declared +readiness_state: verified workload_identity: name: canned-prompts package_type: manifest-managed-platform-service @@ -34,11 +34,11 @@ composition: upstream_components: - name: canned-prompts source: forgejo.coulomb.social/coulomb/canned-prompts - # Published 2026-09-07 from canned-prompts service/Dockerfile, tag 0.1.0. + # Published 2026-09-07 from canned-prompts service/Dockerfile, tag 0.1.4. # Pinned by digest rather than tag: a tag can be moved, and # live-image-digest-match would then pass against something that is no # longer what this repo reviewed. - version: sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123 + version: sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf rollout_contract: default_mode: kubectl-server-side-apply smoke_contract: diff --git a/docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md b/docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md new file mode 100644 index 0000000..ff60f7a --- /dev/null +++ b/docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md @@ -0,0 +1,48 @@ +# RCP-WP-0002-T04 — first deployment evidence + +**Date:** 2026-09-08 +**Image:** `sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf` (tag 0.1.4) +**Cluster:** railiance01 / k3s, namespace `canned-prompts` +**Schema:** alembic revision 0002, applied by the migration Job as `canned_prompts_migrate` with `SET ROLE canned_prompts_owner` + +## `tools/smoke.sh` + +```text +PASS external-secrets-ready:canned-prompts-postgres-runtime +PASS external-secrets-ready:canned-prompts-postgres-migration +PASS private-service-only:type +PASS private-service-only:no-ingress +PASS networkpolicies-present +PASS live-image-digest-match +--- service-level (/home/worsch/canned-prompts/service/tools/smoke.py) --- +PASS liveness-ok 200 {'status': 'ok'} +PASS readiness-ok 200 ok +PASS state-health-ok 200 connected +PASS migration-at-head running 0002, expected 0002 +PASS index-queryable 200 +PASS registry-queryable 200 + +all 6 checks passed + +all deployment checks passed +``` + +## Read-only posture verified + +`POST /packages` returns **503**, not a 500 and not an acceptance: + +```json +{"detail":"publishing is not configured: this service has no publisher identity, so it cannot tell who is calling and refuses writes rather than accepting anonymous publishes (§ 20.1)"} +``` + +This is the intended state. `creds/canned-prompts-publish` was deliberately not issued (rapp-postgres receipt, 2026-09-08). The read surface answers normally: `/packages` and `/index` both return empty result sets rather than errors. + +## Defects found and fixed during this rollout + +| Defect | Consequence had it shipped | +|---|---| +| `env.py` read `database_url` rather than `resolved_database_url` | Migration could never run in the cluster, where the credential is a mounted file | +| `SET ROLE` opened an implicit transaction Alembic then nested inside | Every migration logged as applied and was silently rolled back — an empty database reported as success | +| Egress NetworkPolicy selected `name`, not `part-of` | Migration Job matched only the default-deny; it succeeded once purely because it ran before the policies existed | +| Missing optional publish-token file treated as a hard failure | The documented read-only posture returned 500 instead of an explanatory 503 | +| `smoke.sh` extracted the digest with a line-offset `grep` | `live-image-digest-match` silently degraded to "not pinned" and could never have passed | diff --git a/manifests/database-secrets.yaml b/manifests/database-secrets.yaml index ef9eef3..e9f76d7 100644 --- a/manifests/database-secrets.yaml +++ b/manifests/database-secrets.yaml @@ -69,22 +69,19 @@ spec: - extract: key: creds/canned-prompts-migration --- -# The publish token. Absent, the service is read-only — which is the correct -# posture until per-publisher identity exists (canned-prompts service/auth.py). -apiVersion: external-secrets.io/v1 -kind: ExternalSecret -metadata: - name: canned-prompts-publish-token - namespace: canned-prompts -spec: - refreshInterval: 15m - secretStoreRef: - kind: ClusterSecretStore - name: openbao-canned-prompts-database - target: - name: canned-prompts-publish-token - creationPolicy: Owner - deletionPolicy: Retain - dataFrom: - - extract: - key: creds/canned-prompts-publish +# The publish token is deliberately NOT declared as an ExternalSecret. +# +# rapp-postgres confirmed on 2026-09-08 that `creds/canned-prompts-publish` +# does not exist and is not being created, at our request. An ExternalSecret +# pointing at it would sit permanently unresolved — indistinguishable from a +# broken deployment, and an invitation for the next operator to "fix" it by +# minting a credential nobody decided to issue. +# +# Without the secret the service runs READ-ONLY, which is the intended posture +# until per-publisher identity exists upstream (RCP-WP-0002-T05). The runtime +# Deployment mounts the secret `optional: true`, so it starts cleanly without it +# and picks the token up if one is ever added. +# +# To enable publishing later: have rapp-postgres issue +# `creds/canned-prompts-publish`, add the ExternalSecret here, and revisit the +# NetworkPolicy ingress rule, which currently admits any namespace. diff --git a/manifests/migration.yaml b/manifests/migration.yaml index 846df10..6c3bbf7 100644 --- a/manifests/migration.yaml +++ b/manifests/migration.yaml @@ -17,6 +17,10 @@ spec: metadata: labels: app.kubernetes.io/name: canned-prompts-migration + # Carries part-of so the egress NetworkPolicy selects this Job too. + # Without it the Job matches only the default-deny and cannot reach + # PostgreSQL or DNS. + app.kubernetes.io/part-of: canned-prompts spec: automountServiceAccountToken: false restartPolicy: Never @@ -29,7 +33,7 @@ spec: type: RuntimeDefault containers: - name: migrate - image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123 + image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf command: ["alembic"] args: ["upgrade", "head"] workingDir: /app @@ -37,6 +41,12 @@ spec: # The migration role owns the schema; the runtime role does not. - name: CANNED_PROMPTS_DATABASE_URL_FILE value: /var/run/secrets/postgres-migration/url + # Authenticate as the leased migration login, then SET ROLE to the + # durable owner before creating anything. Leases are revoked; an + # object owned by a dead login has to be normalized afterwards. + # Required by the rapp-postgres database-owner boundary. + - name: CANNED_PROMPTS_MIGRATION_ROLE + value: canned_prompts_owner securityContext: allowPrivilegeEscalation: false capabilities: diff --git a/manifests/runtime.yaml b/manifests/runtime.yaml index 3bb3ab9..de102f1 100644 --- a/manifests/runtime.yaml +++ b/manifests/runtime.yaml @@ -27,7 +27,7 @@ spec: type: RuntimeDefault containers: - name: canned-prompts - image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123 + image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf imagePullPolicy: IfNotPresent ports: - name: http @@ -134,19 +134,36 @@ spec: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: - name: canned-prompts-runtime + name: canned-prompts-ingress namespace: canned-prompts spec: + # Ingress is for the serving pods only. The migration Job serves nothing and + # must not be reachable. podSelector: matchLabels: app.kubernetes.io/name: canned-prompts - policyTypes: [Ingress, Egress] + policyTypes: [Ingress] ingress: - from: - namespaceSelector: {} ports: - protocol: TCP port: 8000 +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: canned-prompts-egress + namespace: canned-prompts +spec: + # Selects on part-of, not name, so it covers the migration Job as well as the + # serving pods. Selecting only `name: canned-prompts` left the Job matched by + # nothing but the default-deny — it succeeded once purely because it ran + # before these policies existed, and the next migration would have failed. + podSelector: + matchLabels: + app.kubernetes.io/part-of: canned-prompts + policyTypes: [Egress] egress: # PostgreSQL and DNS only. The service fetches nothing: a package arrives # by publish, never by the registry reaching out, so it needs no egress to diff --git a/tools/smoke.sh b/tools/smoke.sh index 9354900..9adcea2 100755 --- a/tools/smoke.sh +++ b/tools/smoke.sh @@ -11,7 +11,15 @@ set -euo pipefail NS=${NS:-canned-prompts} -EXPECT_DIGEST=${EXPECT_DIGEST:-$(grep -A1 'name: canned-prompts$' declarations/rapp.yaml | grep 'version:' | awk '{print $2}' || true)} +# Parsed, not grepped. A line-offset grep silently returned empty once comments +# were added above `version:`, and the check then degraded to "not pinned yet" +# instead of failing — it could not have passed for any pin. +EXPECT_DIGEST=${EXPECT_DIGEST:-$(python3 -c " +import yaml, sys +d = yaml.safe_load(open('declarations/rapp.yaml')) +c = [u for u in d['composition']['upstream_components'] if u['name'] == 'canned-prompts'] +print(c[0]['version'] if c else '') +" 2>/dev/null || true)} FAILED=0 check() { # name, condition-output diff --git a/workplans/RCP-WP-0002-first-deployment.md b/workplans/RCP-WP-0002-first-deployment.md index c4c2b58..1978050 100644 --- a/workplans/RCP-WP-0002-first-deployment.md +++ b/workplans/RCP-WP-0002-first-deployment.md @@ -4,7 +4,7 @@ type: workplan title: "First deployment of canned-prompts on Railiance" domain: agents repo: rapp-canned-prompts -status: active +status: finished owner: codex topic_slug: practice created: "2026-09-06" @@ -62,7 +62,7 @@ output. `readiness_state` moved `draft` → `declared`. ```task id: RCP-WP-0002-T02 -status: wait +status: done priority: high state_hub_task_id: "61f5cbd7-8fcf-5156-87da-57239ae55d8f" ``` @@ -91,13 +91,19 @@ OpenBao credentials — was deliberately **not** run: credential issuance is `rapp-postgres`' to perform, and running it from the consuming side would take a decision that is not this repo's, however available the script happens to be. -Blocked here until those credentials exist. T03 and T04 are blocked behind it. +**Received 2026-09-08.** `rapp-postgres` provisioned and sent a database-owner +receipt with 12 checks proven live: runtime DDL denied (SQLSTATE 42501), +statement timeouts and `search_path` as declared, and both logins refused +CONNECT on `sbom_nexus` — the cell is shared, so that last one matters. + +`creds/canned-prompts-publish` was **not** created, as asked. The service runs +read-only, which is the intended posture rather than a gap. ## Apply and migrate ```task id: RCP-WP-0002-T03 -status: wait +status: done priority: high state_hub_task_id: "0b8d206e-bd28-5950-abf7-d824015c09a4" ``` @@ -120,11 +126,28 @@ The runtime deliberately does not migrate at start-up. Migrations as a Job keep a schema rollback separate from a code rollback and stop replicas racing each other. +**Done 2026-09-08, after four defects that only a real rollout could expose.** +Each is recorded in +`docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md`; the two worth +repeating here failed *silently*: + +- `SET ROLE` opened an implicit transaction that Alembic then nested inside + rather than owning, so every revision logged as applied and was rolled back. + Alembic reported success against an empty database. +- The egress NetworkPolicy selected `app.kubernetes.io/name`, which the + migration Job does not carry. The Job matched only the default-deny and + succeeded exactly once — because it ran before the policies existed. The next + migration would have failed with a DNS error. Now selects `part-of`, and + ingress is a separate policy so the Job is never reachable. + +`rapp-postgres` asked to be told when the first revision landed so they can +re-run ownership reconciliation. Notified. + ## Verify and record evidence ```task id: RCP-WP-0002-T04 -status: wait +status: done priority: high state_hub_task_id: "6d9eb97c-57e2-5b74-b6eb-455076713417" ``` @@ -138,6 +161,19 @@ second opinion about whether the service is healthy. Record the output as evidence, then move `readiness_state` to `deployed`, and to `verified` only with that evidence attached. +**Done 2026-09-08.** All six deployment checks and all six service-level checks +pass; evidence at +`docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md`. +`readiness_state` is `verified`, with the evidence attached rather than ahead of +it. + +**The check that could never have passed.** `live-image-digest-match` read the +pin with a line-offset `grep`, which returned empty once comments were added +above `version:` — and the check then degraded to reporting "not pinned yet" +instead of failing. It reported that while a digest *was* pinned. Now parsed as +YAML. A verification step that cannot fail is worth less than none, because it +is trusted. + ## Decide per-publisher identity ```task