First deployment: verified on railiance01
RCP-WP-0002 T02-T04 done, readiness_state verified with evidence attached rather than ahead of it. rapp-postgres provisioned canned_prompts on platform-pg-2 and sent a database-owner receipt with 12 checks proven live. creds/canned-prompts-publish was deliberately not issued, so the service runs read-only and POST /packages returns 503 explaining why — the intended posture, not a gap. Four defects surfaced that only a real rollout could expose, two of them silent: - SET ROLE opened an implicit transaction that Alembic nested inside rather than owning, so every revision logged as applied and was rolled back. Alembic reported success against an empty database. - The egress NetworkPolicy selected app.kubernetes.io/name, which the migration Job does not carry. The Job matched only the default-deny and succeeded exactly once, because it ran before the policies existed; the next migration would have failed on DNS. Now selects part-of, with ingress split into its own policy so the Job is never reachable. - env.py read database_url rather than resolved_database_url, so the migration could never run where the credential is a mounted file. - live-image-digest-match extracted the pin with a line-offset grep, which returned empty once comments were added above `version:`. The check degraded to reporting "not pinned yet" while a digest was pinned — it could not have passed for any pin. Now parsed as YAML. A verification step that cannot fail is worth less than none, because it is trusted. Evidence: docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM Assistant: claude-code Assistant-Model: opus Assistant-Process: 388925@bnt-lap001 Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
This commit is contained in:
parent
1bf062828f
commit
4d5af7698d
8 changed files with 157 additions and 39 deletions
16
SCOPE.md
16
SCOPE.md
|
|
@ -27,15 +27,17 @@ Railiance, without moving product ownership into an operations repository.
|
|||
|
||||
## Current State
|
||||
|
||||
**Declared.** The image is published and pinned by digest
|
||||
(`sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123`), the manifests parse, and
|
||||
`readiness_state` is `declared`. Nothing is deployed yet.
|
||||
**Verified and running** on railiance01 as of 2026-09-08. Image
|
||||
`sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf`
|
||||
(tag 0.1.4), schema at alembic `0002`, all deployment and service-level smoke
|
||||
checks passing. Evidence in `docs/evidence/`.
|
||||
|
||||
One thing blocks a first rollout: the database and its roles do not exist.
|
||||
`rapp-postgres` has the request as `consumers/canned-prompts.yaml`; minting the
|
||||
OpenBao credentials is its operator's step, not this repo's.
|
||||
The service is **read-only by design**: `creds/canned-prompts-publish` was
|
||||
deliberately not issued, so `POST /packages` returns 503 explaining why. That is
|
||||
the intended posture until per-publisher identity exists upstream — not a
|
||||
misconfiguration. Do not "fix" it by minting a credential.
|
||||
|
||||
`workplans/RCP-WP-0002-first-deployment.md` carries the remaining sequence.
|
||||
`workplans/RCP-WP-0002-first-deployment.md` records how it got there.
|
||||
|
||||
## Verification
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ rapp_id: rapp-canned-prompts
|
|||
repo: rapp-canned-prompts
|
||||
ownership_repo: canned-prompts
|
||||
contract_version: 1.0.0
|
||||
readiness_state: declared
|
||||
readiness_state: verified
|
||||
workload_identity:
|
||||
name: canned-prompts
|
||||
package_type: manifest-managed-platform-service
|
||||
|
|
@ -34,11 +34,11 @@ composition:
|
|||
upstream_components:
|
||||
- name: canned-prompts
|
||||
source: forgejo.coulomb.social/coulomb/canned-prompts
|
||||
# Published 2026-09-07 from canned-prompts service/Dockerfile, tag 0.1.0.
|
||||
# Published 2026-09-07 from canned-prompts service/Dockerfile, tag 0.1.4.
|
||||
# Pinned by digest rather than tag: a tag can be moved, and
|
||||
# live-image-digest-match would then pass against something that is no
|
||||
# longer what this repo reviewed.
|
||||
version: sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123
|
||||
version: sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf
|
||||
rollout_contract:
|
||||
default_mode: kubectl-server-side-apply
|
||||
smoke_contract:
|
||||
|
|
|
|||
48
docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md
Normal file
48
docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
# RCP-WP-0002-T04 — first deployment evidence
|
||||
|
||||
**Date:** 2026-09-08
|
||||
**Image:** `sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf` (tag 0.1.4)
|
||||
**Cluster:** railiance01 / k3s, namespace `canned-prompts`
|
||||
**Schema:** alembic revision 0002, applied by the migration Job as `canned_prompts_migrate` with `SET ROLE canned_prompts_owner`
|
||||
|
||||
## `tools/smoke.sh`
|
||||
|
||||
```text
|
||||
PASS external-secrets-ready:canned-prompts-postgres-runtime
|
||||
PASS external-secrets-ready:canned-prompts-postgres-migration
|
||||
PASS private-service-only:type
|
||||
PASS private-service-only:no-ingress
|
||||
PASS networkpolicies-present
|
||||
PASS live-image-digest-match
|
||||
--- service-level (/home/worsch/canned-prompts/service/tools/smoke.py) ---
|
||||
PASS liveness-ok 200 {'status': 'ok'}
|
||||
PASS readiness-ok 200 ok
|
||||
PASS state-health-ok 200 connected
|
||||
PASS migration-at-head running 0002, expected 0002
|
||||
PASS index-queryable 200
|
||||
PASS registry-queryable 200
|
||||
|
||||
all 6 checks passed
|
||||
|
||||
all deployment checks passed
|
||||
```
|
||||
|
||||
## Read-only posture verified
|
||||
|
||||
`POST /packages` returns **503**, not a 500 and not an acceptance:
|
||||
|
||||
```json
|
||||
{"detail":"publishing is not configured: this service has no publisher identity, so it cannot tell who is calling and refuses writes rather than accepting anonymous publishes (§ 20.1)"}
|
||||
```
|
||||
|
||||
This is the intended state. `creds/canned-prompts-publish` was deliberately not issued (rapp-postgres receipt, 2026-09-08). The read surface answers normally: `/packages` and `/index` both return empty result sets rather than errors.
|
||||
|
||||
## Defects found and fixed during this rollout
|
||||
|
||||
| Defect | Consequence had it shipped |
|
||||
|---|---|
|
||||
| `env.py` read `database_url` rather than `resolved_database_url` | Migration could never run in the cluster, where the credential is a mounted file |
|
||||
| `SET ROLE` opened an implicit transaction Alembic then nested inside | Every migration logged as applied and was silently rolled back — an empty database reported as success |
|
||||
| Egress NetworkPolicy selected `name`, not `part-of` | Migration Job matched only the default-deny; it succeeded once purely because it ran before the policies existed |
|
||||
| Missing optional publish-token file treated as a hard failure | The documented read-only posture returned 500 instead of an explanatory 503 |
|
||||
| `smoke.sh` extracted the digest with a line-offset `grep` | `live-image-digest-match` silently degraded to "not pinned" and could never have passed |
|
||||
|
|
@ -69,22 +69,19 @@ spec:
|
|||
- extract:
|
||||
key: creds/canned-prompts-migration
|
||||
---
|
||||
# The publish token. Absent, the service is read-only — which is the correct
|
||||
# posture until per-publisher identity exists (canned-prompts service/auth.py).
|
||||
apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: canned-prompts-publish-token
|
||||
namespace: canned-prompts
|
||||
spec:
|
||||
refreshInterval: 15m
|
||||
secretStoreRef:
|
||||
kind: ClusterSecretStore
|
||||
name: openbao-canned-prompts-database
|
||||
target:
|
||||
name: canned-prompts-publish-token
|
||||
creationPolicy: Owner
|
||||
deletionPolicy: Retain
|
||||
dataFrom:
|
||||
- extract:
|
||||
key: creds/canned-prompts-publish
|
||||
# The publish token is deliberately NOT declared as an ExternalSecret.
|
||||
#
|
||||
# rapp-postgres confirmed on 2026-09-08 that `creds/canned-prompts-publish`
|
||||
# does not exist and is not being created, at our request. An ExternalSecret
|
||||
# pointing at it would sit permanently unresolved — indistinguishable from a
|
||||
# broken deployment, and an invitation for the next operator to "fix" it by
|
||||
# minting a credential nobody decided to issue.
|
||||
#
|
||||
# Without the secret the service runs READ-ONLY, which is the intended posture
|
||||
# until per-publisher identity exists upstream (RCP-WP-0002-T05). The runtime
|
||||
# Deployment mounts the secret `optional: true`, so it starts cleanly without it
|
||||
# and picks the token up if one is ever added.
|
||||
#
|
||||
# To enable publishing later: have rapp-postgres issue
|
||||
# `creds/canned-prompts-publish`, add the ExternalSecret here, and revisit the
|
||||
# NetworkPolicy ingress rule, which currently admits any namespace.
|
||||
|
|
|
|||
|
|
@ -17,6 +17,10 @@ spec:
|
|||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: canned-prompts-migration
|
||||
# Carries part-of so the egress NetworkPolicy selects this Job too.
|
||||
# Without it the Job matches only the default-deny and cannot reach
|
||||
# PostgreSQL or DNS.
|
||||
app.kubernetes.io/part-of: canned-prompts
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
restartPolicy: Never
|
||||
|
|
@ -29,7 +33,7 @@ spec:
|
|||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: migrate
|
||||
image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123
|
||||
image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf
|
||||
command: ["alembic"]
|
||||
args: ["upgrade", "head"]
|
||||
workingDir: /app
|
||||
|
|
@ -37,6 +41,12 @@ spec:
|
|||
# The migration role owns the schema; the runtime role does not.
|
||||
- name: CANNED_PROMPTS_DATABASE_URL_FILE
|
||||
value: /var/run/secrets/postgres-migration/url
|
||||
# Authenticate as the leased migration login, then SET ROLE to the
|
||||
# durable owner before creating anything. Leases are revoked; an
|
||||
# object owned by a dead login has to be normalized afterwards.
|
||||
# Required by the rapp-postgres database-owner boundary.
|
||||
- name: CANNED_PROMPTS_MIGRATION_ROLE
|
||||
value: canned_prompts_owner
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ spec:
|
|||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: canned-prompts
|
||||
image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123
|
||||
image: forgejo.coulomb.social/coulomb/canned-prompts@sha256:2fbac3c0d1609a76d9478e540ae5450c5425e8c04e0565cccc6a4eb0180ebcaf
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- name: http
|
||||
|
|
@ -134,19 +134,36 @@ spec:
|
|||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: canned-prompts-runtime
|
||||
name: canned-prompts-ingress
|
||||
namespace: canned-prompts
|
||||
spec:
|
||||
# Ingress is for the serving pods only. The migration Job serves nothing and
|
||||
# must not be reachable.
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: canned-prompts
|
||||
policyTypes: [Ingress, Egress]
|
||||
policyTypes: [Ingress]
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector: {}
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8000
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: canned-prompts-egress
|
||||
namespace: canned-prompts
|
||||
spec:
|
||||
# Selects on part-of, not name, so it covers the migration Job as well as the
|
||||
# serving pods. Selecting only `name: canned-prompts` left the Job matched by
|
||||
# nothing but the default-deny — it succeeded once purely because it ran
|
||||
# before these policies existed, and the next migration would have failed.
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/part-of: canned-prompts
|
||||
policyTypes: [Egress]
|
||||
egress:
|
||||
# PostgreSQL and DNS only. The service fetches nothing: a package arrives
|
||||
# by publish, never by the registry reaching out, so it needs no egress to
|
||||
|
|
|
|||
|
|
@ -11,7 +11,15 @@
|
|||
set -euo pipefail
|
||||
|
||||
NS=${NS:-canned-prompts}
|
||||
EXPECT_DIGEST=${EXPECT_DIGEST:-$(grep -A1 'name: canned-prompts$' declarations/rapp.yaml | grep 'version:' | awk '{print $2}' || true)}
|
||||
# Parsed, not grepped. A line-offset grep silently returned empty once comments
|
||||
# were added above `version:`, and the check then degraded to "not pinned yet"
|
||||
# instead of failing — it could not have passed for any pin.
|
||||
EXPECT_DIGEST=${EXPECT_DIGEST:-$(python3 -c "
|
||||
import yaml, sys
|
||||
d = yaml.safe_load(open('declarations/rapp.yaml'))
|
||||
c = [u for u in d['composition']['upstream_components'] if u['name'] == 'canned-prompts']
|
||||
print(c[0]['version'] if c else '')
|
||||
" 2>/dev/null || true)}
|
||||
FAILED=0
|
||||
|
||||
check() { # name, condition-output
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "First deployment of canned-prompts on Railiance"
|
||||
domain: agents
|
||||
repo: rapp-canned-prompts
|
||||
status: active
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: practice
|
||||
created: "2026-09-06"
|
||||
|
|
@ -62,7 +62,7 @@ output. `readiness_state` moved `draft` → `declared`.
|
|||
|
||||
```task
|
||||
id: RCP-WP-0002-T02
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "61f5cbd7-8fcf-5156-87da-57239ae55d8f"
|
||||
```
|
||||
|
|
@ -91,13 +91,19 @@ OpenBao credentials — was deliberately **not** run: credential issuance is
|
|||
`rapp-postgres`' to perform, and running it from the consuming side would take a
|
||||
decision that is not this repo's, however available the script happens to be.
|
||||
|
||||
Blocked here until those credentials exist. T03 and T04 are blocked behind it.
|
||||
**Received 2026-09-08.** `rapp-postgres` provisioned and sent a database-owner
|
||||
receipt with 12 checks proven live: runtime DDL denied (SQLSTATE 42501),
|
||||
statement timeouts and `search_path` as declared, and both logins refused
|
||||
CONNECT on `sbom_nexus` — the cell is shared, so that last one matters.
|
||||
|
||||
`creds/canned-prompts-publish` was **not** created, as asked. The service runs
|
||||
read-only, which is the intended posture rather than a gap.
|
||||
|
||||
## Apply and migrate
|
||||
|
||||
```task
|
||||
id: RCP-WP-0002-T03
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "0b8d206e-bd28-5950-abf7-d824015c09a4"
|
||||
```
|
||||
|
|
@ -120,11 +126,28 @@ The runtime deliberately does not migrate at start-up. Migrations as a Job keep
|
|||
a schema rollback separate from a code rollback and stop replicas racing each
|
||||
other.
|
||||
|
||||
**Done 2026-09-08, after four defects that only a real rollout could expose.**
|
||||
Each is recorded in
|
||||
`docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md`; the two worth
|
||||
repeating here failed *silently*:
|
||||
|
||||
- `SET ROLE` opened an implicit transaction that Alembic then nested inside
|
||||
rather than owning, so every revision logged as applied and was rolled back.
|
||||
Alembic reported success against an empty database.
|
||||
- The egress NetworkPolicy selected `app.kubernetes.io/name`, which the
|
||||
migration Job does not carry. The Job matched only the default-deny and
|
||||
succeeded exactly once — because it ran before the policies existed. The next
|
||||
migration would have failed with a DNS error. Now selects `part-of`, and
|
||||
ingress is a separate policy so the Job is never reachable.
|
||||
|
||||
`rapp-postgres` asked to be told when the first revision landed so they can
|
||||
re-run ownership reconciliation. Notified.
|
||||
|
||||
## Verify and record evidence
|
||||
|
||||
```task
|
||||
id: RCP-WP-0002-T04
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "6d9eb97c-57e2-5b74-b6eb-455076713417"
|
||||
```
|
||||
|
|
@ -138,6 +161,19 @@ second opinion about whether the service is healthy.
|
|||
Record the output as evidence, then move `readiness_state` to `deployed`, and to
|
||||
`verified` only with that evidence attached.
|
||||
|
||||
**Done 2026-09-08.** All six deployment checks and all six service-level checks
|
||||
pass; evidence at
|
||||
`docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md`.
|
||||
`readiness_state` is `verified`, with the evidence attached rather than ahead of
|
||||
it.
|
||||
|
||||
**The check that could never have passed.** `live-image-digest-match` read the
|
||||
pin with a line-offset `grep`, which returned empty once comments were added
|
||||
above `version:` — and the check then degraded to reporting "not pinned yet"
|
||||
instead of failing. It reported that while a digest *was* pinned. Now parsed as
|
||||
YAML. A verification step that cannot fail is worth less than none, because it
|
||||
is trusted.
|
||||
|
||||
## Decide per-publisher identity
|
||||
|
||||
```task
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue