rapp-canned-prompts/workplans/RCP-WP-0002-first-deployment.md

---
id: RCP-WP-0002
type: workplan
title: "First deployment of canned-prompts on Railiance"
domain: agents
repo: rapp-canned-prompts
status: active
owner: codex
topic_slug: practice
created: "2026-09-06"
updated: "2026-09-06"
state_hub_workstream_id: "11874f32-ac36-5bb9-a0a5-e7a259f5972c"
---

# First deployment of canned-prompts on Railiance

The package is written and validated; what remains needs credentials and a
published image, which are operator actions rather than authoring ones.

`readiness_state` is `draft` and stays there until T04 produces evidence.
Topology is not readiness (ADR-0006): binding this rapp to a reef does not make
it deployed.

## Publish the image and pin its digest

```task
id: RCP-WP-0002-T01
status: done
priority: high
state_hub_task_id: "3e7faf50-8fe9-5ef3-8a75-1d9e586d6c0f"
```

`declarations/rapp.yaml` carries `version: pending-publication`, and both
manifests carry a matching tag. This is deliberate: a placeholder shaped like a
real digest could be mistaken for something deployable, and a rapp that pins
nothing is a stub that tells the fleet tooling a deployment exists when none
does.

The image builds and was verified locally in `canned-prompts`
(`CANP-WP-0006-T06`): it starts non-root, all six service-level smoke checks
pass against the running container, and the reference CLI installs a package
from it over HTTP.

**Done, 2026-09-07.** Published as
`forgejo.coulomb.social/coulomb/canned-prompts:0.1.0` and pinned by **digest**
in the declaration and both manifests:

```
sha256:e0ded3c7fe2548c910445deaa31ec42e84f129a92aa324841e231a53fee1f123
```

Pinned by digest rather than tag deliberately: a tag can be moved, and
`live-image-digest-match` would then pass against something that is no longer
what this repo reviewed.

Verified before pushing — the image resolves a file-mounted credential, carries
the reference validator, and ships the migration scripts — and verified after,
by fetching the manifest back **by digest** rather than trusting the push
output. `readiness_state` moved `draft` → `declared`.

## Provision database roles and credentials

```task
id: RCP-WP-0002-T02
status: done
priority: high
state_hub_task_id: "61f5cbd7-8fcf-5156-87da-57239ae55d8f"
```

Two roles, deliberately separate: a runtime role that reads and writes rows, and
a migration role that owns the schema. A service able to `ALTER` its own tables
at runtime turns any code defect into a schema defect.

Needed from `rapp-postgres` and the credential broker:

- database `canned_prompts` on `platform-pg-2`;
- `creds/canned-prompts-runtime` and `creds/canned-prompts-migration` in OpenBao
  under the `database` path;
- the `openbao-canned-prompts-eso-token` secret in `external-secrets`.

`creds/canned-prompts-publish` is **optional**. Without it the service is
read-only, which is the correct posture until per-publisher identity exists —
not a misconfiguration to be worked around.

**Requested, 2026-09-07.** `rapp-postgres` now carries
`consumers/canned-prompts.yaml`, authored against its documented
`PostgresConsumer` shape, and its agent has the request in its inbox.

The declaration only. `make provision-consumers` — the step that actually mints
OpenBao credentials — was deliberately **not** run: credential issuance is
`rapp-postgres`' to perform, and running it from the consuming side would take a
decision that is not this repo's, however available the script happens to be.

**Received 2026-09-08.** `rapp-postgres` provisioned and sent a database-owner
receipt with 12 checks proven live: runtime DDL denied (SQLSTATE 42501),
statement timeouts and `search_path` as declared, and both logins refused
CONNECT on `sbom_nexus` — the cell is shared, so that last one matters.

`creds/canned-prompts-publish` was **not** created, as asked. The service runs
read-only, which is the intended posture rather than a gap.

## Apply and migrate

```task
id: RCP-WP-0002-T03
status: done
priority: high
state_hub_task_id: "0b8d206e-bd28-5950-abf7-d824015c09a4"
```

Order matters, and the ordering is the point rather than a convenience:

Verified against the live cluster before writing: `platform-pg-2` exists and is
healthy, and the `railiance.io/postgres-client: platform-pg-2` label matches
what `sbom-nexus` actually carries in the cluster rather than only what its repo
says.

1. `manifests/00-namespace.yaml` — the `railiance.io/postgres-client` label is
   what lets the database namespace accept traffic;
2. `manifests/database-secrets.yaml`, then wait for the secrets to materialize;
3. `manifests/migration.yaml` — the Job runs `alembic upgrade head` under the
   migration role and must complete before any replica serves;
4. `manifests/runtime.yaml`.

The runtime deliberately does not migrate at start-up. Migrations as a Job keep
a schema rollback separate from a code rollback and stop replicas racing each
other.

**Done 2026-09-08, after four defects that only a real rollout could expose.**
Each is recorded in
`docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md`; the two worth
repeating here failed *silently*:

- `SET ROLE` opened an implicit transaction that Alembic then nested inside
  rather than owning, so every revision logged as applied and was rolled back.
  Alembic reported success against an empty database.
- The egress NetworkPolicy selected `app.kubernetes.io/name`, which the
  migration Job does not carry. The Job matched only the default-deny and
  succeeded exactly once — because it ran before the policies existed. The next
  migration would have failed with a DNS error. Now selects `part-of`, and
  ingress is a separate policy so the Job is never reachable.

`rapp-postgres` asked to be told when the first revision landed so they can
re-run ownership reconciliation. Notified.

## Verify and record evidence

```task
id: RCP-WP-0002-T04
status: done
priority: high
state_hub_task_id: "6d9eb97c-57e2-5b74-b6eb-455076713417"
```

Run `tools/smoke.sh`. It checks what only the cluster can answer — secrets
materialized, Service is ClusterIP with no Ingress, NetworkPolicies present,
live image digest matches the pin — and calls `canned-prompts`'
`service/tools/smoke.py` for health and migration head rather than holding a
second opinion about whether the service is healthy.

Record the output as evidence, then move `readiness_state` to `deployed`, and to
`verified` only with that evidence attached.

**Done 2026-09-08.** All six deployment checks and all six service-level checks
pass; evidence at
`docs/evidence/RCP-WP-0002-T04-first-deployment-2026-09-08.md`.
**Then corrected.** I recorded `verified` after a passing smoke run and found
the deployment `0/1` eight hours later: platform credentials are 30-minute
leases, and the service read the mounted URL once at start-up. External Secrets
kept the file current; the engine held the URL it booted with. Fixed in 0.1.5 by
re-reading the credential for every new connection, with `pool_recycle` inside
the lease TTL.

`readiness_state` was returned to `deployed`, and is now **`verified`** on
evidence: continuously ready across a full lease rotation, zero readiness
failures, no restarts. Log at
`docs/evidence/RCP-WP-0002-T04-lease-rotation-2026-09-08.log`.

The bar for `verified` is observation across a rotation, because a smoke run
inside the first window cannot tell a service that works from one that works
*once* — an availability property is not provable by a single sample. It took
three attempts to build an instrument that could return that verdict honestly;
all three failures are recorded in the evidence file.

**The check that could never have passed.** `live-image-digest-match` read the
pin with a line-offset `grep`, which returned empty once comments were added
above `version:` — and the check then degraded to reporting "not pinned yet"
instead of failing. It reported that while a digest *was* pinned. Now parsed as
YAML. A verification step that cannot fail is worth less than none, because it
is trusted.

## Decide per-publisher identity

```task
id: RCP-WP-0002-T05
status: wait
priority: medium
state_hub_task_id: "1252f3a1-12ce-5bc1-b003-3e8999d03459"
```

Blocked on a decision in `canned-prompts`, recorded here because it is the
thing that decides what this deployment is *for*.

Today the service authenticates a single shared bearer token proving "the
operator". That is adequate for a private in-cluster registry and inadequate for
the collaborative prompting platform the operator described: every token holder
is indistinguishable, so § 20.1 namespace ownership can be enforced against
anonymous callers but not attributed among publishers.

This repo must not paper over that with cluster configuration implying finer
control than exists. When identity lands upstream, revisit the publish-token
secret and the NetworkPolicy ingress rule, which currently admits any namespace.