Plan PostgreSQL production store (TEN-WP-0009)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

WORK-RECORDS.md

@@ -16,6 +16,7 @@
 | workplan | TEN-WP-0005 | finished | — | workplans/TEN-WP-0005-tenant-update-and-retirement-api.md |
 | workplan | TEN-WP-0006 | finished | — | workplans/TEN-WP-0006-guardrail-quota-policy.md |
 | workplan | TEN-WP-0007 | finished | — | workplans/TEN-WP-0007-guardrail-production-rollout.md |
+| workplan | TEN-WP-0008 | ready | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md |
 | task | ADHOC-2026-07-24-T01 | done | — | workplans/ADHOC-2026-07-24.md |
 | task | TEN-WP-0001-T01 | done | — | workplans/TEN-WP-0001-statehub-bootstrap.md |
 | task | TEN-WP-0001-T02 | done | — | workplans/TEN-WP-0001-statehub-bootstrap.md |
@@ -48,3 +49,7 @@
 | task | TEN-WP-0007-T02 | done | — | workplans/TEN-WP-0007-guardrail-production-rollout.md |
 | task | TEN-WP-0007-T03 | done | — | workplans/TEN-WP-0007-guardrail-production-rollout.md |
 | task | TEN-WP-0007-T04 | done | — | workplans/TEN-WP-0007-guardrail-production-rollout.md |
+| task | TEN-WP-0008-T01 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md |
+| task | TEN-WP-0008-T02 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md |
+| task | TEN-WP-0008-T03 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md |
+| task | TEN-WP-0008-T04 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md |

workplans/TEN-WP-0008-staged-promotion-onboarding.md

@@ -12,6 +12,7 @@ updated: "2026-08-16"
 depends_on:
   - TEN-WP-0007
 unblocks: []
+state_hub_workstream_id: "e9da1c15-7beb-48d9-97ab-b657fdf9366e"
 ---
 
 # TEN-WP-0008 - Staged-promotion onboarding
@@ -60,6 +61,7 @@ is a `railiance-bootstrap` decision to request, not one to make here.
 id: TEN-WP-0008-T01
 status: todo
 priority: high
+state_hub_task_id: "0231e4d1-2e1a-4e77-92ba-2cfaf23692ab"
 ```
 
 Author `railiance/app.toml` against `railiance.app.v1`
@@ -99,6 +101,7 @@ mode.
 id: TEN-WP-0008-T02
 status: todo
 priority: high
+state_hub_task_id: "5f2ad011-1de8-45df-8e8f-dd6bcf51ece4"
 ```
 
 Expected friction, and the real work of this workplan. The stage 3 promote path
@@ -133,6 +136,7 @@ this repo's deployment shape, verified in plan mode.
 id: TEN-WP-0008-T03
 status: todo
 priority: medium
+state_hub_task_id: "c4b79dc1-c280-428e-9dc5-b17c05b084c6"
 ```
 
 Register the live digest `@sha256:44ca65f3…` as the recorded stable release, so
@@ -150,6 +154,7 @@ rather than only in `deploy/tenant-engine.yaml`.
 id: TEN-WP-0008-T04
 status: todo
 priority: low
+state_hub_task_id: "b816a862-51e0-4abd-a2f3-ee40359526cc"
 ```
 
 Tell `flex-auth` what onboarding actually cost, including anything the contract

workplans/TEN-WP-0009-postgres-production-store.md

@@ -0,0 +1,246 @@
+---
+id: TEN-WP-0009
+type: workplan
+title: "PostgreSQL as the production store, SQLite for dev and test"
+domain: infotech
+repo: tenant-engine
+status: ready
+owner: claude
+topic_slug: tenant-lifecycle
+created: "2026-08-16"
+updated: "2026-08-16"
+depends_on:
+  - TEN-WP-0007
+unblocks:
+  - TEN-WP-0008
+---
+
+# TEN-WP-0009 - PostgreSQL production store
+
+Move the production store from SQLite-on-a-PVC to the fleet's CloudNativePG
+infrastructure. SQLite stays, deliberately, as the dev and test backend.
+
+## Why this reverses an earlier decision
+
+TEN-WP-0005-T02 chose SQLite over PostgreSQL on purpose, and the reasoning was
+sound at the time: TEN-WP-0004 had shipped SQLite on a PVC, so *"adding an
+unused Postgres path would have been dead code"*. The `TenantStore` Protocol
+was kept precisely as the seam for this change.
+
+Three things have changed since:
+
+1. **The PVC is now blocking work.** The volume is `ReadWriteOnce`, which forces
+   `strategy: Recreate` and makes a side-by-side canary impossible — the
+   constraint TEN-WP-0008-T02 ran into when onboarding to staged promotion. The
+   store choice is no longer invisible to anything outside this repo.
+2. **The infrastructure is already there and already proven.** The
+   CloudNativePG operator runs on railiance01 with seven clusters, and
+   `user-engine` — our sibling service, sharing the `tenant_id` key — has run
+   its own `user-engine-pg` for 19 days.
+3. **A Postgres path is no longer unused.** It becomes the production path, so
+   the dead-code objection no longer applies.
+
+SQLite is *not* being removed. It stays as the dev and test backend, where a
+zero-setup file-backed store is genuinely the right tool, and it keeps the
+conformance suite honest by forcing the Protocol to stay a real seam rather
+than a formality wrapped around one implementation.
+
+## Credential handling
+
+CloudNativePG mints the connection secret itself: a cluster named `X` produces
+secret `X-app` with key `uri`. The Deployment references it via
+`secretKeyRef`, exactly as `user-engine` does with
+`USER_ENGINE_DATABASE_URL` ← `user-engine-pg-app/uri`.
+
+So **no secret enters this repo, this workplan, or any evidence dump**, and no
+`warden route` lookup is needed — this is not a credential we fetch, custody,
+or rotate. Runtime secret custody remains OpenBao's per `SCOPE.md`; an
+operator-minted in-cluster secret is outside that lane.
+
+## T01 - Decide the topology and write the cluster manifest
+
+```task
+id: TEN-WP-0009-T01
+status: todo
+priority: high
+```
+
+Decide between a dedicated `tenant-engine-pg` cluster in our namespace and
+sharing an existing one (`net-kingdom-pg` or `apps-pg` in `databases`).
+
+Argue it rather than defaulting. Points to weigh: `user-engine` chose dedicated
+and is the closest precedent; a dedicated cluster keeps blast radius and backup
+policy per-service; a shared cluster is cheaper on a single node and
+`net-kingdom-pg` already exists for this product family. Note that
+`tenant-engine` is consulted by `flex-auth` on the authorization path, so its
+availability envelope is not purely its own.
+
+Write the CNPG `Cluster` manifest into `deploy/`, matching the fleet's existing
+image and storage conventions rather than inventing new ones.
+
+Also decide **backups**. The `scheduledbackups.postgresql.cnpg.io` CRD is
+available; SQLite-on-a-PVC had no backup story at all, so this is a gain to
+claim explicitly rather than inherit by accident.
+
+Done when the topology decision is recorded with its reasoning and the manifest
+is committed.
+
+## T02 - Implement `PostgresTenantStore`
+
+```task
+id: TEN-WP-0009-T02
+status: todo
+priority: high
+```
+
+Implement the full `TenantStore` Protocol — including
+`guardrail_overrides` / `guardrail_changes` / `set_guardrail_override` from
+TEN-WP-0006 — and add PostgreSQL to the conformance suites, which are already
+parametrised over in-memory and SQLite. A third backend should be a fixture
+parameter, not a new test file.
+
+Invariants that must hold identically, not approximately:
+
+- `mutate_tenant()` commits idempotency replay, version CAS, mutation, and
+  audit event **in one transaction**;
+- `set_guardrail_override()` does the same for the override, its audit record,
+  and its receipt;
+- a replayed `Idempotency-Key` short-circuits *before* the version check;
+- reads and writes never observe a row mid-transaction.
+
+That last one is where PostgreSQL should simplify rather than complicate:
+SQLite needed an explicit `RLock` around reads because one connection is shared
+across request threads, and the concurrent-writer test caught a real defect
+there. Postgres gives proper per-connection transaction isolation, so the lock
+should disappear rather than be ported. **Do not transliterate the SQLite
+implementation** — port the semantics and let the backend do what it is good
+at. Use a connection pool and appropriate isolation, and prove the guarantees
+with the same concurrent-writer test that caught the SQLite bug.
+
+Adds a runtime dependency (`psycopg`); keep it out of the base install path if
+SQLite-only dev is meant to stay dependency-light.
+
+Done when all three backends pass the same conformance suites unchanged.
+
+## T03 - Backend selection that fails closed
+
+```task
+id: TEN-WP-0009-T03
+status: todo
+priority: high
+```
+
+`Settings` currently carries only `database_path`. Add a database URL and make
+selection explicit:
+
+- URL set → PostgreSQL;
+- path set → SQLite;
+- **both set → refuse to start.** Do not silently prefer one. An ambiguous
+  store configuration in production is exactly the class of silent
+  misconfiguration this repo fails closed on everywhere else, and picking a
+  winner would let a stale `TENANT_ENGINE_DATABASE_PATH` quietly shadow the
+  real database;
+- neither set → in-memory, as today, which is correct for tests and wrong for
+  production.
+
+Consider surfacing the active backend on `/health`. The last two production
+incidents were invisible because the service looked fine from outside; "which
+store am I actually using" is cheap to answer and expensive to guess.
+
+Done when misconfiguration is a startup failure with a clear message, covered
+by tests.
+
+## T04 - Migrate production data
+
+```task
+id: TEN-WP-0009-T04
+status: todo
+priority: high
+```
+
+Move the live SQLite database to PostgreSQL. Currently small — two tenants
+(`tenant:trial:portalcheck` active v1, `tenant:trial:ten-wp-0005-t05` retired
+v5) plus the T04 disposables from TEN-WP-0007 — but small is not the same as
+trivial.
+
+Everything must move, not just the tenants:
+
+- `tenants` with versions and lifecycle timestamps **preserved exactly** — a
+  reset version silently breaks every consumer holding an ETag;
+- `grants` including revoked ones, because the trail is append-only and its
+  history is the audit record;
+- `plans`;
+- `events` — the audit correlation contract;
+- `idempotency_receipts` — dropping these lets an in-flight retry double-apply
+  a mutation that already happened;
+- `guardrail_overrides` and `guardrail_changes`.
+
+SQLite is single-writer and the app writes to it, so the cutover must stop
+writes rather than race them: scale to zero, export, import, verify, then start
+against PostgreSQL. Verify by comparing row counts *and* spot-checking the two
+known tenants' versions and lifecycle state, not by trusting the exporter.
+
+Keep the SQLite file and its PVC until T05's soak passes. It is the rollback.
+
+Done when a verified copy is live in PostgreSQL and the comparison evidence is
+recorded here.
+
+## T05 - Cut over the deployment
+
+```task
+id: TEN-WP-0009-T05
+status: todo
+priority: high
+```
+
+Update `deploy/tenant-engine.yaml`:
+
+- drop the PVC and its volume mount;
+- add `TENANT_ENGINE_DATABASE_URL` from the CNPG-minted `secretKeyRef`;
+- switch `strategy: Recreate` → `RollingUpdate`, now that no `ReadWriteOnce`
+  volume forces serialization — **this is what unblocks TEN-WP-0008-T02's
+  canary**;
+- extend the `NetworkPolicy` egress to the Postgres service, and no wider.
+  Ingress stays restricted to `user-engine`.
+
+Then decide, deliberately, whether to raise `replicas` above 1. It becomes
+possible here for the first time; possible is not the same as warranted, and a
+second replica changes failure modes for a service on the authorization path.
+
+Update `make verify-pin` if the expected shape changes, and re-run it.
+
+Rollback: re-pin the previous digest and re-attach the PVC. State plainly that
+this becomes lossy once writes have landed in PostgreSQL — after that point the
+rollback is "restore from Postgres", not "swap back to the file".
+
+Done when production runs on PostgreSQL, `make verify-pin` passes, and the
+TEN-WP-0007 T04 verification list still passes end to end against the live
+service.
+
+## T06 - Tell the fleet, and close the loop on TEN-WP-0008
+
+```task
+id: TEN-WP-0009-T06
+status: todo
+priority: medium
+```
+
+- `flex-auth`: availability characteristics of their data source changed. No
+  contract or action change.
+- `user-engine`: no contract change either, but they run the same CNPG pattern
+  and may have operational lessons worth having before we repeat their mistakes.
+- **TEN-WP-0008-T02**: record that the `ReadWriteOnce` canary constraint is
+  gone, so staged-promotion onboarding no longer has to describe a canary that
+  cannot run.
+
+Done when the notes are sent and TEN-WP-0008-T02 is updated.
+
+## Out of scope
+
+- **Removing SQLite.** It stays as the dev and test backend, and as the thing
+  that keeps `TenantStore` an honest seam.
+- **Schema redesign.** Port the existing schema; a store migration and a model
+  change at the same time makes any regression ambiguous.
+- **Sharing a database with `user-engine`.** `tenant_id` is the only key the
+  two services share, and it stays that way — separate stores are the boundary,
+  not an implementation detail.