From 781359e7d4e6717035146d2889f721de15015449 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 16 Aug 2026 18:25:35 +0200 Subject: [PATCH] Plan PostgreSQL production store (TEN-WP-0009) Co-Authored-By: Claude Opus 5 --- WORK-RECORDS.md | 5 + ...TEN-WP-0008-staged-promotion-onboarding.md | 5 + .../TEN-WP-0009-postgres-production-store.md | 246 ++++++++++++++++++ 3 files changed, 256 insertions(+) create mode 100644 workplans/TEN-WP-0009-postgres-production-store.md diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 4218ec1..46ca33d 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -16,6 +16,7 @@ | workplan | TEN-WP-0005 | finished | — | workplans/TEN-WP-0005-tenant-update-and-retirement-api.md | | workplan | TEN-WP-0006 | finished | — | workplans/TEN-WP-0006-guardrail-quota-policy.md | | workplan | TEN-WP-0007 | finished | — | workplans/TEN-WP-0007-guardrail-production-rollout.md | +| workplan | TEN-WP-0008 | ready | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md | | task | ADHOC-2026-07-24-T01 | done | — | workplans/ADHOC-2026-07-24.md | | task | TEN-WP-0001-T01 | done | — | workplans/TEN-WP-0001-statehub-bootstrap.md | | task | TEN-WP-0001-T02 | done | — | workplans/TEN-WP-0001-statehub-bootstrap.md | @@ -48,3 +49,7 @@ | task | TEN-WP-0007-T02 | done | — | workplans/TEN-WP-0007-guardrail-production-rollout.md | | task | TEN-WP-0007-T03 | done | — | workplans/TEN-WP-0007-guardrail-production-rollout.md | | task | TEN-WP-0007-T04 | done | — | workplans/TEN-WP-0007-guardrail-production-rollout.md | +| task | TEN-WP-0008-T01 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md | +| task | TEN-WP-0008-T02 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md | +| task | TEN-WP-0008-T03 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md | +| task | TEN-WP-0008-T04 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md | diff --git a/workplans/TEN-WP-0008-staged-promotion-onboarding.md b/workplans/TEN-WP-0008-staged-promotion-onboarding.md index cf4efe1..e436609 100644 --- a/workplans/TEN-WP-0008-staged-promotion-onboarding.md +++ b/workplans/TEN-WP-0008-staged-promotion-onboarding.md @@ -12,6 +12,7 @@ updated: "2026-08-16" depends_on: - TEN-WP-0007 unblocks: [] +state_hub_workstream_id: "e9da1c15-7beb-48d9-97ab-b657fdf9366e" --- # TEN-WP-0008 - Staged-promotion onboarding @@ -60,6 +61,7 @@ is a `railiance-bootstrap` decision to request, not one to make here. id: TEN-WP-0008-T01 status: todo priority: high +state_hub_task_id: "0231e4d1-2e1a-4e77-92ba-2cfaf23692ab" ``` Author `railiance/app.toml` against `railiance.app.v1` @@ -99,6 +101,7 @@ mode. id: TEN-WP-0008-T02 status: todo priority: high +state_hub_task_id: "5f2ad011-1de8-45df-8e8f-dd6bcf51ece4" ``` Expected friction, and the real work of this workplan. The stage 3 promote path @@ -133,6 +136,7 @@ this repo's deployment shape, verified in plan mode. id: TEN-WP-0008-T03 status: todo priority: medium +state_hub_task_id: "c4b79dc1-c280-428e-9dc5-b17c05b084c6" ``` Register the live digest `@sha256:44ca65f3…` as the recorded stable release, so @@ -150,6 +154,7 @@ rather than only in `deploy/tenant-engine.yaml`. id: TEN-WP-0008-T04 status: todo priority: low +state_hub_task_id: "b816a862-51e0-4abd-a2f3-ee40359526cc" ``` Tell `flex-auth` what onboarding actually cost, including anything the contract diff --git a/workplans/TEN-WP-0009-postgres-production-store.md b/workplans/TEN-WP-0009-postgres-production-store.md new file mode 100644 index 0000000..507fee5 --- /dev/null +++ b/workplans/TEN-WP-0009-postgres-production-store.md @@ -0,0 +1,246 @@ +--- +id: TEN-WP-0009 +type: workplan +title: "PostgreSQL as the production store, SQLite for dev and test" +domain: infotech +repo: tenant-engine +status: ready +owner: claude +topic_slug: tenant-lifecycle +created: "2026-08-16" +updated: "2026-08-16" +depends_on: + - TEN-WP-0007 +unblocks: + - TEN-WP-0008 +--- + +# TEN-WP-0009 - PostgreSQL production store + +Move the production store from SQLite-on-a-PVC to the fleet's CloudNativePG +infrastructure. SQLite stays, deliberately, as the dev and test backend. + +## Why this reverses an earlier decision + +TEN-WP-0005-T02 chose SQLite over PostgreSQL on purpose, and the reasoning was +sound at the time: TEN-WP-0004 had shipped SQLite on a PVC, so *"adding an +unused Postgres path would have been dead code"*. The `TenantStore` Protocol +was kept precisely as the seam for this change. + +Three things have changed since: + +1. **The PVC is now blocking work.** The volume is `ReadWriteOnce`, which forces + `strategy: Recreate` and makes a side-by-side canary impossible — the + constraint TEN-WP-0008-T02 ran into when onboarding to staged promotion. The + store choice is no longer invisible to anything outside this repo. +2. **The infrastructure is already there and already proven.** The + CloudNativePG operator runs on railiance01 with seven clusters, and + `user-engine` — our sibling service, sharing the `tenant_id` key — has run + its own `user-engine-pg` for 19 days. +3. **A Postgres path is no longer unused.** It becomes the production path, so + the dead-code objection no longer applies. + +SQLite is *not* being removed. It stays as the dev and test backend, where a +zero-setup file-backed store is genuinely the right tool, and it keeps the +conformance suite honest by forcing the Protocol to stay a real seam rather +than a formality wrapped around one implementation. + +## Credential handling + +CloudNativePG mints the connection secret itself: a cluster named `X` produces +secret `X-app` with key `uri`. The Deployment references it via +`secretKeyRef`, exactly as `user-engine` does with +`USER_ENGINE_DATABASE_URL` ← `user-engine-pg-app/uri`. + +So **no secret enters this repo, this workplan, or any evidence dump**, and no +`warden route` lookup is needed — this is not a credential we fetch, custody, +or rotate. Runtime secret custody remains OpenBao's per `SCOPE.md`; an +operator-minted in-cluster secret is outside that lane. + +## T01 - Decide the topology and write the cluster manifest + +```task +id: TEN-WP-0009-T01 +status: todo +priority: high +``` + +Decide between a dedicated `tenant-engine-pg` cluster in our namespace and +sharing an existing one (`net-kingdom-pg` or `apps-pg` in `databases`). + +Argue it rather than defaulting. Points to weigh: `user-engine` chose dedicated +and is the closest precedent; a dedicated cluster keeps blast radius and backup +policy per-service; a shared cluster is cheaper on a single node and +`net-kingdom-pg` already exists for this product family. Note that +`tenant-engine` is consulted by `flex-auth` on the authorization path, so its +availability envelope is not purely its own. + +Write the CNPG `Cluster` manifest into `deploy/`, matching the fleet's existing +image and storage conventions rather than inventing new ones. + +Also decide **backups**. The `scheduledbackups.postgresql.cnpg.io` CRD is +available; SQLite-on-a-PVC had no backup story at all, so this is a gain to +claim explicitly rather than inherit by accident. + +Done when the topology decision is recorded with its reasoning and the manifest +is committed. + +## T02 - Implement `PostgresTenantStore` + +```task +id: TEN-WP-0009-T02 +status: todo +priority: high +``` + +Implement the full `TenantStore` Protocol — including +`guardrail_overrides` / `guardrail_changes` / `set_guardrail_override` from +TEN-WP-0006 — and add PostgreSQL to the conformance suites, which are already +parametrised over in-memory and SQLite. A third backend should be a fixture +parameter, not a new test file. + +Invariants that must hold identically, not approximately: + +- `mutate_tenant()` commits idempotency replay, version CAS, mutation, and + audit event **in one transaction**; +- `set_guardrail_override()` does the same for the override, its audit record, + and its receipt; +- a replayed `Idempotency-Key` short-circuits *before* the version check; +- reads and writes never observe a row mid-transaction. + +That last one is where PostgreSQL should simplify rather than complicate: +SQLite needed an explicit `RLock` around reads because one connection is shared +across request threads, and the concurrent-writer test caught a real defect +there. Postgres gives proper per-connection transaction isolation, so the lock +should disappear rather than be ported. **Do not transliterate the SQLite +implementation** — port the semantics and let the backend do what it is good +at. Use a connection pool and appropriate isolation, and prove the guarantees +with the same concurrent-writer test that caught the SQLite bug. + +Adds a runtime dependency (`psycopg`); keep it out of the base install path if +SQLite-only dev is meant to stay dependency-light. + +Done when all three backends pass the same conformance suites unchanged. + +## T03 - Backend selection that fails closed + +```task +id: TEN-WP-0009-T03 +status: todo +priority: high +``` + +`Settings` currently carries only `database_path`. Add a database URL and make +selection explicit: + +- URL set → PostgreSQL; +- path set → SQLite; +- **both set → refuse to start.** Do not silently prefer one. An ambiguous + store configuration in production is exactly the class of silent + misconfiguration this repo fails closed on everywhere else, and picking a + winner would let a stale `TENANT_ENGINE_DATABASE_PATH` quietly shadow the + real database; +- neither set → in-memory, as today, which is correct for tests and wrong for + production. + +Consider surfacing the active backend on `/health`. The last two production +incidents were invisible because the service looked fine from outside; "which +store am I actually using" is cheap to answer and expensive to guess. + +Done when misconfiguration is a startup failure with a clear message, covered +by tests. + +## T04 - Migrate production data + +```task +id: TEN-WP-0009-T04 +status: todo +priority: high +``` + +Move the live SQLite database to PostgreSQL. Currently small — two tenants +(`tenant:trial:portalcheck` active v1, `tenant:trial:ten-wp-0005-t05` retired +v5) plus the T04 disposables from TEN-WP-0007 — but small is not the same as +trivial. + +Everything must move, not just the tenants: + +- `tenants` with versions and lifecycle timestamps **preserved exactly** — a + reset version silently breaks every consumer holding an ETag; +- `grants` including revoked ones, because the trail is append-only and its + history is the audit record; +- `plans`; +- `events` — the audit correlation contract; +- `idempotency_receipts` — dropping these lets an in-flight retry double-apply + a mutation that already happened; +- `guardrail_overrides` and `guardrail_changes`. + +SQLite is single-writer and the app writes to it, so the cutover must stop +writes rather than race them: scale to zero, export, import, verify, then start +against PostgreSQL. Verify by comparing row counts *and* spot-checking the two +known tenants' versions and lifecycle state, not by trusting the exporter. + +Keep the SQLite file and its PVC until T05's soak passes. It is the rollback. + +Done when a verified copy is live in PostgreSQL and the comparison evidence is +recorded here. + +## T05 - Cut over the deployment + +```task +id: TEN-WP-0009-T05 +status: todo +priority: high +``` + +Update `deploy/tenant-engine.yaml`: + +- drop the PVC and its volume mount; +- add `TENANT_ENGINE_DATABASE_URL` from the CNPG-minted `secretKeyRef`; +- switch `strategy: Recreate` → `RollingUpdate`, now that no `ReadWriteOnce` + volume forces serialization — **this is what unblocks TEN-WP-0008-T02's + canary**; +- extend the `NetworkPolicy` egress to the Postgres service, and no wider. + Ingress stays restricted to `user-engine`. + +Then decide, deliberately, whether to raise `replicas` above 1. It becomes +possible here for the first time; possible is not the same as warranted, and a +second replica changes failure modes for a service on the authorization path. + +Update `make verify-pin` if the expected shape changes, and re-run it. + +Rollback: re-pin the previous digest and re-attach the PVC. State plainly that +this becomes lossy once writes have landed in PostgreSQL — after that point the +rollback is "restore from Postgres", not "swap back to the file". + +Done when production runs on PostgreSQL, `make verify-pin` passes, and the +TEN-WP-0007 T04 verification list still passes end to end against the live +service. + +## T06 - Tell the fleet, and close the loop on TEN-WP-0008 + +```task +id: TEN-WP-0009-T06 +status: todo +priority: medium +``` + +- `flex-auth`: availability characteristics of their data source changed. No + contract or action change. +- `user-engine`: no contract change either, but they run the same CNPG pattern + and may have operational lessons worth having before we repeat their mistakes. +- **TEN-WP-0008-T02**: record that the `ReadWriteOnce` canary constraint is + gone, so staged-promotion onboarding no longer has to describe a canary that + cannot run. + +Done when the notes are sent and TEN-WP-0008-T02 is updated. + +## Out of scope + +- **Removing SQLite.** It stays as the dev and test backend, and as the thing + that keeps `TenantStore` an honest seam. +- **Schema redesign.** Port the existing schema; a store migration and a model + change at the same time makes any regression ambiguous. +- **Sharing a database with `user-engine`.** `tenant_id` is the only key the + two services share, and it stays that way — separate stores are the boundary, + not an implementation detail.