--- id: TEN-WP-0009 type: workplan title: "PostgreSQL as the production store, SQLite for dev and test" domain: infotech repo: tenant-engine status: ready owner: claude topic_slug: tenant-lifecycle created: "2026-08-16" updated: "2026-08-16" depends_on: - TEN-WP-0007 unblocks: - TEN-WP-0008 --- # TEN-WP-0009 - PostgreSQL production store Move the production store from SQLite-on-a-PVC to the fleet's CloudNativePG infrastructure. SQLite stays, deliberately, as the dev and test backend. ## Why this reverses an earlier decision TEN-WP-0005-T02 chose SQLite over PostgreSQL on purpose, and the reasoning was sound at the time: TEN-WP-0004 had shipped SQLite on a PVC, so *"adding an unused Postgres path would have been dead code"*. The `TenantStore` Protocol was kept precisely as the seam for this change. Three things have changed since: 1. **The PVC is now blocking work.** The volume is `ReadWriteOnce`, which forces `strategy: Recreate` and makes a side-by-side canary impossible — the constraint TEN-WP-0008-T02 ran into when onboarding to staged promotion. The store choice is no longer invisible to anything outside this repo. 2. **The infrastructure is already there and already proven.** The CloudNativePG operator runs on railiance01 with seven clusters, and `user-engine` — our sibling service, sharing the `tenant_id` key — has run its own `user-engine-pg` for 19 days. 3. **A Postgres path is no longer unused.** It becomes the production path, so the dead-code objection no longer applies. SQLite is *not* being removed. It stays as the dev and test backend, where a zero-setup file-backed store is genuinely the right tool, and it keeps the conformance suite honest by forcing the Protocol to stay a real seam rather than a formality wrapped around one implementation. ## Credential handling CloudNativePG mints the connection secret itself: a cluster named `X` produces secret `X-app` with key `uri`. The Deployment references it via `secretKeyRef`, exactly as `user-engine` does with `USER_ENGINE_DATABASE_URL` ← `user-engine-pg-app/uri`. So **no secret enters this repo, this workplan, or any evidence dump**, and no `warden route` lookup is needed — this is not a credential we fetch, custody, or rotate. Runtime secret custody remains OpenBao's per `SCOPE.md`; an operator-minted in-cluster secret is outside that lane. ## T01 - Decide the topology and write the cluster manifest ```task id: TEN-WP-0009-T01 status: todo priority: high ``` Decide between a dedicated `tenant-engine-pg` cluster in our namespace and sharing an existing one (`net-kingdom-pg` or `apps-pg` in `databases`). Argue it rather than defaulting. Points to weigh: `user-engine` chose dedicated and is the closest precedent; a dedicated cluster keeps blast radius and backup policy per-service; a shared cluster is cheaper on a single node and `net-kingdom-pg` already exists for this product family. Note that `tenant-engine` is consulted by `flex-auth` on the authorization path, so its availability envelope is not purely its own. Write the CNPG `Cluster` manifest into `deploy/`, matching the fleet's existing image and storage conventions rather than inventing new ones. Also decide **backups**. The `scheduledbackups.postgresql.cnpg.io` CRD is available; SQLite-on-a-PVC had no backup story at all, so this is a gain to claim explicitly rather than inherit by accident. Done when the topology decision is recorded with its reasoning and the manifest is committed. ## T02 - Implement `PostgresTenantStore` ```task id: TEN-WP-0009-T02 status: todo priority: high ``` Implement the full `TenantStore` Protocol — including `guardrail_overrides` / `guardrail_changes` / `set_guardrail_override` from TEN-WP-0006 — and add PostgreSQL to the conformance suites, which are already parametrised over in-memory and SQLite. A third backend should be a fixture parameter, not a new test file. Invariants that must hold identically, not approximately: - `mutate_tenant()` commits idempotency replay, version CAS, mutation, and audit event **in one transaction**; - `set_guardrail_override()` does the same for the override, its audit record, and its receipt; - a replayed `Idempotency-Key` short-circuits *before* the version check; - reads and writes never observe a row mid-transaction. That last one is where PostgreSQL should simplify rather than complicate: SQLite needed an explicit `RLock` around reads because one connection is shared across request threads, and the concurrent-writer test caught a real defect there. Postgres gives proper per-connection transaction isolation, so the lock should disappear rather than be ported. **Do not transliterate the SQLite implementation** — port the semantics and let the backend do what it is good at. Use a connection pool and appropriate isolation, and prove the guarantees with the same concurrent-writer test that caught the SQLite bug. Adds a runtime dependency (`psycopg`); keep it out of the base install path if SQLite-only dev is meant to stay dependency-light. Done when all three backends pass the same conformance suites unchanged. ## T03 - Backend selection that fails closed ```task id: TEN-WP-0009-T03 status: todo priority: high ``` `Settings` currently carries only `database_path`. Add a database URL and make selection explicit: - URL set → PostgreSQL; - path set → SQLite; - **both set → refuse to start.** Do not silently prefer one. An ambiguous store configuration in production is exactly the class of silent misconfiguration this repo fails closed on everywhere else, and picking a winner would let a stale `TENANT_ENGINE_DATABASE_PATH` quietly shadow the real database; - neither set → in-memory, as today, which is correct for tests and wrong for production. Consider surfacing the active backend on `/health`. The last two production incidents were invisible because the service looked fine from outside; "which store am I actually using" is cheap to answer and expensive to guess. Done when misconfiguration is a startup failure with a clear message, covered by tests. ## T04 - Migrate production data ```task id: TEN-WP-0009-T04 status: todo priority: high ``` Move the live SQLite database to PostgreSQL. Currently small — two tenants (`tenant:trial:portalcheck` active v1, `tenant:trial:ten-wp-0005-t05` retired v5) plus the T04 disposables from TEN-WP-0007 — but small is not the same as trivial. Everything must move, not just the tenants: - `tenants` with versions and lifecycle timestamps **preserved exactly** — a reset version silently breaks every consumer holding an ETag; - `grants` including revoked ones, because the trail is append-only and its history is the audit record; - `plans`; - `events` — the audit correlation contract; - `idempotency_receipts` — dropping these lets an in-flight retry double-apply a mutation that already happened; - `guardrail_overrides` and `guardrail_changes`. SQLite is single-writer and the app writes to it, so the cutover must stop writes rather than race them: scale to zero, export, import, verify, then start against PostgreSQL. Verify by comparing row counts *and* spot-checking the two known tenants' versions and lifecycle state, not by trusting the exporter. Keep the SQLite file and its PVC until T05's soak passes. It is the rollback. Done when a verified copy is live in PostgreSQL and the comparison evidence is recorded here. ## T05 - Cut over the deployment ```task id: TEN-WP-0009-T05 status: todo priority: high ``` Update `deploy/tenant-engine.yaml`: - drop the PVC and its volume mount; - add `TENANT_ENGINE_DATABASE_URL` from the CNPG-minted `secretKeyRef`; - switch `strategy: Recreate` → `RollingUpdate`, now that no `ReadWriteOnce` volume forces serialization — **this is what unblocks TEN-WP-0008-T02's canary**; - extend the `NetworkPolicy` egress to the Postgres service, and no wider. Ingress stays restricted to `user-engine`. Then decide, deliberately, whether to raise `replicas` above 1. It becomes possible here for the first time; possible is not the same as warranted, and a second replica changes failure modes for a service on the authorization path. Update `make verify-pin` if the expected shape changes, and re-run it. Rollback: re-pin the previous digest and re-attach the PVC. State plainly that this becomes lossy once writes have landed in PostgreSQL — after that point the rollback is "restore from Postgres", not "swap back to the file". Done when production runs on PostgreSQL, `make verify-pin` passes, and the TEN-WP-0007 T04 verification list still passes end to end against the live service. ## T06 - Tell the fleet, and close the loop on TEN-WP-0008 ```task id: TEN-WP-0009-T06 status: todo priority: medium ``` - `flex-auth`: availability characteristics of their data source changed. No contract or action change. - `user-engine`: no contract change either, but they run the same CNPG pattern and may have operational lessons worth having before we repeat their mistakes. - **TEN-WP-0008-T02**: record that the `ReadWriteOnce` canary constraint is gone, so staged-promotion onboarding no longer has to describe a canary that cannot run. Done when the notes are sent and TEN-WP-0008-T02 is updated. ## Out of scope - **Removing SQLite.** It stays as the dev and test backend, and as the thing that keeps `TenantStore` an honest seam. - **Schema redesign.** Port the existing schema; a store migration and a model change at the same time makes any regression ambiguous. - **Sharing a database with `user-engine`.** `tenant_id` is the only key the two services share, and it stays that way — separate stores are the boundary, not an implementation detail.