Plan PostgreSQL production store (TEN-WP-0009)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
bd61df3ff8
commit
781359e7d4
3 changed files with 256 additions and 0 deletions
|
|
@ -16,6 +16,7 @@
|
|||
| workplan | TEN-WP-0005 | finished | — | workplans/TEN-WP-0005-tenant-update-and-retirement-api.md |
|
||||
| workplan | TEN-WP-0006 | finished | — | workplans/TEN-WP-0006-guardrail-quota-policy.md |
|
||||
| workplan | TEN-WP-0007 | finished | — | workplans/TEN-WP-0007-guardrail-production-rollout.md |
|
||||
| workplan | TEN-WP-0008 | ready | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md |
|
||||
| task | ADHOC-2026-07-24-T01 | done | — | workplans/ADHOC-2026-07-24.md |
|
||||
| task | TEN-WP-0001-T01 | done | — | workplans/TEN-WP-0001-statehub-bootstrap.md |
|
||||
| task | TEN-WP-0001-T02 | done | — | workplans/TEN-WP-0001-statehub-bootstrap.md |
|
||||
|
|
@ -48,3 +49,7 @@
|
|||
| task | TEN-WP-0007-T02 | done | — | workplans/TEN-WP-0007-guardrail-production-rollout.md |
|
||||
| task | TEN-WP-0007-T03 | done | — | workplans/TEN-WP-0007-guardrail-production-rollout.md |
|
||||
| task | TEN-WP-0007-T04 | done | — | workplans/TEN-WP-0007-guardrail-production-rollout.md |
|
||||
| task | TEN-WP-0008-T01 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md |
|
||||
| task | TEN-WP-0008-T02 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md |
|
||||
| task | TEN-WP-0008-T03 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md |
|
||||
| task | TEN-WP-0008-T04 | todo | — | workplans/TEN-WP-0008-staged-promotion-onboarding.md |
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ updated: "2026-08-16"
|
|||
depends_on:
|
||||
- TEN-WP-0007
|
||||
unblocks: []
|
||||
state_hub_workstream_id: "e9da1c15-7beb-48d9-97ab-b657fdf9366e"
|
||||
---
|
||||
|
||||
# TEN-WP-0008 - Staged-promotion onboarding
|
||||
|
|
@ -60,6 +61,7 @@ is a `railiance-bootstrap` decision to request, not one to make here.
|
|||
id: TEN-WP-0008-T01
|
||||
status: todo
|
||||
priority: high
|
||||
state_hub_task_id: "0231e4d1-2e1a-4e77-92ba-2cfaf23692ab"
|
||||
```
|
||||
|
||||
Author `railiance/app.toml` against `railiance.app.v1`
|
||||
|
|
@ -99,6 +101,7 @@ mode.
|
|||
id: TEN-WP-0008-T02
|
||||
status: todo
|
||||
priority: high
|
||||
state_hub_task_id: "5f2ad011-1de8-45df-8e8f-dd6bcf51ece4"
|
||||
```
|
||||
|
||||
Expected friction, and the real work of this workplan. The stage 3 promote path
|
||||
|
|
@ -133,6 +136,7 @@ this repo's deployment shape, verified in plan mode.
|
|||
id: TEN-WP-0008-T03
|
||||
status: todo
|
||||
priority: medium
|
||||
state_hub_task_id: "c4b79dc1-c280-428e-9dc5-b17c05b084c6"
|
||||
```
|
||||
|
||||
Register the live digest `@sha256:44ca65f3…` as the recorded stable release, so
|
||||
|
|
@ -150,6 +154,7 @@ rather than only in `deploy/tenant-engine.yaml`.
|
|||
id: TEN-WP-0008-T04
|
||||
status: todo
|
||||
priority: low
|
||||
state_hub_task_id: "b816a862-51e0-4abd-a2f3-ee40359526cc"
|
||||
```
|
||||
|
||||
Tell `flex-auth` what onboarding actually cost, including anything the contract
|
||||
|
|
|
|||
246
workplans/TEN-WP-0009-postgres-production-store.md
Normal file
246
workplans/TEN-WP-0009-postgres-production-store.md
Normal file
|
|
@ -0,0 +1,246 @@
|
|||
---
|
||||
id: TEN-WP-0009
|
||||
type: workplan
|
||||
title: "PostgreSQL as the production store, SQLite for dev and test"
|
||||
domain: infotech
|
||||
repo: tenant-engine
|
||||
status: ready
|
||||
owner: claude
|
||||
topic_slug: tenant-lifecycle
|
||||
created: "2026-08-16"
|
||||
updated: "2026-08-16"
|
||||
depends_on:
|
||||
- TEN-WP-0007
|
||||
unblocks:
|
||||
- TEN-WP-0008
|
||||
---
|
||||
|
||||
# TEN-WP-0009 - PostgreSQL production store
|
||||
|
||||
Move the production store from SQLite-on-a-PVC to the fleet's CloudNativePG
|
||||
infrastructure. SQLite stays, deliberately, as the dev and test backend.
|
||||
|
||||
## Why this reverses an earlier decision
|
||||
|
||||
TEN-WP-0005-T02 chose SQLite over PostgreSQL on purpose, and the reasoning was
|
||||
sound at the time: TEN-WP-0004 had shipped SQLite on a PVC, so *"adding an
|
||||
unused Postgres path would have been dead code"*. The `TenantStore` Protocol
|
||||
was kept precisely as the seam for this change.
|
||||
|
||||
Three things have changed since:
|
||||
|
||||
1. **The PVC is now blocking work.** The volume is `ReadWriteOnce`, which forces
|
||||
`strategy: Recreate` and makes a side-by-side canary impossible — the
|
||||
constraint TEN-WP-0008-T02 ran into when onboarding to staged promotion. The
|
||||
store choice is no longer invisible to anything outside this repo.
|
||||
2. **The infrastructure is already there and already proven.** The
|
||||
CloudNativePG operator runs on railiance01 with seven clusters, and
|
||||
`user-engine` — our sibling service, sharing the `tenant_id` key — has run
|
||||
its own `user-engine-pg` for 19 days.
|
||||
3. **A Postgres path is no longer unused.** It becomes the production path, so
|
||||
the dead-code objection no longer applies.
|
||||
|
||||
SQLite is *not* being removed. It stays as the dev and test backend, where a
|
||||
zero-setup file-backed store is genuinely the right tool, and it keeps the
|
||||
conformance suite honest by forcing the Protocol to stay a real seam rather
|
||||
than a formality wrapped around one implementation.
|
||||
|
||||
## Credential handling
|
||||
|
||||
CloudNativePG mints the connection secret itself: a cluster named `X` produces
|
||||
secret `X-app` with key `uri`. The Deployment references it via
|
||||
`secretKeyRef`, exactly as `user-engine` does with
|
||||
`USER_ENGINE_DATABASE_URL` ← `user-engine-pg-app/uri`.
|
||||
|
||||
So **no secret enters this repo, this workplan, or any evidence dump**, and no
|
||||
`warden route` lookup is needed — this is not a credential we fetch, custody,
|
||||
or rotate. Runtime secret custody remains OpenBao's per `SCOPE.md`; an
|
||||
operator-minted in-cluster secret is outside that lane.
|
||||
|
||||
## T01 - Decide the topology and write the cluster manifest
|
||||
|
||||
```task
|
||||
id: TEN-WP-0009-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Decide between a dedicated `tenant-engine-pg` cluster in our namespace and
|
||||
sharing an existing one (`net-kingdom-pg` or `apps-pg` in `databases`).
|
||||
|
||||
Argue it rather than defaulting. Points to weigh: `user-engine` chose dedicated
|
||||
and is the closest precedent; a dedicated cluster keeps blast radius and backup
|
||||
policy per-service; a shared cluster is cheaper on a single node and
|
||||
`net-kingdom-pg` already exists for this product family. Note that
|
||||
`tenant-engine` is consulted by `flex-auth` on the authorization path, so its
|
||||
availability envelope is not purely its own.
|
||||
|
||||
Write the CNPG `Cluster` manifest into `deploy/`, matching the fleet's existing
|
||||
image and storage conventions rather than inventing new ones.
|
||||
|
||||
Also decide **backups**. The `scheduledbackups.postgresql.cnpg.io` CRD is
|
||||
available; SQLite-on-a-PVC had no backup story at all, so this is a gain to
|
||||
claim explicitly rather than inherit by accident.
|
||||
|
||||
Done when the topology decision is recorded with its reasoning and the manifest
|
||||
is committed.
|
||||
|
||||
## T02 - Implement `PostgresTenantStore`
|
||||
|
||||
```task
|
||||
id: TEN-WP-0009-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Implement the full `TenantStore` Protocol — including
|
||||
`guardrail_overrides` / `guardrail_changes` / `set_guardrail_override` from
|
||||
TEN-WP-0006 — and add PostgreSQL to the conformance suites, which are already
|
||||
parametrised over in-memory and SQLite. A third backend should be a fixture
|
||||
parameter, not a new test file.
|
||||
|
||||
Invariants that must hold identically, not approximately:
|
||||
|
||||
- `mutate_tenant()` commits idempotency replay, version CAS, mutation, and
|
||||
audit event **in one transaction**;
|
||||
- `set_guardrail_override()` does the same for the override, its audit record,
|
||||
and its receipt;
|
||||
- a replayed `Idempotency-Key` short-circuits *before* the version check;
|
||||
- reads and writes never observe a row mid-transaction.
|
||||
|
||||
That last one is where PostgreSQL should simplify rather than complicate:
|
||||
SQLite needed an explicit `RLock` around reads because one connection is shared
|
||||
across request threads, and the concurrent-writer test caught a real defect
|
||||
there. Postgres gives proper per-connection transaction isolation, so the lock
|
||||
should disappear rather than be ported. **Do not transliterate the SQLite
|
||||
implementation** — port the semantics and let the backend do what it is good
|
||||
at. Use a connection pool and appropriate isolation, and prove the guarantees
|
||||
with the same concurrent-writer test that caught the SQLite bug.
|
||||
|
||||
Adds a runtime dependency (`psycopg`); keep it out of the base install path if
|
||||
SQLite-only dev is meant to stay dependency-light.
|
||||
|
||||
Done when all three backends pass the same conformance suites unchanged.
|
||||
|
||||
## T03 - Backend selection that fails closed
|
||||
|
||||
```task
|
||||
id: TEN-WP-0009-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
`Settings` currently carries only `database_path`. Add a database URL and make
|
||||
selection explicit:
|
||||
|
||||
- URL set → PostgreSQL;
|
||||
- path set → SQLite;
|
||||
- **both set → refuse to start.** Do not silently prefer one. An ambiguous
|
||||
store configuration in production is exactly the class of silent
|
||||
misconfiguration this repo fails closed on everywhere else, and picking a
|
||||
winner would let a stale `TENANT_ENGINE_DATABASE_PATH` quietly shadow the
|
||||
real database;
|
||||
- neither set → in-memory, as today, which is correct for tests and wrong for
|
||||
production.
|
||||
|
||||
Consider surfacing the active backend on `/health`. The last two production
|
||||
incidents were invisible because the service looked fine from outside; "which
|
||||
store am I actually using" is cheap to answer and expensive to guess.
|
||||
|
||||
Done when misconfiguration is a startup failure with a clear message, covered
|
||||
by tests.
|
||||
|
||||
## T04 - Migrate production data
|
||||
|
||||
```task
|
||||
id: TEN-WP-0009-T04
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Move the live SQLite database to PostgreSQL. Currently small — two tenants
|
||||
(`tenant:trial:portalcheck` active v1, `tenant:trial:ten-wp-0005-t05` retired
|
||||
v5) plus the T04 disposables from TEN-WP-0007 — but small is not the same as
|
||||
trivial.
|
||||
|
||||
Everything must move, not just the tenants:
|
||||
|
||||
- `tenants` with versions and lifecycle timestamps **preserved exactly** — a
|
||||
reset version silently breaks every consumer holding an ETag;
|
||||
- `grants` including revoked ones, because the trail is append-only and its
|
||||
history is the audit record;
|
||||
- `plans`;
|
||||
- `events` — the audit correlation contract;
|
||||
- `idempotency_receipts` — dropping these lets an in-flight retry double-apply
|
||||
a mutation that already happened;
|
||||
- `guardrail_overrides` and `guardrail_changes`.
|
||||
|
||||
SQLite is single-writer and the app writes to it, so the cutover must stop
|
||||
writes rather than race them: scale to zero, export, import, verify, then start
|
||||
against PostgreSQL. Verify by comparing row counts *and* spot-checking the two
|
||||
known tenants' versions and lifecycle state, not by trusting the exporter.
|
||||
|
||||
Keep the SQLite file and its PVC until T05's soak passes. It is the rollback.
|
||||
|
||||
Done when a verified copy is live in PostgreSQL and the comparison evidence is
|
||||
recorded here.
|
||||
|
||||
## T05 - Cut over the deployment
|
||||
|
||||
```task
|
||||
id: TEN-WP-0009-T05
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Update `deploy/tenant-engine.yaml`:
|
||||
|
||||
- drop the PVC and its volume mount;
|
||||
- add `TENANT_ENGINE_DATABASE_URL` from the CNPG-minted `secretKeyRef`;
|
||||
- switch `strategy: Recreate` → `RollingUpdate`, now that no `ReadWriteOnce`
|
||||
volume forces serialization — **this is what unblocks TEN-WP-0008-T02's
|
||||
canary**;
|
||||
- extend the `NetworkPolicy` egress to the Postgres service, and no wider.
|
||||
Ingress stays restricted to `user-engine`.
|
||||
|
||||
Then decide, deliberately, whether to raise `replicas` above 1. It becomes
|
||||
possible here for the first time; possible is not the same as warranted, and a
|
||||
second replica changes failure modes for a service on the authorization path.
|
||||
|
||||
Update `make verify-pin` if the expected shape changes, and re-run it.
|
||||
|
||||
Rollback: re-pin the previous digest and re-attach the PVC. State plainly that
|
||||
this becomes lossy once writes have landed in PostgreSQL — after that point the
|
||||
rollback is "restore from Postgres", not "swap back to the file".
|
||||
|
||||
Done when production runs on PostgreSQL, `make verify-pin` passes, and the
|
||||
TEN-WP-0007 T04 verification list still passes end to end against the live
|
||||
service.
|
||||
|
||||
## T06 - Tell the fleet, and close the loop on TEN-WP-0008
|
||||
|
||||
```task
|
||||
id: TEN-WP-0009-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
- `flex-auth`: availability characteristics of their data source changed. No
|
||||
contract or action change.
|
||||
- `user-engine`: no contract change either, but they run the same CNPG pattern
|
||||
and may have operational lessons worth having before we repeat their mistakes.
|
||||
- **TEN-WP-0008-T02**: record that the `ReadWriteOnce` canary constraint is
|
||||
gone, so staged-promotion onboarding no longer has to describe a canary that
|
||||
cannot run.
|
||||
|
||||
Done when the notes are sent and TEN-WP-0008-T02 is updated.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- **Removing SQLite.** It stays as the dev and test backend, and as the thing
|
||||
that keeps `TenantStore` an honest seam.
|
||||
- **Schema redesign.** Port the existing schema; a store migration and a model
|
||||
change at the same time makes any regression ambiguous.
|
||||
- **Sharing a database with `user-engine`.** `tenant_id` is the only key the
|
||||
two services share, and it stays that way — separate stores are the boundary,
|
||||
not an implementation detail.
|
||||
Loading…
Add table
Add a link
Reference in a new issue