tenant-engine/workplans/TEN-WP-0008-staged-promotion-onboarding.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	depends_on	unblocks	state_hub_workstream_id
TEN-WP-0008	workplan	Bring tenant-engine under the staged-promotion contract	infotech	tenant-engine	finished	grok	tenant-guardrails	2026-08-16	2026-08-29	TEN-WP-0007		e9da1c15-7beb-48d9-97ab-b657fdf9366e

TEN-WP-0008 - Staged-promotion onboarding

tenant-engine runs in production but was never brought under the RAIL-BS-WP-0006 staged-promotion contract: no railiance/app.toml, no stage commands, no canary or approval evidence. Rollouts are a hand-edited digest in deploy/tenant-engine.yaml plus a kubectl apply.

Why now

Two silent pin rollbacks inside one week, discovered by accident rather than by any signal:

• flex-auth rolled back to a four-action policy — lifecycle mutations denied with 403 write_denied.
• tenant-engine rolled back to the pre-lifecycle TEN-WP-0004 image — lifecycle routes absent, 404.

Neither produced an alert. Both looked like ordinary responses rather than failures, so nothing watching for crashes or non-2xx would have noticed. user-engine was the live consumer of both and its contract-level tests passed throughout, because they were right about the contract while production was not serving it.

TEN-WP-0007 added make verify-pin, which compares the repo's digest, the Deployment spec, the running pod, and the live routes. That is worth having but it is not the answer: it runs when someone remembers to run it, from a workstation, against one deployment. The failure being guarded against is precisely the one nobody was watching for. A local check makes us individually lucky; staged promotion makes "the digest that should be live" a recorded fact that drift can be measured against.

Boundary note

This workplan onboards this repo to a contract owned by railiance-bootstrap. It does not modify the contract, the CLI, or the schema, and it does not build a drift-detection system — if staged promotion already records the intended digest, drift detection is a query against something that exists. If the contract needs changing to fit a plain-manifest workload, that is a railiance-bootstrap decision to request, not one to make here.

T01 - Write and validate railiance/app.toml

id: TEN-WP-0008-T01
status: done
priority: high
state_hub_task_id: "0231e4d1-2e1a-4e77-92ba-2cfaf23692ab"

Author railiance/app.toml against railiance.app.v1 (railiance-bootstrap/schemas/railiance-app.schema.json). Required sections: app, source, platform, secrets, observability, rollback, stages (1–3), checks.

Model it on qonto-assistant/railiance/app.toml — the same fleet convention this repo follows for small headless services — not on a larger workload whose shape does not apply.

Decisions this task must make and record:

• criticality. tenant-engine is a data source flex-auth consults for authorization decisions and key-cape consults at token issuance. It holds no secrets and is not internet-facing, but an outage makes a PDP fail closed fleet-wide. Argue the level rather than defaulting it.
• secrets.references. This repo has none — runtime secret custody is OpenBao's per SCOPE.md. An empty list is the correct answer, and stating it explicitly is the point.
• Health endpoints for stage1 (local) and stage2 (cluster), using /health on port 8090.
• Rollback strategy, which must record the corrected ladder from TEN-WP-0007: 44ca65f3… (lifecycle + guardrails) → 08be0b1d… (lifecycle only) → 2249e8c6… (neither). Rolling back past 08be0b1d… removes lifecycle authority user-engine depends on.

Validate with bin/railiance run <repo> --pretty in plan mode. Save the JSON result as non-secret evidence.

Done when the file validates against the schema and stage 1 passes in plan mode.

Done 2026-08-29: railiance/app.toml validates; bin/railiance run . passed (docs/evidence/TEN-WP-0008-stage1-result.json). criticality=high (argument in [app].description); secrets.references = []; /health on 8090; rollback strategy image-digest.

T02 - Reconcile the deployment shape with the contract

id: TEN-WP-0008-T02
status: done
priority: high
state_hub_task_id: "5f2ad011-1de8-45df-8e8f-dd6bcf51ece4"

Expected friction, and the real work of this workplan. The stage 3 promote path expects a Helm chart or a kustomization; tenant-engine ships a single five-document manifest (Namespace, PVC, Deployment, Service, NetworkPolicy) applied with plain kubectl apply -f. qonto-assistant uses kustomize, so it is not a precedent for this shape.

Establish which of these is true and act on it:

1. the contract already supports a plain-manifest workload and the app.toml just needs the right stage commands;
2. converting to kustomize is cheap and correct;
3. the contract genuinely does not cover this shape, in which case ask railiance-bootstrap rather than working around it.

Do not paper over a gap in the contract with a bespoke local step — that reproduces the problem this workplan exists to fix.

A further constraint: the Deployment uses Recreate because the SQLite PVC is ReadWriteOnce. A canary that runs alongside the stable pod cannot mount the same volume, so canary_mode must reflect that a true side-by-side canary is impossible here. Say so explicitly rather than declaring a canary that cannot run.

Done when the stage commands in app.toml are ones that actually work against this repo's deployment shape, verified in plan mode.

Constraint update 2026-08-21: TEN-WP-0009 removed the ReadWriteOnce volume from the production Deployment, moved the workload to PostgreSQL and changed the strategy to RollingUpdate. A side-by-side canary is now storage-feasible; T02 must no longer encode the old "canary impossible" exception. The staged contract still owns the exact canary mode and promotion commands. Current production digest: sha256:a8e8086ffc5b772c1391b166f5e1884b90f7d327b152c205eceae129df555c24.

Done 2026-08-29: option 2 — kustomize wrap (deploy/ production, deploy/canary/ isolated). Stage commands are kubectl apply -k, verified by Stage 1 render. bin/railiance deploy|promote --plan still requires Helm; not papered over. Requested as RAIL-BS-IN-0001. Evidence: docs/evidence/TEN-WP-0008-stage2-plan.json, docs/evidence/TEN-WP-0008-stage3-plan.json.

T03 - Record current production as the stable baseline

id: TEN-WP-0008-T03
status: done
priority: medium
state_hub_task_id: "c4b79dc1-c280-428e-9dc5-b17c05b084c6"

Register the live digest @sha256:44ca65f3… as the recorded stable release, so there is an authoritative answer to "what should be running" that drift can be measured against. Until this exists, make verify-pin compares the cluster to a repo file that a human edits by hand — better than nothing, but still two copies of the truth rather than one.

Done when the stable digest is recorded through the contract's own mechanism rather than only in deploy/tenant-engine.yaml.

Done 2026-08-29: [stages.stage3].previous_stable = tenant-engine@sha256:a8e8086ffc5b772c1391b166f5e1884b90f7d327b152c205eceae129df555c24 (current PostgreSQL runtime, same pin make verify-pin reads). Stage 3 plan-mode precheck previous-stable passed. The T03 draft's 44ca65f3… pin was stale after TEN-WP-0009.

T04 - Hand the pattern back to the fleet

id: TEN-WP-0008-T04
status: done
priority: low
state_hub_task_id: "b816a862-51e0-4abd-a2f3-ee40359526cc"

Tell flex-auth what onboarding actually cost, including anything the contract did not cover. They filed FLEX-WP-0011 for the same gap and asked whether there is a fleet-level answer to drift; a second data point on the same contract is more useful to them than our local script was.

If T02 surfaced a genuine contract gap, make sure railiance-bootstrap has it as a request rather than leaving two overlay repos to work around it independently.

Done when the note is sent.

Done 2026-08-29: note to flex-auth (cost of onboarding a non-Helm workload) and RAIL-BS-IN-0001 plus a State Hub message to railiance-bootstrap.

Out of scope

• Changing the staged-promotion contract, CLI, or schema. Request, do not implement.
• Building a drift-detection service. The point of this workplan is that staged promotion should make one unnecessary.
• Re-deploying. Production is healthy on a8e8086ffc… (TEN-WP-0009); onboarding describes what is already running, it does not ship TEN-WP-0011 or any newer image.