ops-warden WARDEN-WP-0031 T04 minted audience-flex-auth tokens as system:serviceaccount:ops-warden:ops-warden and held the warn-log count. Helm rev 2 is enforce. Live probe 200/401/403. policy.enabled is theirs.
6 KiB
flex-auth production deployment
The sanctioned promotion path is the Railiance overlay in
../railiance/ (railiance/app.toml,
charts/flex-auth, values/*.yaml). These flattened files are the
emergency kubectl path recovered on 2026-08-11 from live
last-applied-configuration, kept so a rollback does not depend on a
cluster annotation or on Helm history.
Caller authentication (FLEX-WP-0015-T02) is pinned here in warn,
matching values/user-engine.yaml and values/tenant-engine.yaml. Do not
edit these files back to enforce and apply them as a shortcut: that is
the hazard. Flip callerAuth.mode per consumer in the overlay after that
consumer's warn logs are clean, then re-render or edit only that pin.
Manifests for the two cluster-local flex-auth policy-decision services.
| File | Deployment | Consumer | Service DNS |
|---|---|---|---|
flex-auth-tenant-engine.yaml |
flex-auth-tenant-engine |
tenant-engine write API | flex-auth-tenant-engine.flex-auth.svc.cluster.local:8080 |
flex-auth-user-engine.yaml |
flex-auth-user-engine |
user-engine portal | flex-auth-user-engine.flex-auth.svc.cluster.local:8080 |
Each file is a three-document manifest: Deployment, Service, and a
default-deny NetworkPolicy whose ingress is restricted to the one approved
consumer workload and which permits no egress.
A harmless diff on apply
kubectl apply reports the two NetworkPolicies as configured rather than
unchanged, every time. That is not drift: the manifests carry an explicit
egress: [], which the API server normalises away on read. With
policyTypes: [Ingress, Egress] and no egress rules, deny-all egress holds
either way. The empty list is kept because it states the intent to a reader
instead of leaving it implicit. Deployments and Services do round-trip as
unchanged.
One image, two deployments
Both Deployments run the same image repository and differ only in their
--registry / --policy arguments. The Containerfile does
COPY examples /opt/flex-auth/examples, so every image contains every
consumer's policy package; the arguments select which one that instance
serves.
Consequence worth remembering: rebuilding to pick up one consumer's policy change also re-bakes every other consumer's policy into the new image. The two Deployments are pinned to different digests precisely so that one can be rolled without moving the other. Roll only the Deployment whose policy actually changed.
Rolling out a policy change
The policy packages are baked into the image, not mounted from a ConfigMap, so a policy change requires a rebuild — there is no hot reload.
Do not build images on a workstation. The fleet builds in CI so that an
artifact's provenance is a forge revision rather than someone's working tree.
.forgejo/workflows/image.yaml handles it, and examples/** is one of its
trigger paths precisely because policy changes are image changes.
# 1. Push the commit you intend to ship; CI builds it on the container-build
# runner and pushes :latest and :main-<short-sha>
git push origin main
# 2. Take the immutable digest from the workflow's "Report immutable digest"
# step -- deploy by digest, never by tag
# 3. Edit the image digest in the relevant manifest, then apply
kubectl apply -f deploy/flex-auth-<consumer>.yaml
kubectl -n flex-auth rollout status deploy/flex-auth-<consumer> --timeout=120s
# 4. Verify the new policy actually took effect, from outside the cluster
kubectl -n flex-auth port-forward svc/flex-auth-<consumer> 19099:8080 &
curl -s -X POST http://127.0.0.1:19099/v1/check \
-H 'Content-Type: application/json' \
-d @examples/<consumer>/<a request that exercises the change>.json
Step 4 is not optional. Because the policy ships inside the image, a
successful rollout status only proves the container started — it says
nothing about which policy revision is being served.
Rollback
kubectl -n flex-auth rollout undo deploy/flex-auth-<consumer>
If the ReplicaSet history has been pruned, re-apply the manifest with the last-known-good digest below.
| Deployment | Last-known-good digest | Policy state |
|---|---|---|
flex-auth-tenant-engine |
sha256:138aa3471c46bca6e814691fa1e6520aedda3dffd743e6b09141ab433afdb64b |
live — caller-auth enforce (FLEX-WP-0015-T02), CI main-3de72fe, A2 probe 2026-08-19 |
flex-auth-tenant-engine (previous) |
sha256:1bf060e61122693ce98359c167cc5fe8bdafc84e097e090eaa71af94d0f27cbc |
nine-action policy (FLEX-WP-0014), CI-built from f304688 |
flex-auth-tenant-engine (previous) |
sha256:9320df394a642eff24da8af4a0ee8886a7bb78b0f14d8ee1deeb30ea8eeeaba7 |
seven-action policy, FLEX-WP-0013 restore; guardrail actions deny unknown_action |
flex-auth-tenant-engine (rollback) |
sha256:c25fc34a6cd7e64d955f8723ec70e176a583d5ae71d76280c4e2d89fba0fe0aa |
four-action policy; lifecycle actions deny unknown_action |
flex-auth-ops-warden |
sha256:138aa3471c46bca6e814691fa1e6520aedda3dffd743e6b09141ab433afdb64b |
live isolated enforce — ops-warden production registry (FLEX-WP-0016), CI main-3de72fe, A2 probe 2026-08-19 |
flex-auth-user-engine |
sha256:138aa3471c46bca6e814691fa1e6520aedda3dffd743e6b09141ab433afdb64b |
live — caller-auth enforce (FLEX-WP-0015-T02), CI main-3de72fe, A2 probe 2026-08-19 |
flex-auth-user-engine (previous) |
sha256:1f5290376dc5fcf456dc7a785e394d8b90949dabecd1d3e856f38557149bb5f4 |
FLEX-WP-0009-T04, nine fixtures (incl. registration-applicant) verified live 2026-08-16 |
flex-auth-user-engine (previous) |
sha256:a31961c45215aa6baf3bc748c6741ab703c2c8325e61aa7983a355026195e51b |
FLEX-WP-0009-T03, six fixtures verified live 2026-08-10 |
Rolling back the tenant-engine service to c25fc34a… restores fail-closed
behaviour for the lifecycle actions — tenant-engine's lifecycle endpoints
return 403 write_denied rather than writing. That is a safe failure mode,
not an outage of the older four actions, which keep working.