From 2456287e8c3a72063ac24ea9e3ea37b9bb28505e Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 11 Aug 2026 02:21:06 +0200 Subject: [PATCH 1/2] Commit production deployment manifests and rollout runbook The two flex-auth Deployments were applied from a YAML file that lived outside this repo, so the only surviving copy of their authored spec was the last-applied-configuration annotation on the live objects. That made rollback depend on a cluster annotation -- recovered them into deploy/ before the FLEX-WP-0010 rollout rather than after an incident. Covers Deployment, Service, and the default-deny NetworkPolicy for both flex-auth-tenant-engine and flex-auth-user-engine. Verified with kubectl apply --dry-run=server: Deployments and Services round-trip as unchanged. The NetworkPolicies report "configured" because the manifests keep an explicit egress: [] that the API server normalises away -- same deny-all semantics, documented so nobody chases it. The runbook records that policy packages are baked into the image with no hot reload, that both Deployments share one image repo at deliberately different digests (so roll only the one whose policy changed), and pins both last-known-good digests as rollback targets. Co-Authored-By: Claude Opus 5 --- deploy/README.md | 88 ++++++++++++++++++++++++++ deploy/flex-auth-tenant-engine.yaml | 95 +++++++++++++++++++++++++++++ deploy/flex-auth-user-engine.yaml | 95 +++++++++++++++++++++++++++++ 3 files changed, 278 insertions(+) create mode 100644 deploy/README.md create mode 100644 deploy/flex-auth-tenant-engine.yaml create mode 100644 deploy/flex-auth-user-engine.yaml diff --git a/deploy/README.md b/deploy/README.md new file mode 100644 index 0000000..ab04e27 --- /dev/null +++ b/deploy/README.md @@ -0,0 +1,88 @@ +# flex-auth production deployment + +Manifests for the two cluster-local flex-auth policy-decision services. +Both were previously applied with `kubectl apply` from a file that lived +outside this repo; these were recovered from the live objects' +`kubectl.kubernetes.io/last-applied-configuration` on 2026-08-11 and +committed so that a rollback does not depend on a cluster annotation. + +| File | Deployment | Consumer | Service DNS | +| --- | --- | --- | --- | +| `flex-auth-tenant-engine.yaml` | `flex-auth-tenant-engine` | tenant-engine write API | `flex-auth-tenant-engine.flex-auth.svc.cluster.local:8080` | +| `flex-auth-user-engine.yaml` | `flex-auth-user-engine` | user-engine portal | `flex-auth-user-engine.flex-auth.svc.cluster.local:8080` | + +Each file is a three-document manifest: `Deployment`, `Service`, and a +default-deny `NetworkPolicy` whose ingress is restricted to the one approved +consumer workload and which permits no egress. + +### A harmless diff on apply + +`kubectl apply` reports the two NetworkPolicies as `configured` rather than +`unchanged`, every time. That is not drift: the manifests carry an explicit +`egress: []`, which the API server normalises away on read. With +`policyTypes: [Ingress, Egress]` and no egress rules, deny-all egress holds +either way. The empty list is kept because it states the intent to a reader +instead of leaving it implicit. Deployments and Services do round-trip as +`unchanged`. + +## One image, two deployments + +Both Deployments run the **same image repository** and differ only in their +`--registry` / `--policy` arguments. The `Containerfile` does +`COPY examples /opt/flex-auth/examples`, so every image contains *every* +consumer's policy package; the arguments select which one that instance +serves. + +Consequence worth remembering: rebuilding to pick up one consumer's policy +change also re-bakes every other consumer's policy into the new image. The +two Deployments are pinned to **different digests** precisely so that one can +be rolled without moving the other. Roll only the Deployment whose policy +actually changed. + +## Rolling out a policy change + +The policy packages are baked into the image, not mounted from a ConfigMap, +so a policy change requires a rebuild — there is no hot reload. + +```bash +# 1. Build and push from a clean checkout of the commit you intend to ship +docker build -t forgejo.coulomb.social/coulomb/flex-auth: -f Containerfile . +docker push forgejo.coulomb.social/coulomb/flex-auth: + +# 2. Resolve the digest and pin it — deploy by digest, never by tag +docker inspect --format='{{index .RepoDigests 0}}' \ + forgejo.coulomb.social/coulomb/flex-auth: + +# 3. Edit the image digest in the relevant manifest, then apply +kubectl apply -f deploy/flex-auth-.yaml +kubectl -n flex-auth rollout status deploy/flex-auth- --timeout=120s + +# 4. Verify the new policy actually took effect, from outside the cluster +kubectl -n flex-auth port-forward svc/flex-auth- 19099:8080 & +curl -s -X POST http://127.0.0.1:19099/v1/check \ + -H 'Content-Type: application/json' \ + -d @examples//.json +``` + +Step 4 is not optional. Because the policy ships inside the image, a +successful `rollout status` only proves the container started — it says +nothing about which policy revision is being served. + +## Rollback + +```bash +kubectl -n flex-auth rollout undo deploy/flex-auth- +``` + +If the ReplicaSet history has been pruned, re-apply the manifest with the +last-known-good digest below. + +| Deployment | Last-known-good digest | Policy state | +| --- | --- | --- | +| `flex-auth-tenant-engine` | `sha256:c25fc34a6cd7e64d955f8723ec70e176a583d5ae71d76280c4e2d89fba0fe0aa` | four-action policy, pre-FLEX-WP-0010 (lifecycle actions deny `unknown_action`) | +| `flex-auth-user-engine` | `sha256:a31961c45215aa6baf3bc748c6741ab703c2c8325e61aa7983a355026195e51b` | FLEX-WP-0009-T03, six fixtures verified live 2026-08-10 | + +Rolling back the tenant-engine service to `c25fc34a…` restores fail-closed +behaviour for the lifecycle actions — tenant-engine's lifecycle endpoints +return `403 write_denied` rather than writing. That is a safe failure mode, +not an outage of the older four actions, which keep working. diff --git a/deploy/flex-auth-tenant-engine.yaml b/deploy/flex-auth-tenant-engine.yaml new file mode 100644 index 0000000..e0a71ad --- /dev/null +++ b/deploy/flex-auth-tenant-engine.yaml @@ -0,0 +1,95 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: flex-auth-tenant-engine + namespace: flex-auth +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: flex-auth-tenant-engine + template: + metadata: + labels: + app.kubernetes.io/name: flex-auth-tenant-engine + spec: + automountServiceAccountToken: false + containers: + - args: + - serve + - --addr + - 0.0.0.0:8080 + - --registry + - /opt/flex-auth/examples/tenant-engine/registry_snapshot.json + - --policy + - /opt/flex-auth/examples/tenant-engine/policy_package.md + image: forgejo.coulomb.social/coulomb/flex-auth@sha256:c25fc34a6cd7e64d955f8723ec70e176a583d5ae71d76280c4e2d89fba0fe0aa + livenessProbe: + httpGet: + path: /healthz + port: http + periodSeconds: 20 + name: flex-auth + ports: + - containerPort: 8080 + name: http + readinessProbe: + httpGet: + path: /healthz + port: http + periodSeconds: 5 + resources: + limits: + cpu: 300m + memory: 192Mi + requests: + cpu: 25m + memory: 32Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault +--- +apiVersion: v1 +kind: Service +metadata: + name: flex-auth-tenant-engine + namespace: flex-auth +spec: + ports: + - name: http + port: 8080 + targetPort: http + selector: + app.kubernetes.io/name: flex-auth-tenant-engine +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: flex-auth-tenant-engine + namespace: flex-auth +spec: + egress: [] + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: tenant-engine + podSelector: + matchLabels: + app.kubernetes.io/name: tenant-engine + ports: + - port: 8080 + protocol: TCP + podSelector: + matchLabels: + app.kubernetes.io/name: flex-auth-tenant-engine + policyTypes: + - Ingress + - Egress diff --git a/deploy/flex-auth-user-engine.yaml b/deploy/flex-auth-user-engine.yaml new file mode 100644 index 0000000..e973c37 --- /dev/null +++ b/deploy/flex-auth-user-engine.yaml @@ -0,0 +1,95 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: flex-auth-user-engine + namespace: flex-auth +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: flex-auth-user-engine + template: + metadata: + labels: + app.kubernetes.io/name: flex-auth-user-engine + spec: + automountServiceAccountToken: false + containers: + - args: + - serve + - --addr + - 0.0.0.0:8080 + - --registry + - /opt/flex-auth/examples/user-engine/registry_snapshot.json + - --policy + - /opt/flex-auth/examples/user-engine/policy_package.md + image: forgejo.coulomb.social/coulomb/flex-auth@sha256:a31961c45215aa6baf3bc748c6741ab703c2c8325e61aa7983a355026195e51b + livenessProbe: + httpGet: + path: /healthz + port: http + periodSeconds: 20 + name: flex-auth + ports: + - containerPort: 8080 + name: http + readinessProbe: + httpGet: + path: /healthz + port: http + periodSeconds: 5 + resources: + limits: + cpu: 300m + memory: 192Mi + requests: + cpu: 25m + memory: 32Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault +--- +apiVersion: v1 +kind: Service +metadata: + name: flex-auth-user-engine + namespace: flex-auth +spec: + ports: + - name: http + port: 8080 + targetPort: http + selector: + app.kubernetes.io/name: flex-auth-user-engine +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: flex-auth-user-engine + namespace: flex-auth +spec: + egress: [] + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: user-engine + podSelector: + matchLabels: + app.kubernetes.io/name: user-engine + ports: + - port: 8080 + protocol: TCP + podSelector: + matchLabels: + app.kubernetes.io/name: flex-auth-user-engine + policyTypes: + - Ingress + - Egress From e9911eb77f0eb6309fd83002ee5fba19eaab4e0a Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 11 Aug 2026 10:35:34 +0200 Subject: [PATCH 2/2] Publish images via CI; file staged-promotion overlay debt flex-auth had no image workflow, so its production images were hand-built on a workstation and pushed with workstation credentials -- an artifact whose provenance was someone's working tree rather than a forge revision. Ports activity-core's canonical image.yaml: the container-build runner fetches a tarball of the pushed commit, builds, and pushes :latest and :main-, then reports the immutable digest for the rollout step. examples/** is a build-trigger path on purpose -- policy packages are COPYed into the image and there is no hot reload, so a policy change is an image change. Runbook updated to say plainly that images are not built on workstations. FLEX-WP-0011 filed for the rest of the gap: flex-auth runs two production Deployments on railiance01 but has never been brought under the staged-promotion contract (RAIL-BS-WP-0006). No railiance/app.toml, no stage commands, no canary or approval evidence; the deploy/ directory is a rescue of specs that existed nowhere, not the sanctioned overlay shape. Pre-existing debt found during the FLEX-WP-0010 rollout, not a regression from it, and not a blocker for routine policy rollouts -- flex-auth fails closed. T03 also flags that the coulombcore drain plan still lists flex-auth on a host it no longer runs on. Co-Authored-By: Claude Opus 5 --- .forgejo/workflows/image.yaml | 65 ++++++++++ deploy/README.md | 18 ++- ...0011-railiance-staged-promotion-overlay.md | 120 ++++++++++++++++++ 3 files changed, 196 insertions(+), 7 deletions(-) create mode 100644 .forgejo/workflows/image.yaml create mode 100644 workplans/FLEX-WP-0011-railiance-staged-promotion-overlay.md diff --git a/.forgejo/workflows/image.yaml b/.forgejo/workflows/image.yaml new file mode 100644 index 0000000..29aa359 --- /dev/null +++ b/.forgejo/workflows/image.yaml @@ -0,0 +1,65 @@ +name: Build and Publish Container Image + +# Modelled on activity-core/.forgejo/workflows/image.yaml -- the fleet's +# canonical image-publish pattern. Images are built by CI from a tarball of +# the pushed commit, never from a workstation working tree, so the artifact's +# provenance is a forge revision. +# +# `examples/**` is a build-trigger path on purpose: the policy packages are +# COPYed into the image (see Containerfile) and there is no hot reload, so a +# policy change is a new image. + +on: + push: + branches: + - main + paths: + - ".forgejo/workflows/image.yaml" + - "Containerfile" + - "cmd/**" + - "internal/**" + - "pkg/**" + - "examples/**" + - "go.mod" + - "go.sum" + workflow_dispatch: + +env: + REGISTRY: forgejo.coulomb.social + IMAGE_NAME: coulomb/flex-auth + DOCKER_HOST: tcp://127.0.0.1:2375 + +jobs: + build-and-push: + runs-on: container-build + steps: + - name: Build and push image + env: + REGISTRY_USER: ${{ secrets.REGISTRY_USER }} + REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }} + run: | + set -eu + REF="${GITHUB_SHA:-main}" + SHORT="${REF:0:7}" + mkdir -p buildctx "${HOME}/bin" + wget -qO /tmp/repo.tar.gz \ + "https://forgejo.coulomb.social/${GITHUB_REPOSITORY}/archive/${SHORT}.tar.gz" + tar xzf /tmp/repo.tar.gz -C buildctx --strip-components=1 + wget -qO- https://download.docker.com/linux/static/stable/x86_64/docker-27.3.1.tgz \ + | tar xz --strip-components=1 -C "${HOME}/bin" docker/docker + export PATH="${HOME}/bin:${PATH}" + echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" -u "${REGISTRY_USER}" --password-stdin + IMAGE="${REGISTRY}/${IMAGE_NAME}" + docker build -f buildctx/Containerfile -t "${IMAGE}:latest" -t "${IMAGE}:main-${SHORT}" buildctx + docker push "${IMAGE}:latest" + docker push "${IMAGE}:main-${SHORT}" + echo "pushed ${IMAGE}:latest and ${IMAGE}:main-${SHORT}" + + - name: Report immutable digest + run: | + set -eu + export PATH="${HOME}/bin:${PATH}" + IMAGE="${REGISTRY}/${IMAGE_NAME}" + SHORT="${GITHUB_SHA:0:7}" + # Deployments pin by digest, never by tag -- print it for the rollout step. + docker inspect --format='{{index .RepoDigests 0}}' "${IMAGE}:main-${SHORT}" diff --git a/deploy/README.md b/deploy/README.md index ab04e27..3484ae6 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -44,14 +44,18 @@ actually changed. The policy packages are baked into the image, not mounted from a ConfigMap, so a policy change requires a rebuild — there is no hot reload. -```bash -# 1. Build and push from a clean checkout of the commit you intend to ship -docker build -t forgejo.coulomb.social/coulomb/flex-auth: -f Containerfile . -docker push forgejo.coulomb.social/coulomb/flex-auth: +**Do not build images on a workstation.** The fleet builds in CI so that an +artifact's provenance is a forge revision rather than someone's working tree. +`.forgejo/workflows/image.yaml` handles it, and `examples/**` is one of its +trigger paths precisely because policy changes are image changes. -# 2. Resolve the digest and pin it — deploy by digest, never by tag -docker inspect --format='{{index .RepoDigests 0}}' \ - forgejo.coulomb.social/coulomb/flex-auth: +```bash +# 1. Push the commit you intend to ship; CI builds it on the container-build +# runner and pushes :latest and :main- +git push origin main + +# 2. Take the immutable digest from the workflow's "Report immutable digest" +# step -- deploy by digest, never by tag # 3. Edit the image digest in the relevant manifest, then apply kubectl apply -f deploy/flex-auth-.yaml diff --git a/workplans/FLEX-WP-0011-railiance-staged-promotion-overlay.md b/workplans/FLEX-WP-0011-railiance-staged-promotion-overlay.md new file mode 100644 index 0000000..9424017 --- /dev/null +++ b/workplans/FLEX-WP-0011-railiance-staged-promotion-overlay.md @@ -0,0 +1,120 @@ +--- +id: FLEX-WP-0011 +type: workplan +title: "Bring flex-auth under the railiance staged-promotion contract" +domain: infotech +repo: flex-auth +status: proposed +owner: codex +topic_slug: netkingdom +planning_priority: P2 +planning_order: 110 +depends_on_workplans: + - FLEX-WP-0009 +related_workplans: + - RAIL-BS-WP-0006 +created: "2026-08-11" +updated: "2026-08-11" +--- + +# FLEX-WP-0011 - Bring flex-auth under the railiance staged-promotion contract + +flex-auth runs two production Deployments on **railiance01** +(`flex-auth-tenant-engine`, `flex-auth-user-engine`) but has never been +brought under the staged-promotion contract (`RAIL-BS-WP-0006`) that defines +what counts as production on that host. This is pre-existing debt discovered +during the `FLEX-WP-0010` rollout, not a regression introduced by it. + +## What is missing + +The contract's gate table requires: + +| Gate | Requirement | flex-auth today | +|---|---|---| +| Overlay repo | `railiance//` with `app.toml` and stage manifests | **missing** — manifests live in `deploy/*.yaml`, a parallel convention | +| Stage commands | `stage deploy`, `stage observe`, `stage promote`, `stage rollback` proven | **missing** | +| Evidence | Backup/restore drill, canary observation, operator approval recorded | partial — probe evidence exists in workplans, no canary or approval record | +| Registry | Image in forge OCI registry with immutable tag | **met** — Deployments pin by digest | + +Two related facts found at the same time: + +- The Deployments were originally created by `kubectl-client-side-apply` from + a file that existed in no repository. `FLEX-WP-0010` recovered them into + `deploy/` from the live objects' `last-applied-configuration` so that + rollback no longer depends on a cluster annotation. That directory is a + rescue, not the sanctioned shape, and this workplan should absorb it. +- CI image publishing was added in `FLEX-WP-0010` + (`.forgejo/workflows/image.yaml`, modelled on activity-core). Before that, + images were hand-built on a workstation with no forge-verifiable + provenance. The registry gate is now met by construction. + +## Not a blocker for existing traffic + +flex-auth is already serving production decisions and its failure mode is +fail-closed: consumers receive `deny`/`403` rather than an unauthorized +allow. This workplan closes a governance and recoverability gap, not an +active safety incident. It should not be used as a reason to hold routine +policy rollouts. + +## T01 - Author the railiance overlay + +```task +id: FLEX-WP-0011-T01 +status: todo +priority: medium +``` + +Write `railiance/app.toml` against schema `railiance.app.v1`, using +`qonto-assistant/railiance/app.toml` as the reference. Declare `[app]` with +an honest `criticality` (flex-auth is an authorization PDP whose outage +blocks consumer writes fail-closed), `[source]` with `digest_policy = +"required"`, `[rollback]` naming the concrete rollback command, and +`observability.health_endpoints` for both Services' `/healthz`. + +Model stage1/stage2/stage3 on the two-Deployment reality: the two consumers +are independently pinned and must be independently rollable. Do not collapse +them into one release. + +Reshape `deploy/*.yaml` into the overlay layout (compare +`activity-core/k8s/railiance/`) and keep `deploy/README.md`'s rollout and +rollback runbook content rather than discarding it. + +Done when the overlay renders and a server-side dry run of the full manifest +set is accepted. + +## T02 - Prove the stage commands + +```task +id: FLEX-WP-0011-T02 +status: todo +priority: medium +``` + +Prove `stage deploy`, `stage observe`, `stage promote`, and `stage rollback` +against a real candidate image, with a canary observation window and a +recorded operator approval. Record the previous stable digest as the rollback +target before promoting. + +Done when a full promote-then-rollback cycle has been demonstrated and the +evidence ids are recorded here. + +## T03 - Correct the stale drain-plan row + +```task +id: FLEX-WP-0011-T03 +status: todo +priority: low +``` + +`the-custodian/docs/coulombcore-drain-placement-plan.md` row 23 lists +flex-auth as living on coulombcore, drain wave 7, status `grandfathered`, +target railiance01. flex-auth in fact already runs on railiance01 +(`92.205.62.239`); the row is stale and understates progress. + +This is custodian canon, not a flex-auth file — raise it with the custodian +owner rather than editing it from this repo. Confirm at the same time whether +a coulombcore flex-auth deployment still exists and needs retiring, or +whether wave 7 can be closed. + +Done when the row reflects reality or a custodian decision id explains why it +stands.