diff --git a/.forgejo/workflows/image.yaml b/.forgejo/workflows/image.yaml new file mode 100644 index 0000000..29aa359 --- /dev/null +++ b/.forgejo/workflows/image.yaml @@ -0,0 +1,65 @@ +name: Build and Publish Container Image + +# Modelled on activity-core/.forgejo/workflows/image.yaml -- the fleet's +# canonical image-publish pattern. Images are built by CI from a tarball of +# the pushed commit, never from a workstation working tree, so the artifact's +# provenance is a forge revision. +# +# `examples/**` is a build-trigger path on purpose: the policy packages are +# COPYed into the image (see Containerfile) and there is no hot reload, so a +# policy change is a new image. + +on: + push: + branches: + - main + paths: + - ".forgejo/workflows/image.yaml" + - "Containerfile" + - "cmd/**" + - "internal/**" + - "pkg/**" + - "examples/**" + - "go.mod" + - "go.sum" + workflow_dispatch: + +env: + REGISTRY: forgejo.coulomb.social + IMAGE_NAME: coulomb/flex-auth + DOCKER_HOST: tcp://127.0.0.1:2375 + +jobs: + build-and-push: + runs-on: container-build + steps: + - name: Build and push image + env: + REGISTRY_USER: ${{ secrets.REGISTRY_USER }} + REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }} + run: | + set -eu + REF="${GITHUB_SHA:-main}" + SHORT="${REF:0:7}" + mkdir -p buildctx "${HOME}/bin" + wget -qO /tmp/repo.tar.gz \ + "https://forgejo.coulomb.social/${GITHUB_REPOSITORY}/archive/${SHORT}.tar.gz" + tar xzf /tmp/repo.tar.gz -C buildctx --strip-components=1 + wget -qO- https://download.docker.com/linux/static/stable/x86_64/docker-27.3.1.tgz \ + | tar xz --strip-components=1 -C "${HOME}/bin" docker/docker + export PATH="${HOME}/bin:${PATH}" + echo "${REGISTRY_TOKEN}" | docker login "${REGISTRY}" -u "${REGISTRY_USER}" --password-stdin + IMAGE="${REGISTRY}/${IMAGE_NAME}" + docker build -f buildctx/Containerfile -t "${IMAGE}:latest" -t "${IMAGE}:main-${SHORT}" buildctx + docker push "${IMAGE}:latest" + docker push "${IMAGE}:main-${SHORT}" + echo "pushed ${IMAGE}:latest and ${IMAGE}:main-${SHORT}" + + - name: Report immutable digest + run: | + set -eu + export PATH="${HOME}/bin:${PATH}" + IMAGE="${REGISTRY}/${IMAGE_NAME}" + SHORT="${GITHUB_SHA:0:7}" + # Deployments pin by digest, never by tag -- print it for the rollout step. + docker inspect --format='{{index .RepoDigests 0}}' "${IMAGE}:main-${SHORT}" diff --git a/deploy/README.md b/deploy/README.md new file mode 100644 index 0000000..3484ae6 --- /dev/null +++ b/deploy/README.md @@ -0,0 +1,92 @@ +# flex-auth production deployment + +Manifests for the two cluster-local flex-auth policy-decision services. +Both were previously applied with `kubectl apply` from a file that lived +outside this repo; these were recovered from the live objects' +`kubectl.kubernetes.io/last-applied-configuration` on 2026-08-11 and +committed so that a rollback does not depend on a cluster annotation. + +| File | Deployment | Consumer | Service DNS | +| --- | --- | --- | --- | +| `flex-auth-tenant-engine.yaml` | `flex-auth-tenant-engine` | tenant-engine write API | `flex-auth-tenant-engine.flex-auth.svc.cluster.local:8080` | +| `flex-auth-user-engine.yaml` | `flex-auth-user-engine` | user-engine portal | `flex-auth-user-engine.flex-auth.svc.cluster.local:8080` | + +Each file is a three-document manifest: `Deployment`, `Service`, and a +default-deny `NetworkPolicy` whose ingress is restricted to the one approved +consumer workload and which permits no egress. + +### A harmless diff on apply + +`kubectl apply` reports the two NetworkPolicies as `configured` rather than +`unchanged`, every time. That is not drift: the manifests carry an explicit +`egress: []`, which the API server normalises away on read. With +`policyTypes: [Ingress, Egress]` and no egress rules, deny-all egress holds +either way. The empty list is kept because it states the intent to a reader +instead of leaving it implicit. Deployments and Services do round-trip as +`unchanged`. + +## One image, two deployments + +Both Deployments run the **same image repository** and differ only in their +`--registry` / `--policy` arguments. The `Containerfile` does +`COPY examples /opt/flex-auth/examples`, so every image contains *every* +consumer's policy package; the arguments select which one that instance +serves. + +Consequence worth remembering: rebuilding to pick up one consumer's policy +change also re-bakes every other consumer's policy into the new image. The +two Deployments are pinned to **different digests** precisely so that one can +be rolled without moving the other. Roll only the Deployment whose policy +actually changed. + +## Rolling out a policy change + +The policy packages are baked into the image, not mounted from a ConfigMap, +so a policy change requires a rebuild — there is no hot reload. + +**Do not build images on a workstation.** The fleet builds in CI so that an +artifact's provenance is a forge revision rather than someone's working tree. +`.forgejo/workflows/image.yaml` handles it, and `examples/**` is one of its +trigger paths precisely because policy changes are image changes. + +```bash +# 1. Push the commit you intend to ship; CI builds it on the container-build +# runner and pushes :latest and :main- +git push origin main + +# 2. Take the immutable digest from the workflow's "Report immutable digest" +# step -- deploy by digest, never by tag + +# 3. Edit the image digest in the relevant manifest, then apply +kubectl apply -f deploy/flex-auth-.yaml +kubectl -n flex-auth rollout status deploy/flex-auth- --timeout=120s + +# 4. Verify the new policy actually took effect, from outside the cluster +kubectl -n flex-auth port-forward svc/flex-auth- 19099:8080 & +curl -s -X POST http://127.0.0.1:19099/v1/check \ + -H 'Content-Type: application/json' \ + -d @examples//.json +``` + +Step 4 is not optional. Because the policy ships inside the image, a +successful `rollout status` only proves the container started — it says +nothing about which policy revision is being served. + +## Rollback + +```bash +kubectl -n flex-auth rollout undo deploy/flex-auth- +``` + +If the ReplicaSet history has been pruned, re-apply the manifest with the +last-known-good digest below. + +| Deployment | Last-known-good digest | Policy state | +| --- | --- | --- | +| `flex-auth-tenant-engine` | `sha256:c25fc34a6cd7e64d955f8723ec70e176a583d5ae71d76280c4e2d89fba0fe0aa` | four-action policy, pre-FLEX-WP-0010 (lifecycle actions deny `unknown_action`) | +| `flex-auth-user-engine` | `sha256:a31961c45215aa6baf3bc748c6741ab703c2c8325e61aa7983a355026195e51b` | FLEX-WP-0009-T03, six fixtures verified live 2026-08-10 | + +Rolling back the tenant-engine service to `c25fc34a…` restores fail-closed +behaviour for the lifecycle actions — tenant-engine's lifecycle endpoints +return `403 write_denied` rather than writing. That is a safe failure mode, +not an outage of the older four actions, which keep working. diff --git a/deploy/flex-auth-tenant-engine.yaml b/deploy/flex-auth-tenant-engine.yaml new file mode 100644 index 0000000..e0a71ad --- /dev/null +++ b/deploy/flex-auth-tenant-engine.yaml @@ -0,0 +1,95 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: flex-auth-tenant-engine + namespace: flex-auth +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: flex-auth-tenant-engine + template: + metadata: + labels: + app.kubernetes.io/name: flex-auth-tenant-engine + spec: + automountServiceAccountToken: false + containers: + - args: + - serve + - --addr + - 0.0.0.0:8080 + - --registry + - /opt/flex-auth/examples/tenant-engine/registry_snapshot.json + - --policy + - /opt/flex-auth/examples/tenant-engine/policy_package.md + image: forgejo.coulomb.social/coulomb/flex-auth@sha256:c25fc34a6cd7e64d955f8723ec70e176a583d5ae71d76280c4e2d89fba0fe0aa + livenessProbe: + httpGet: + path: /healthz + port: http + periodSeconds: 20 + name: flex-auth + ports: + - containerPort: 8080 + name: http + readinessProbe: + httpGet: + path: /healthz + port: http + periodSeconds: 5 + resources: + limits: + cpu: 300m + memory: 192Mi + requests: + cpu: 25m + memory: 32Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault +--- +apiVersion: v1 +kind: Service +metadata: + name: flex-auth-tenant-engine + namespace: flex-auth +spec: + ports: + - name: http + port: 8080 + targetPort: http + selector: + app.kubernetes.io/name: flex-auth-tenant-engine +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: flex-auth-tenant-engine + namespace: flex-auth +spec: + egress: [] + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: tenant-engine + podSelector: + matchLabels: + app.kubernetes.io/name: tenant-engine + ports: + - port: 8080 + protocol: TCP + podSelector: + matchLabels: + app.kubernetes.io/name: flex-auth-tenant-engine + policyTypes: + - Ingress + - Egress diff --git a/deploy/flex-auth-user-engine.yaml b/deploy/flex-auth-user-engine.yaml new file mode 100644 index 0000000..e973c37 --- /dev/null +++ b/deploy/flex-auth-user-engine.yaml @@ -0,0 +1,95 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: flex-auth-user-engine + namespace: flex-auth +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: flex-auth-user-engine + template: + metadata: + labels: + app.kubernetes.io/name: flex-auth-user-engine + spec: + automountServiceAccountToken: false + containers: + - args: + - serve + - --addr + - 0.0.0.0:8080 + - --registry + - /opt/flex-auth/examples/user-engine/registry_snapshot.json + - --policy + - /opt/flex-auth/examples/user-engine/policy_package.md + image: forgejo.coulomb.social/coulomb/flex-auth@sha256:a31961c45215aa6baf3bc748c6741ab703c2c8325e61aa7983a355026195e51b + livenessProbe: + httpGet: + path: /healthz + port: http + periodSeconds: 20 + name: flex-auth + ports: + - containerPort: 8080 + name: http + readinessProbe: + httpGet: + path: /healthz + port: http + periodSeconds: 5 + resources: + limits: + cpu: 300m + memory: 192Mi + requests: + cpu: 25m + memory: 32Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault +--- +apiVersion: v1 +kind: Service +metadata: + name: flex-auth-user-engine + namespace: flex-auth +spec: + ports: + - name: http + port: 8080 + targetPort: http + selector: + app.kubernetes.io/name: flex-auth-user-engine +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: flex-auth-user-engine + namespace: flex-auth +spec: + egress: [] + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: user-engine + podSelector: + matchLabels: + app.kubernetes.io/name: user-engine + ports: + - port: 8080 + protocol: TCP + podSelector: + matchLabels: + app.kubernetes.io/name: flex-auth-user-engine + policyTypes: + - Ingress + - Egress diff --git a/workplans/FLEX-WP-0011-railiance-staged-promotion-overlay.md b/workplans/FLEX-WP-0011-railiance-staged-promotion-overlay.md new file mode 100644 index 0000000..9424017 --- /dev/null +++ b/workplans/FLEX-WP-0011-railiance-staged-promotion-overlay.md @@ -0,0 +1,120 @@ +--- +id: FLEX-WP-0011 +type: workplan +title: "Bring flex-auth under the railiance staged-promotion contract" +domain: infotech +repo: flex-auth +status: proposed +owner: codex +topic_slug: netkingdom +planning_priority: P2 +planning_order: 110 +depends_on_workplans: + - FLEX-WP-0009 +related_workplans: + - RAIL-BS-WP-0006 +created: "2026-08-11" +updated: "2026-08-11" +--- + +# FLEX-WP-0011 - Bring flex-auth under the railiance staged-promotion contract + +flex-auth runs two production Deployments on **railiance01** +(`flex-auth-tenant-engine`, `flex-auth-user-engine`) but has never been +brought under the staged-promotion contract (`RAIL-BS-WP-0006`) that defines +what counts as production on that host. This is pre-existing debt discovered +during the `FLEX-WP-0010` rollout, not a regression introduced by it. + +## What is missing + +The contract's gate table requires: + +| Gate | Requirement | flex-auth today | +|---|---|---| +| Overlay repo | `railiance//` with `app.toml` and stage manifests | **missing** — manifests live in `deploy/*.yaml`, a parallel convention | +| Stage commands | `stage deploy`, `stage observe`, `stage promote`, `stage rollback` proven | **missing** | +| Evidence | Backup/restore drill, canary observation, operator approval recorded | partial — probe evidence exists in workplans, no canary or approval record | +| Registry | Image in forge OCI registry with immutable tag | **met** — Deployments pin by digest | + +Two related facts found at the same time: + +- The Deployments were originally created by `kubectl-client-side-apply` from + a file that existed in no repository. `FLEX-WP-0010` recovered them into + `deploy/` from the live objects' `last-applied-configuration` so that + rollback no longer depends on a cluster annotation. That directory is a + rescue, not the sanctioned shape, and this workplan should absorb it. +- CI image publishing was added in `FLEX-WP-0010` + (`.forgejo/workflows/image.yaml`, modelled on activity-core). Before that, + images were hand-built on a workstation with no forge-verifiable + provenance. The registry gate is now met by construction. + +## Not a blocker for existing traffic + +flex-auth is already serving production decisions and its failure mode is +fail-closed: consumers receive `deny`/`403` rather than an unauthorized +allow. This workplan closes a governance and recoverability gap, not an +active safety incident. It should not be used as a reason to hold routine +policy rollouts. + +## T01 - Author the railiance overlay + +```task +id: FLEX-WP-0011-T01 +status: todo +priority: medium +``` + +Write `railiance/app.toml` against schema `railiance.app.v1`, using +`qonto-assistant/railiance/app.toml` as the reference. Declare `[app]` with +an honest `criticality` (flex-auth is an authorization PDP whose outage +blocks consumer writes fail-closed), `[source]` with `digest_policy = +"required"`, `[rollback]` naming the concrete rollback command, and +`observability.health_endpoints` for both Services' `/healthz`. + +Model stage1/stage2/stage3 on the two-Deployment reality: the two consumers +are independently pinned and must be independently rollable. Do not collapse +them into one release. + +Reshape `deploy/*.yaml` into the overlay layout (compare +`activity-core/k8s/railiance/`) and keep `deploy/README.md`'s rollout and +rollback runbook content rather than discarding it. + +Done when the overlay renders and a server-side dry run of the full manifest +set is accepted. + +## T02 - Prove the stage commands + +```task +id: FLEX-WP-0011-T02 +status: todo +priority: medium +``` + +Prove `stage deploy`, `stage observe`, `stage promote`, and `stage rollback` +against a real candidate image, with a canary observation window and a +recorded operator approval. Record the previous stable digest as the rollback +target before promoting. + +Done when a full promote-then-rollback cycle has been demonstrated and the +evidence ids are recorded here. + +## T03 - Correct the stale drain-plan row + +```task +id: FLEX-WP-0011-T03 +status: todo +priority: low +``` + +`the-custodian/docs/coulombcore-drain-placement-plan.md` row 23 lists +flex-auth as living on coulombcore, drain wave 7, status `grandfathered`, +target railiance01. flex-auth in fact already runs on railiance01 +(`92.205.62.239`); the row is stale and understates progress. + +This is custodian canon, not a flex-auth file — raise it with the custodian +owner rather than editing it from this repo. Confirm at the same time whether +a coulombcore flex-auth deployment still exists and needs retiring, or +whether wave 7 can be closed. + +Done when the row reflects reality or a custodian decision id explains why it +stands.