Commit production deployment manifests and rollout runbook
The two flex-auth Deployments were applied from a YAML file that lived outside this repo, so the only surviving copy of their authored spec was the last-applied-configuration annotation on the live objects. That made rollback depend on a cluster annotation -- recovered them into deploy/ before the FLEX-WP-0010 rollout rather than after an incident. Covers Deployment, Service, and the default-deny NetworkPolicy for both flex-auth-tenant-engine and flex-auth-user-engine. Verified with kubectl apply --dry-run=server: Deployments and Services round-trip as unchanged. The NetworkPolicies report "configured" because the manifests keep an explicit egress: [] that the API server normalises away -- same deny-all semantics, documented so nobody chases it. The runbook records that policy packages are baked into the image with no hot reload, that both Deployments share one image repo at deliberately different digests (so roll only the one whose policy changed), and pins both last-known-good digests as rollback targets. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
6f7ab6fd99
commit
2456287e8c
3 changed files with 278 additions and 0 deletions
88
deploy/README.md
Normal file
88
deploy/README.md
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
# flex-auth production deployment
|
||||
|
||||
Manifests for the two cluster-local flex-auth policy-decision services.
|
||||
Both were previously applied with `kubectl apply` from a file that lived
|
||||
outside this repo; these were recovered from the live objects'
|
||||
`kubectl.kubernetes.io/last-applied-configuration` on 2026-08-11 and
|
||||
committed so that a rollback does not depend on a cluster annotation.
|
||||
|
||||
| File | Deployment | Consumer | Service DNS |
|
||||
| --- | --- | --- | --- |
|
||||
| `flex-auth-tenant-engine.yaml` | `flex-auth-tenant-engine` | tenant-engine write API | `flex-auth-tenant-engine.flex-auth.svc.cluster.local:8080` |
|
||||
| `flex-auth-user-engine.yaml` | `flex-auth-user-engine` | user-engine portal | `flex-auth-user-engine.flex-auth.svc.cluster.local:8080` |
|
||||
|
||||
Each file is a three-document manifest: `Deployment`, `Service`, and a
|
||||
default-deny `NetworkPolicy` whose ingress is restricted to the one approved
|
||||
consumer workload and which permits no egress.
|
||||
|
||||
### A harmless diff on apply
|
||||
|
||||
`kubectl apply` reports the two NetworkPolicies as `configured` rather than
|
||||
`unchanged`, every time. That is not drift: the manifests carry an explicit
|
||||
`egress: []`, which the API server normalises away on read. With
|
||||
`policyTypes: [Ingress, Egress]` and no egress rules, deny-all egress holds
|
||||
either way. The empty list is kept because it states the intent to a reader
|
||||
instead of leaving it implicit. Deployments and Services do round-trip as
|
||||
`unchanged`.
|
||||
|
||||
## One image, two deployments
|
||||
|
||||
Both Deployments run the **same image repository** and differ only in their
|
||||
`--registry` / `--policy` arguments. The `Containerfile` does
|
||||
`COPY examples /opt/flex-auth/examples`, so every image contains *every*
|
||||
consumer's policy package; the arguments select which one that instance
|
||||
serves.
|
||||
|
||||
Consequence worth remembering: rebuilding to pick up one consumer's policy
|
||||
change also re-bakes every other consumer's policy into the new image. The
|
||||
two Deployments are pinned to **different digests** precisely so that one can
|
||||
be rolled without moving the other. Roll only the Deployment whose policy
|
||||
actually changed.
|
||||
|
||||
## Rolling out a policy change
|
||||
|
||||
The policy packages are baked into the image, not mounted from a ConfigMap,
|
||||
so a policy change requires a rebuild — there is no hot reload.
|
||||
|
||||
```bash
|
||||
# 1. Build and push from a clean checkout of the commit you intend to ship
|
||||
docker build -t forgejo.coulomb.social/coulomb/flex-auth:<tag> -f Containerfile .
|
||||
docker push forgejo.coulomb.social/coulomb/flex-auth:<tag>
|
||||
|
||||
# 2. Resolve the digest and pin it — deploy by digest, never by tag
|
||||
docker inspect --format='{{index .RepoDigests 0}}' \
|
||||
forgejo.coulomb.social/coulomb/flex-auth:<tag>
|
||||
|
||||
# 3. Edit the image digest in the relevant manifest, then apply
|
||||
kubectl apply -f deploy/flex-auth-<consumer>.yaml
|
||||
kubectl -n flex-auth rollout status deploy/flex-auth-<consumer> --timeout=120s
|
||||
|
||||
# 4. Verify the new policy actually took effect, from outside the cluster
|
||||
kubectl -n flex-auth port-forward svc/flex-auth-<consumer> 19099:8080 &
|
||||
curl -s -X POST http://127.0.0.1:19099/v1/check \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d @examples/<consumer>/<a request that exercises the change>.json
|
||||
```
|
||||
|
||||
Step 4 is not optional. Because the policy ships inside the image, a
|
||||
successful `rollout status` only proves the container started — it says
|
||||
nothing about which policy revision is being served.
|
||||
|
||||
## Rollback
|
||||
|
||||
```bash
|
||||
kubectl -n flex-auth rollout undo deploy/flex-auth-<consumer>
|
||||
```
|
||||
|
||||
If the ReplicaSet history has been pruned, re-apply the manifest with the
|
||||
last-known-good digest below.
|
||||
|
||||
| Deployment | Last-known-good digest | Policy state |
|
||||
| --- | --- | --- |
|
||||
| `flex-auth-tenant-engine` | `sha256:c25fc34a6cd7e64d955f8723ec70e176a583d5ae71d76280c4e2d89fba0fe0aa` | four-action policy, pre-FLEX-WP-0010 (lifecycle actions deny `unknown_action`) |
|
||||
| `flex-auth-user-engine` | `sha256:a31961c45215aa6baf3bc748c6741ab703c2c8325e61aa7983a355026195e51b` | FLEX-WP-0009-T03, six fixtures verified live 2026-08-10 |
|
||||
|
||||
Rolling back the tenant-engine service to `c25fc34a…` restores fail-closed
|
||||
behaviour for the lifecycle actions — tenant-engine's lifecycle endpoints
|
||||
return `403 write_denied` rather than writing. That is a safe failure mode,
|
||||
not an outage of the older four actions, which keep working.
|
||||
95
deploy/flex-auth-tenant-engine.yaml
Normal file
95
deploy/flex-auth-tenant-engine.yaml
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: flex-auth-tenant-engine
|
||||
namespace: flex-auth
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: flex-auth-tenant-engine
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: flex-auth-tenant-engine
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
containers:
|
||||
- args:
|
||||
- serve
|
||||
- --addr
|
||||
- 0.0.0.0:8080
|
||||
- --registry
|
||||
- /opt/flex-auth/examples/tenant-engine/registry_snapshot.json
|
||||
- --policy
|
||||
- /opt/flex-auth/examples/tenant-engine/policy_package.md
|
||||
image: forgejo.coulomb.social/coulomb/flex-auth@sha256:c25fc34a6cd7e64d955f8723ec70e176a583d5ae71d76280c4e2d89fba0fe0aa
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
periodSeconds: 20
|
||||
name: flex-auth
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
periodSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: 300m
|
||||
memory: 192Mi
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: flex-auth-tenant-engine
|
||||
namespace: flex-auth
|
||||
spec:
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
targetPort: http
|
||||
selector:
|
||||
app.kubernetes.io/name: flex-auth-tenant-engine
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: flex-auth-tenant-engine
|
||||
namespace: flex-auth
|
||||
spec:
|
||||
egress: []
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: tenant-engine
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tenant-engine
|
||||
ports:
|
||||
- port: 8080
|
||||
protocol: TCP
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: flex-auth-tenant-engine
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
95
deploy/flex-auth-user-engine.yaml
Normal file
95
deploy/flex-auth-user-engine.yaml
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: flex-auth-user-engine
|
||||
namespace: flex-auth
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: flex-auth-user-engine
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: flex-auth-user-engine
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
containers:
|
||||
- args:
|
||||
- serve
|
||||
- --addr
|
||||
- 0.0.0.0:8080
|
||||
- --registry
|
||||
- /opt/flex-auth/examples/user-engine/registry_snapshot.json
|
||||
- --policy
|
||||
- /opt/flex-auth/examples/user-engine/policy_package.md
|
||||
image: forgejo.coulomb.social/coulomb/flex-auth@sha256:a31961c45215aa6baf3bc748c6741ab703c2c8325e61aa7983a355026195e51b
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
periodSeconds: 20
|
||||
name: flex-auth
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
periodSeconds: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: 300m
|
||||
memory: 192Mi
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: flex-auth-user-engine
|
||||
namespace: flex-auth
|
||||
spec:
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
targetPort: http
|
||||
selector:
|
||||
app.kubernetes.io/name: flex-auth-user-engine
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: flex-auth-user-engine
|
||||
namespace: flex-auth
|
||||
spec:
|
||||
egress: []
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: user-engine
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: user-engine
|
||||
ports:
|
||||
- port: 8080
|
||||
protocol: TCP
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: flex-auth-user-engine
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
Loading…
Add table
Add a link
Reference in a new issue