Add Forgejo break-glass seal/verify tooling and recovery runbook

RAPPS-WP-0016-T01: founder-run targets rotate the forgejo_admin password into OpenBao (platform/workloads/forgejo/break-glass-admin) and record a value-free sign-in. T01 stays in progress until the founder runs them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1031639@bnt-lap001 Assistant-Session: e5073fee-f4ae-4fc3-a75a-6c1fc83e0576
2026-10-17 11:32:09 +02:00 · 2026-10-17 11:32:09 +02:00 · fe5cabecd6
commit fe5cabecd6
parent fbe012789e
5 changed files with 166 additions and 1 deletions
--- a/6
+++ b/6
@ -514,6 +514,12 @@ forgejo-verify: check-railiance01-kubeconfig ## T05 acceptance: smoke, TLS, DB,
 forgejo-operator-bootstrap: ## Idempotent operator user + SSH key bootstrap (no secrets in Git)
 	FORGEJO_BASE_URL="$(FORGEJO_BASE_URL)" tools/forgejo-operator-bootstrap.sh

+forgejo-breakglass-seal: check-railiance01-kubeconfig ## Founder: rotate break-glass admin password into OpenBao (interactive, production-touching)
+	FORGEJO_KUBECONFIG="$(FORGEJO_KUBECONFIG)" FORGEJO_NAMESPACE="$(FORGEJO_NAMESPACE)" tools/forgejo-breakglass-seal.sh
+
+forgejo-breakglass-verify: ## Founder: verify one break-glass sign-in and write value-free evidence
+	FORGEJO_BASE_URL="$(FORGEJO_BASE_URL)" tools/forgejo-breakglass-verify.sh
+
 forgejo-runner-registration-sops-bootstrap: check-railiance01-kubeconfig ## Capture cluster runner token into SOPS (operator age key)
 	KUBECONFIG="$(FORGEJO_KUBECONFIG)" tools/forgejo-runner-registration-sops-bootstrap.sh

--- a/docs/forgejo-on-railiance01.md
+++ b/docs/forgejo-on-railiance01.md
@ -104,6 +104,46 @@ make forgejo-runner-registration-sops-bootstrap
 make check-sops SOPS_SENTINEL=helm/forgejo-runner-registration.sops.yaml
 ```

+## Break-glass administrator (RAPPS-WP-0016-T01)
+
+The floor under every sign-in change: one local site admin that works even when
+NetKingdom, KeyCape or the OAuth source is broken.
+
+| Item | Value |
+| --- | --- |
+| Account | `forgejo_admin` (chart bootstrap admin, local password, site admin) |
+| Custody | OpenBao coulombcore `platform/workloads/forgejo/break-glass-admin`, field `PASSWORD` |
+| Delivered to cluster | No. Not synced by ESO; only a human retrieves it |
+| Who | The founder. Agents never read or hold the value |
+
+The SOPS admin password in `helm/forgejo-secrets.sops.yaml` is only the chart's
+initial value (`passwordMode: initialOnlyRequireReset`); Helm upgrades do not
+reset the account. It is not the break-glass credential.
+
+Seal or rotate, then verify (founder, interactive):
+
+```bash
+make forgejo-breakglass-seal     # new password -> OpenBao first, then Forgejo
+make forgejo-breakglass-verify   # one local sign-in; writes docs/evidence/<date>-forgejo-breakglass-signin.json
+```
+
+Recovery when normal sign-in is broken:
+
+1. `bao kv get -field=PASSWORD platform/workloads/forgejo/break-glass-admin`
+   (operator token on `https://bao.coulomb.social`).
+2. Sign in at `https://forgejo.coulomb.social/user/login` as `forgejo_admin`
+   with local credentials. Local password sign-in stays enabled for this account
+   even if RAPPS-WP-0016-T04 restricts it for everyone else.
+3. Repair the authentication source or users from Site Administration.
+4. If the password itself is lost: with cluster access, run
+   `make forgejo-breakglass-seal` again (it sets the password through the
+   Forgejo CLI in the pod, so it needs no working sign-in).
+5. Afterwards rotate (`make forgejo-breakglass-seal`), verify, and record the use.
+
+Where it is logged: OpenBao audit log (retrieval), Forgejo log and the account's
+last-login time (sign-in), the evidence file in `docs/evidence/`, and a State Hub
+progress event naming why break-glass was used.
+
 ## SMTP (IONOS) — password reset / notifications

 Decision: `the-custodian/docs/forgejo-production-decisions.md` (IONOS, 2026-07-07).
--- a/tools/forgejo-breakglass-seal.sh
+++ b/tools/forgejo-breakglass-seal.sh
@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+# Seal the Forgejo break-glass administrator (RAPPS-WP-0016-T01).
+# Founder-run, interactive. Generates a fresh password for the local site admin,
+# stores it only in OpenBao (coulombcore), then sets it in Forgejo. The value is
+# never printed, logged or written to disk.
+set -euo pipefail
+
+BREAKGLASS_USER="${FORGEJO_BREAKGLASS_USER:-forgejo_admin}"
+BREAKGLASS_PATH="${FORGEJO_BREAKGLASS_PATH:-platform/workloads/forgejo/break-glass-admin}"
+BAO_ADDR="${BAO_ADDR:-https://bao.coulomb.social}"
+FORGEJO_KUBECONFIG="${FORGEJO_KUBECONFIG:-$HOME/.kube/config-hosteurope}"
+FORGEJO_NAMESPACE="${FORGEJO_NAMESPACE:-forgejo}"
+
+for cmd in bao kubectl openssl; do
+  command -v "$cmd" >/dev/null 2>&1 || { echo "ERROR: $cmd not found" >&2; exit 1; }
+done
+
+if [[ -z "${BAO_TOKEN:-}" ]]; then
+  read -r -s -p "OpenBao operator token (coulombcore / bao.coulomb.social): " BAO_TOKEN
+  echo >&2
+fi
+[[ -n "${BAO_TOKEN:-}" ]] || { echo "ERROR: empty OpenBao token" >&2; exit 1; }
+export BAO_ADDR BAO_TOKEN
+
+if curl -fsS "$BAO_ADDR/v1/sys/health" | grep -q '"sealed":true'; then
+  echo "ERROR: OpenBao at $BAO_ADDR reports sealed" >&2
+  exit 1
+fi
+
+pod="$(KUBECONFIG="$FORGEJO_KUBECONFIG" kubectl -n "$FORGEJO_NAMESPACE" get pod \
+  -l app.kubernetes.io/name=gitea -o jsonpath='{.items[0].metadata.name}')"
+[[ -n "$pod" ]] || { echo "ERROR: no Forgejo pod in $FORGEJO_NAMESPACE" >&2; exit 1; }
+
+echo "Break-glass user: $BREAKGLASS_USER (site admin, local sign-in)"
+echo "Custody path:     $BREAKGLASS_PATH (field PASSWORD) on $BAO_ADDR"
+echo "Forgejo pod:      $pod"
+read -r -p "Rotate the break-glass password now? [y/N] " answer
+[[ "$answer" == "y" ]] || { echo "Aborted; nothing changed."; exit 1; }
+
+password="$(openssl rand -base64 48 | tr -d '\n/+=' | cut -c1-40)"
+
+# Store first: if this fails, Forgejo still has the old password.
+printf '%s' "$password" | bao kv put "$BREAKGLASS_PATH" \
+  PASSWORD=- USERNAME="$BREAKGLASS_USER" ROTATED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" >/dev/null
+stored="$(bao kv get -field=PASSWORD "$BREAKGLASS_PATH")"
+[[ "$stored" == "$password" ]] || { echo "ERROR: OpenBao read-back mismatch; Forgejo unchanged" >&2; exit 1; }
+unset stored
+
+# Password goes to the pod on stdin so it never appears in process arguments
+# on the workstation.
+printf '%s\n' "$password" | KUBECONFIG="$FORGEJO_KUBECONFIG" kubectl -n "$FORGEJO_NAMESPACE" \
+  exec -i "$pod" -c gitea -- sh -c \
+  'read -r p; forgejo admin user change-password --username "$0" --password "$p" --must-change-password=false >/dev/null' \
+  "$BREAKGLASS_USER"
+unset password
+
+echo "Sealed: $BREAKGLASS_USER password rotated and stored at $BREAKGLASS_PATH."
+echo "Next: make forgejo-breakglass-verify"
--- a/tools/forgejo-breakglass-verify.sh
+++ b/tools/forgejo-breakglass-verify.sh
@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+# Verify one break-glass sign-in (RAPPS-WP-0016-T01) and write value-free evidence.
+# Founder-run. Reads the password from OpenBao into memory, authenticates against
+# the Forgejo API with local credentials, and records only the outcome.
+set -euo pipefail
+
+BREAKGLASS_USER="${FORGEJO_BREAKGLASS_USER:-forgejo_admin}"
+BREAKGLASS_PATH="${FORGEJO_BREAKGLASS_PATH:-platform/workloads/forgejo/break-glass-admin}"
+BAO_ADDR="${BAO_ADDR:-https://bao.coulomb.social}"
+FORGEJO_BASE_URL="${FORGEJO_BASE_URL:-https://forgejo.coulomb.social}"
+EVIDENCE="${FORGEJO_BREAKGLASS_EVIDENCE:-docs/evidence/$(date -u +%Y-%m-%d)-forgejo-breakglass-signin.json}"
+
+command -v bao >/dev/null 2>&1 || { echo "ERROR: bao CLI not found" >&2; exit 1; }
+if [[ -z "${BAO_TOKEN:-}" ]]; then
+  read -r -s -p "OpenBao operator token (coulombcore / bao.coulomb.social): " BAO_TOKEN
+  echo >&2
+fi
+export BAO_ADDR BAO_TOKEN
+
+password="$(bao kv get -field=PASSWORD "$BREAKGLASS_PATH")"
+version="$(bao kv metadata get -format=json "$BREAKGLASS_PATH" \
+  | python3 -c 'import json,sys; print(json.load(sys.stdin)["data"]["current_version"])')"
+
+# Credentials reach curl through a config on stdin, not argv.
+body_file="$(mktemp)"
+trap 'rm -f "$body_file"' EXIT
+code="$(printf 'user = "%s:%s"\n' "$BREAKGLASS_USER" "$password" \
+  | curl -sS -K - -o "$body_file" -w '%{http_code}' "$FORGEJO_BASE_URL/api/v1/user")"
+unset password
+
+python3 - "$code" "$BREAKGLASS_USER" "$BREAKGLASS_PATH" "$version" "$FORGEJO_BASE_URL" \
+  "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$EVIDENCE" "$body_file" <<'EOF'
+import json, sys
+code, user, path, version, base, at, out, body = sys.argv[1:]
+try:
+    me = json.load(open(body))
+except ValueError:
+    me = {}
+ok = code == "200" and me.get("login") == user and me.get("is_admin") is True
+record = {
+    "schema": "railiance-apps.forgejo-breakglass-signin.v1",
+    "task": "RAPPS-WP-0016-T01",
+    "forgejo": base,
+    "account": user,
+    "auth": "local password (API basic auth)",
+    "http_status": int(code) if code.isdigit() else code,
+    "login_matches": me.get("login") == user,
+    "is_admin": me.get("is_admin"),
+    "custody": {"openbao_path": path, "field": "PASSWORD", "version": version},
+    "verified_at": at,
+    "result": "pass" if ok else "fail",
+    "values_emitted": False,
+}
+with open(out, "w") as f:
+    json.dump(record, f, indent=2)
+    f.write("\n")
+print(f"{record['result']}: {user} sign-in (HTTP {code}); evidence {out}")
+sys.exit(0 if ok else 1)
+EOF
--- a/workplans/RAPPS-WP-0016-forgejo-netkingdom-login.md
+++ b/workplans/RAPPS-WP-0016-forgejo-netkingdom-login.md
@ -48,7 +48,9 @@ side is key-cape `KEY-WP-0038`.

 ```task
 id: RAPPS-WP-0016-T01
-status: todo
+status: progress
+needs_human: true
+blocking_reason: "Tooling and runbook ready; founder runs make forgejo-breakglass-seal and make forgejo-breakglass-verify."
 priority: high
 state_hub_task_id: "98004ed0-83e6-5f6e-9b18-15b8b2aec2e7"
 ```