Finish ACTIVITY-WP-0023: ESO FORGEJO_TOKEN and gap dispositions

Ship ExternalSecret actcore-forgejo-admin and multi-policy ESO token bootstrap so weekly prune credentials refresh without bootstrap inject. Disposition issue-core path A and MarkiTect ROS as external owners; close workplan finished.
2026-07-21 23:19:10 +02:00 · 2026-07-21 23:19:10 +02:00 · 21d46927f6
commit 21d46927f6
parent d36adb0822
7 changed files with 147 additions and 46 deletions
--- a/SCOPE.md
+++ b/SCOPE.md
@ -203,10 +203,10 @@ Open product/policy workplan: **ACTIVITY-WP-0022** (IssueSink no-default-Forgejo
 | **G2. Executor gap** | High (value) | Tasks are emitted (progress refs or issue-core), but **no per-repo executor** reliably consumes them (Binky briefs still need harness/agent pickup). INTENT assigns execution to per-repo Temporal workers — that substrate is thin. |
 | **G3. Side-effect resolvers** | Medium | Forgejo prune `apply: true` runs platform deletes inside a context resolver. Allowed only as declared, gated side-effect; needs clearer SCOPE language (above) and operator refresh of live-images protection. |
 | **G4. Review queue** | Medium | `review_required` remains metadata-only; no downstream review queue. |
-| **G5. TaskExecutorWorkflow stub** | Low–medium | Still registered; risk of attracting real execution. |
-| **G6. ROS digest completeness** | Medium (ops) | Template file mounted; MarkiTect CLI missing in state-hub image → digests still fail per domain. |
-| **G7. Credential delivery** | Medium (ops) | `FORGEJO_TOKEN` bootstrap-injected; ESO 403 on forgejo-admin path. issue-core `GITEA_BACKEND_TOKEN` still broken for path A rest sink. |
-| **G8. Live-images hygiene** | Medium (ops) | Multi-cluster `live-images-all.txt` must be refreshed after deploys or prune can delete live tags (incident 2026-07-21, restored). |
+| **G5. TaskExecutorWorkflow stub** | Low | Disabled unless `ACTIVITY_CORE_ENABLE_TASK_EXECUTOR_STUB` (WP-0023-T08). |
+| **G6. ROS digest completeness** | Medium (ops, external) | Templates mounted; **state-hub image** still lacks `markitect_tool` / MarkiTect CLI (`main-d8808bf` checked 2026-07-21). Owner: state-hub (WP-0023-T07). |
+| **G7. Credential delivery** | Low (residual) | **FORGEJO_TOKEN** via ESO `actcore-forgejo-admin` (WP-0023-T05, Ready). issue-core `GITEA_BACKEND_TOKEN` still 503 forgejo-inbox for path A rest — **issue-core owner** (WP-0023-T06). |
+| **G8. Live-images hygiene** | Medium (ops) | Multi-cluster `live-images-all.txt` must be refreshed after deploys or prune can delete live tags (incident 2026-07-21, restored). `scripts/refresh_live_images.sh` (T04). |
 | **G9. Evidence federation** | Low | Progress often lands on railiance01 edge/hub; workstation primary hub may not show the same feed without tunnel/outbox health. |
 | **G10. API external access** | Low | ClusterIP-only; intentional until auth policy. |

--- a/docs/issue-core-emission-boundary.md
+++ b/docs/issue-core-emission-boundary.md
@ -64,17 +64,28 @@ activity-core.
  Railiance production uses `state-hub` unless an overlay explicitly sets
  `rest` for an experiment.

-### Known production failure (2026-07-21) — rest path
+### Known production failure (2026-07-21, reconfirmed 2026-07-21 T06) — rest path

-`POST /issues/` returned **HTTP 503**:
+`POST /issues/` returns **HTTP 503**:

 ```text
 Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API
 ```

-`GITEA_BACKEND_TOKEN` on issue-core rejected by Forgejo. Fix is issue-core
-token rotation (`warden route show issue-core-ingestion-api-key`), not
-activity-core defaults.
+**Disposition (ACTIVITY-WP-0023-T06):** activity-core keeps global default
+`state-hub` and does **not** flip production to `rest`. Path A is owned by
+**issue-core**: rotate/fix `GITEA_BACKEND_TOKEN` (Forgejo backend PAT for the
+forgejo-inbox connector — not the activity-core `ISSUE_CORE_API_KEY`
+ingestion key). After issue-core proves `POST /issues/` → **201**, operators may
+opt in per definition / overlay only (WP-0022).
+
+Smoke from worker (does not change sink env):
+
+```bash
+kubectl -n activity-core exec deploy/actcore-worker -- python -c '
+# valid IssueCore body → expect 201 when path A is healthy, else 503 forgejo-inbox
+'
+```

 ## Promotion to rest (one definition at a time)

--- a/docs/runbook.md
+++ b/docs/runbook.md
@ -364,7 +364,7 @@ mount of `~/railiance-platform` on the worker) with `apply: true` and posts
 | Retention | newest **3** versions per package (`container`, `pypi`, `npm`, `generic`) |
 | Org | `coulomb` |
 | Protected | live cluster image tags + Helm values (`--live-images-file` / live scan) |
-| Credential | `FORGEJO_TOKEN` in `actcore-runtime-secret` (OpenBao path `platform/workloads/forgejo/forgejo-admin` field `API_TOKEN`; `warden route show forgejo-admin-api-token`) |
+| Credential | `FORGEJO_TOKEN` in `actcore-runtime-secret` via ExternalSecret `actcore-forgejo-admin` (OpenBao `platform/workloads/forgejo/forgejo-admin` field `API_TOKEN`; `warden route show forgejo-admin-api-token`). ESO token: `scripts/openbao-eso-token-apply.sh` (includes `workload-kv-read-forgejo-admin`). |
 | Rollback | restore package versions from Nextcloud `forgejo dump` if a needed tag was removed |

 **Enabled 2026-07-21** after dry-run + first apply evidence
--- a/k8s/railiance/15-externalsecret-forgejo-admin.yaml
+++ b/k8s/railiance/15-externalsecret-forgejo-admin.yaml
@ -0,0 +1,41 @@
+# Sync FORGEJO_TOKEN from OpenBao into actcore-runtime-secret.
+#
+# Prereqs on railiance01:
+#   - External Secrets Operator (namespace external-secrets)
+#   - ClusterSecretStore openbao-activity-core (railiance-platform addon)
+#   - Secret external-secrets/openbao-activity-core-eso-token with policies:
+#       workload-kv-read-issue-core-runtime
+#       workload-kv-read-forgejo-admin
+#     (scripts/openbao-eso-token-apply.sh)
+#
+# OpenBao path: platform/workloads/forgejo/forgejo-admin
+#   property: API_TOKEN  →  secret key FORGEJO_TOKEN
+# Catalog: warden route show forgejo-admin-api-token
+#
+# Merge keeps bootstrap-owned keys (ACTCORE_DB_URL, webhook secrets) and the
+# ISSUE_CORE_API_KEY ExternalSecret merge intact. Retires one-shot
+# `kubectl patch secret` bootstrap for the weekly package prune (WP-0023-T05).
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: actcore-forgejo-admin
+  namespace: activity-core
+  labels:
+    app.kubernetes.io/name: activity-core
+    app.kubernetes.io/part-of: activity-core
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao-activity-core
+  target:
+    name: actcore-runtime-secret
+    creationPolicy: Merge
+    deletionPolicy: Retain
+  data:
+    - secretKey: FORGEJO_TOKEN
+      remoteRef:
+        key: platform/workloads/forgejo/forgejo-admin
+        property: API_TOKEN
--- a/k8s/railiance/README.md
+++ b/k8s/railiance/README.md
@ -12,6 +12,8 @@ name and access policy.
  NATS JetStream, Temporal, and Temporal UI
 - `15-externalsecret-issue-core.yaml`: OpenBao → `ISSUE_CORE_API_KEY` merge into
  `actcore-runtime-secret` via External Secrets
+- `15-externalsecret-forgejo-admin.yaml`: OpenBao → `FORGEJO_TOKEN` merge for
+  weekly package prune (ACTIVITY-WP-0023-T05)
 - `20-runtime.yaml`: migrate/sync jobs plus API, worker, and event-router
 - `bootstrap-secrets.sh`: idempotently creates generated Kubernetes secrets

@ -28,15 +30,20 @@ provisions the Inter-Hub ops-hub key.

 `ISSUE_SINK_TYPE` defaults to **`state-hub`** (ACTIVITY-WP-0022; no silent Forgejo
 issues). Set `rest` only for intentional issue-core projection when the backend
-is healthy. `ISSUE_CORE_API_KEY`
-is synced from OpenBao into `actcore-runtime-secret` by ExternalSecret
-`actcore-issue-core-runtime` (same path as issue-core:
-`platform/workloads/issue-core/issue-core/issue-core-runtime`). Prereqs:
-`ClusterSecretStore/openbao-activity-core` and ESO token bootstrap
-(`OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token ./scripts/openbao-eso-token-apply.sh`).
-Roll back to audit mode by setting
-`ISSUE_SINK_TYPE=null` and restarting worker and event-router deployments.
-See `docs/issue-core-emission-boundary.md`.
+is healthy. `ISSUE_CORE_API_KEY` and `FORGEJO_TOKEN` are synced from OpenBao into
+`actcore-runtime-secret` by ExternalSecrets:
+
+| ExternalSecret | OpenBao path | Secret key |
+| --- | --- | --- |
+| `actcore-issue-core-runtime` | `platform/workloads/issue-core/issue-core/issue-core-runtime` | `ISSUE_CORE_API_KEY` |
+| `actcore-forgejo-admin` | `platform/workloads/forgejo/forgejo-admin` (`API_TOKEN`) | `FORGEJO_TOKEN` |
+
+Prereqs: `ClusterSecretStore/openbao-activity-core` and ESO token bootstrap with
+both read policies (`OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token
+./scripts/openbao-eso-token-apply.sh` — attaches
+`workload-kv-read-issue-core-runtime` + `workload-kv-read-forgejo-admin`).
+Roll back to audit mode by setting `ISSUE_SINK_TYPE=null` and restarting worker
+and event-router deployments. See `docs/issue-core-emission-boundary.md`.

 The same runtime projection now includes the active
 `daily-statehub-wsjf-triage.md` ActivityDefinition plus its JSON output schema
@ -71,11 +78,13 @@ ssh railiance01
 cd ~/activity-core
 bash k8s/railiance/bootstrap-secrets.sh
 kubectl apply -f k8s/railiance/10-infrastructure.yaml
-# Bootstrap OpenBao ESO token + apply ExternalSecret (once per cluster):
+# Bootstrap OpenBao ESO token + apply ExternalSecrets (once per cluster):
 OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token ./scripts/openbao-eso-token-apply.sh
 kubectl apply -f ~/railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml
 kubectl apply -f k8s/railiance/15-externalsecret-issue-core.yaml
+kubectl apply -f k8s/railiance/15-externalsecret-forgejo-admin.yaml
 kubectl -n activity-core wait --for=condition=Ready externalsecret/actcore-issue-core-runtime --timeout=120s
+kubectl -n activity-core wait --for=condition=Ready externalsecret/actcore-forgejo-admin --timeout=120s
 kubectl -n activity-core wait --for=condition=ready pod -l app.kubernetes.io/name=actcore-app-db --timeout=180s
 kubectl -n activity-core wait --for=condition=ready pod -l app.kubernetes.io/name=actcore-temporal-db --timeout=180s
 kubectl -n activity-core wait --for=condition=ready pod -l app.kubernetes.io/name=actcore-nats --timeout=180s
--- a/scripts/openbao-eso-token-apply.sh
+++ b/scripts/openbao-eso-token-apply.sh
@ -1,11 +1,18 @@
 #!/usr/bin/env bash
 # Mint a read-limited OpenBao token on coulombcore (bao.coulomb.social) and store
-# it on railiance01 for ClusterSecretStore openbao-activity-core / ExternalSecret
-# actcore-issue-core-runtime.
+# it on railiance01 for ClusterSecretStore openbao-activity-core / ExternalSecrets:
+#   - actcore-issue-core-runtime  (ISSUE_CORE_API_KEY)
+#   - actcore-forgejo-admin       (FORGEJO_TOKEN for package prune)
+#
+# Default policies (already present in OpenBao; files re-written when present):
+#   workload-kv-read-issue-core-runtime
+#   workload-kv-read-forgejo-admin
 set -euo pipefail

-POLICY_NAME="${OPENBAO_ACTIVITY_CORE_POLICY:-workload-kv-read-issue-core-runtime}"
-POLICY_FILE="${OPENBAO_ACTIVITY_CORE_POLICY_FILE:-$HOME/railiance-platform/openbao/policies/workload-kv-read-issue-core-runtime.hcl}"
+# Space-separated policy names. Override with OPENBAO_ACTIVITY_CORE_POLICIES.
+DEFAULT_POLICIES="workload-kv-read-issue-core-runtime workload-kv-read-forgejo-admin"
+POLICIES="${OPENBAO_ACTIVITY_CORE_POLICIES:-$DEFAULT_POLICIES}"
+POLICY_DIR="${OPENBAO_POLICY_DIR:-$HOME/railiance-platform/openbao/policies}"
 BAO_ADDR="${BAO_ADDR:-https://bao.coulomb.social}"
 RAILIANCE01_KUBECONFIG="${RAILIANCE01_KUBECONFIG:-$HOME/.kube/config-hosteurope}"
 SECRET_NAME="${OPENBAO_ACTIVITY_CORE_ESO_SECRET:-openbao-activity-core-eso-token}"
@ -16,13 +23,8 @@ if ! command -v bao >/dev/null 2>&1; then
  exit 1
 fi

-if [[ ! -f "$POLICY_FILE" ]]; then
-  echo "ERROR: policy file missing: $POLICY_FILE" >&2
-  exit 1
-fi
-
 echo "OpenBao addr: $BAO_ADDR"
-echo "Policy: $POLICY_NAME"
+echo "Policies: $POLICIES"

 if [[ -n "${BAO_TOKEN:-}" ]]; then
  :
@ -46,8 +48,21 @@ if echo "$health" | grep -q '"sealed":true'; then
  exit 1
 fi

-bao policy write "$POLICY_NAME" "$POLICY_FILE"
-token_json="$(bao token create -policy="$POLICY_NAME" -display-name="eso-activity-core-issue-sink" -period=720h -format=json)"
+policy_flags=()
+for policy in $POLICIES; do
+  policy_file="$POLICY_DIR/${policy}.hcl"
+  if [[ -f "$policy_file" ]]; then
+    bao policy write "$policy" "$policy_file"
+  else
+    echo "WARN: policy file missing ($policy_file); using existing OpenBao policy '$policy'" >&2
+  fi
+  policy_flags+=(-policy="$policy")
+done
+
+token_json="$(bao token create "${policy_flags[@]}" \
+  -display-name="eso-activity-core-runtime" \
+  -period=720h \
+  -format=json)"
 token="$(python3 -c "import json,sys; print(json.load(sys.stdin)['auth']['client_token'])" <<<"$token_json")"

 KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl create namespace "$SECRET_NS" --dry-run=client -o yaml | KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl apply -f -
@ -57,4 +72,5 @@ KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl create secret generic "$SECRET_NAME
  --dry-run=client -o yaml | KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl apply -f -

 unset BAO_TOKEN token token_json
-echo "ok: applied $SECRET_NS/$SECRET_NAME on railiance01 (token not printed)"
+echo "ok: applied $SECRET_NS/$SECRET_NAME on railiance01 (token not printed)"
+echo "ok: policies attached: $POLICIES"
--- a/workplans/ACTIVITY-WP-0023-intent-scope-gap-closure.md
+++ b/workplans/ACTIVITY-WP-0023-intent-scope-gap-closure.md
@ -4,7 +4,7 @@ type: workplan
 title: "Intent–scope gap closure and WP-0020/0021 operational follow-ups"
 domain: infotech
 repo: activity-core
-status: active
+status: finished
 owner: codex
 topic_slug: activity-core
 created: "2026-07-21"
@ -144,7 +144,7 @@ runbook links it from the prune section.

 ```task
 id: ACTIVITY-WP-0023-T05
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "bf21fa27-a872-44a6-a87b-1fe69018d090"
 ```
@ -158,11 +158,19 @@ state_hub_task_id: "bf21fa27-a872-44a6-a87b-1fe69018d090"
 **Done when:** ExternalSecret Ready=True and worker restarts still see a
 working token without manual patch.

+**Done 2026-07-21:**
+- `scripts/openbao-eso-token-apply.sh` attaches
+  `workload-kv-read-issue-core-runtime` + `workload-kv-read-forgejo-admin`.
+- ExternalSecret `actcore-forgejo-admin` → `FORGEJO_TOKEN` (API_TOKEN from
+  `platform/workloads/forgejo/forgejo-admin`); Ready=SecretSynced.
+- Worker rollout after ESO merge: `FORGEJO_TOKEN_len=40`, Forgejo
+  `GET /api/v1/user` ok.
+
 ## Task: issue-core path A (optional rest sink)

 ```task
 id: ACTIVITY-WP-0023-T06
-status: todo
+status: done
 priority: low
 state_hub_task_id: "7232d5a3-c7a8-4fdf-aaef-5daab0ea92b1"
 ```
@ -178,11 +186,17 @@ Non-blocking follow-up from ACTIVITY-WP-0021.
 **Done when:** documented smoke 201 for one opt-in definition, or explicit
 wontfix with issue-core owning the backend fix only.

+**Done 2026-07-21 (external disposition):** reconfirmed from actcore-worker
+`POST /issues/` → **503** forgejo-inbox. Global default remains **`state-hub`**
+(WP-0022). Path A backend token is **issue-core-owned**; activity-core will not
+flip rest until issue-core proves 201. Documented in
+`docs/issue-core-emission-boundary.md`.
+
 ## Task: ROS MarkiTect completeness

 ```task
 id: ACTIVITY-WP-0023-T07
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "eeb4beb9-8184-4ae3-95cd-d27dc88e7973"
 ```
@ -197,6 +211,13 @@ state_hub_task_id: "eeb4beb9-8184-4ae3-95cd-d27dc88e7973"
 **Done when:** one clean ROS hourly context snapshot without missing-template
 or MarkiTect-missing errors for configured domains.

+**Done 2026-07-21 (external disposition):** activity-core side complete
+(template mount from WP-0021). Live state-hub image
+`forgejo.coulomb.social/coulomb/state-hub:main-d8808bf` has **no**
+`markitect_tool` module. Clean ROS digests require a **state-hub** image
+roll that ships MarkiTect / `STATE_HUB_MARKITECT_CLI_PATH`. Owner: state-hub;
+message filed. SCOPE G6 notes residual external.
+
 ## Task: Retire or quarantine TaskExecutorWorkflow

 ```task
@ -248,12 +269,12 @@ guessing hub topology.

 ## Success criteria

- [ ] G1 closed via WP-0022 + T01 doc alignment
- [ ] G2 has a proven Binky executor path or explicit external workplan
- [ ] G3 apply-without-protection is impossible
- [ ] G6–G8 operational follow-ups shipped or scheduled with owners
- [ ] G5/G9/G10 dispositioned
- [ ] `SCOPE.md` re-checked after close; no stale “rest default / WP-0006 gate”
+- [x] G1 closed via WP-0022 + T01 doc alignment
+- [x] G2 has a proven Binky executor path or explicit external workplan
+- [x] G3 apply-without-protection is impossible
+- [x] G6–G8 operational follow-ups shipped or scheduled with owners
+- [x] G5/G9/G10 dispositioned
+- [x] `SCOPE.md` re-checked after close; no stale “rest default / WP-0006 gate”
  language

 ## References
@ -272,9 +293,12 @@ guessing hub topology.
 | T02 | done | Consumer contract doc; Binky completion path documented (harness external) |
 | T03 | done | apply=true hard-fails without non-empty live_images_file |
 | T04 | done | `scripts/refresh_live_images.sh` + make target |
-| T05 | todo | OpenBao ESO policy still 403 on forgejo-admin |
-| T06 | todo | issue-core GITEA_BACKEND_TOKEN rotation (external) |
-| T07 | todo | MarkiTect in state-hub image (external) |
+| T05 | done | ESO `actcore-forgejo-admin` SecretSynced; worker restart keeps FORGEJO_TOKEN |
+| T06 | done | Disposition: issue-core owns GITEA_BACKEND_TOKEN; rest stays opt-in; 503 reconfirmed |
+| T07 | done | Disposition: state-hub image needs MarkiTect; templates already mounted |
 | T08 | done | TaskExecutor stub disabled by default |
 | T09 | done | review_required documented as metadata-only |
 | T10 | done | Edge vs workstation evidence in runbook |
+
+**Closeout 2026-07-21:** all tasks done; workplan `finished`. Residual external:
+issue-core path A token, state-hub MarkiTect image.