From 21d46927f6c59bfaf55357498e8ca0a77b19fd62 Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 21 Jul 2026 23:19:10 +0200 Subject: [PATCH] Finish ACTIVITY-WP-0023: ESO FORGEJO_TOKEN and gap dispositions Ship ExternalSecret actcore-forgejo-admin and multi-policy ESO token bootstrap so weekly prune credentials refresh without bootstrap inject. Disposition issue-core path A and MarkiTect ROS as external owners; close workplan finished. --- SCOPE.md | 8 +-- docs/issue-core-emission-boundary.md | 21 ++++++-- docs/runbook.md | 2 +- .../15-externalsecret-forgejo-admin.yaml | 41 +++++++++++++++ k8s/railiance/README.md | 29 +++++++---- scripts/openbao-eso-token-apply.sh | 42 +++++++++++----- ...TIVITY-WP-0023-intent-scope-gap-closure.md | 50 ++++++++++++++----- 7 files changed, 147 insertions(+), 46 deletions(-) create mode 100644 k8s/railiance/15-externalsecret-forgejo-admin.yaml diff --git a/SCOPE.md b/SCOPE.md index 56a4887..a27b134 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -203,10 +203,10 @@ Open product/policy workplan: **ACTIVITY-WP-0022** (IssueSink no-default-Forgejo | **G2. Executor gap** | High (value) | Tasks are emitted (progress refs or issue-core), but **no per-repo executor** reliably consumes them (Binky briefs still need harness/agent pickup). INTENT assigns execution to per-repo Temporal workers — that substrate is thin. | | **G3. Side-effect resolvers** | Medium | Forgejo prune `apply: true` runs platform deletes inside a context resolver. Allowed only as declared, gated side-effect; needs clearer SCOPE language (above) and operator refresh of live-images protection. | | **G4. Review queue** | Medium | `review_required` remains metadata-only; no downstream review queue. | -| **G5. TaskExecutorWorkflow stub** | Low–medium | Still registered; risk of attracting real execution. | -| **G6. ROS digest completeness** | Medium (ops) | Template file mounted; MarkiTect CLI missing in state-hub image → digests still fail per domain. | -| **G7. Credential delivery** | Medium (ops) | `FORGEJO_TOKEN` bootstrap-injected; ESO 403 on forgejo-admin path. issue-core `GITEA_BACKEND_TOKEN` still broken for path A rest sink. | -| **G8. Live-images hygiene** | Medium (ops) | Multi-cluster `live-images-all.txt` must be refreshed after deploys or prune can delete live tags (incident 2026-07-21, restored). | +| **G5. TaskExecutorWorkflow stub** | Low | Disabled unless `ACTIVITY_CORE_ENABLE_TASK_EXECUTOR_STUB` (WP-0023-T08). | +| **G6. ROS digest completeness** | Medium (ops, external) | Templates mounted; **state-hub image** still lacks `markitect_tool` / MarkiTect CLI (`main-d8808bf` checked 2026-07-21). Owner: state-hub (WP-0023-T07). | +| **G7. Credential delivery** | Low (residual) | **FORGEJO_TOKEN** via ESO `actcore-forgejo-admin` (WP-0023-T05, Ready). issue-core `GITEA_BACKEND_TOKEN` still 503 forgejo-inbox for path A rest — **issue-core owner** (WP-0023-T06). | +| **G8. Live-images hygiene** | Medium (ops) | Multi-cluster `live-images-all.txt` must be refreshed after deploys or prune can delete live tags (incident 2026-07-21, restored). `scripts/refresh_live_images.sh` (T04). | | **G9. Evidence federation** | Low | Progress often lands on railiance01 edge/hub; workstation primary hub may not show the same feed without tunnel/outbox health. | | **G10. API external access** | Low | ClusterIP-only; intentional until auth policy. | diff --git a/docs/issue-core-emission-boundary.md b/docs/issue-core-emission-boundary.md index f804848..2dc806e 100644 --- a/docs/issue-core-emission-boundary.md +++ b/docs/issue-core-emission-boundary.md @@ -64,17 +64,28 @@ activity-core. Railiance production uses `state-hub` unless an overlay explicitly sets `rest` for an experiment. -### Known production failure (2026-07-21) — rest path +### Known production failure (2026-07-21, reconfirmed 2026-07-21 T06) — rest path -`POST /issues/` returned **HTTP 503**: +`POST /issues/` returns **HTTP 503**: ```text Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API ``` -`GITEA_BACKEND_TOKEN` on issue-core rejected by Forgejo. Fix is issue-core -token rotation (`warden route show issue-core-ingestion-api-key`), not -activity-core defaults. +**Disposition (ACTIVITY-WP-0023-T06):** activity-core keeps global default +`state-hub` and does **not** flip production to `rest`. Path A is owned by +**issue-core**: rotate/fix `GITEA_BACKEND_TOKEN` (Forgejo backend PAT for the +forgejo-inbox connector — not the activity-core `ISSUE_CORE_API_KEY` +ingestion key). After issue-core proves `POST /issues/` → **201**, operators may +opt in per definition / overlay only (WP-0022). + +Smoke from worker (does not change sink env): + +```bash +kubectl -n activity-core exec deploy/actcore-worker -- python -c ' +# valid IssueCore body → expect 201 when path A is healthy, else 503 forgejo-inbox +' +``` ## Promotion to rest (one definition at a time) diff --git a/docs/runbook.md b/docs/runbook.md index 1f86904..b6ce806 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -364,7 +364,7 @@ mount of `~/railiance-platform` on the worker) with `apply: true` and posts | Retention | newest **3** versions per package (`container`, `pypi`, `npm`, `generic`) | | Org | `coulomb` | | Protected | live cluster image tags + Helm values (`--live-images-file` / live scan) | -| Credential | `FORGEJO_TOKEN` in `actcore-runtime-secret` (OpenBao path `platform/workloads/forgejo/forgejo-admin` field `API_TOKEN`; `warden route show forgejo-admin-api-token`) | +| Credential | `FORGEJO_TOKEN` in `actcore-runtime-secret` via ExternalSecret `actcore-forgejo-admin` (OpenBao `platform/workloads/forgejo/forgejo-admin` field `API_TOKEN`; `warden route show forgejo-admin-api-token`). ESO token: `scripts/openbao-eso-token-apply.sh` (includes `workload-kv-read-forgejo-admin`). | | Rollback | restore package versions from Nextcloud `forgejo dump` if a needed tag was removed | **Enabled 2026-07-21** after dry-run + first apply evidence diff --git a/k8s/railiance/15-externalsecret-forgejo-admin.yaml b/k8s/railiance/15-externalsecret-forgejo-admin.yaml new file mode 100644 index 0000000..310de41 --- /dev/null +++ b/k8s/railiance/15-externalsecret-forgejo-admin.yaml @@ -0,0 +1,41 @@ +# Sync FORGEJO_TOKEN from OpenBao into actcore-runtime-secret. +# +# Prereqs on railiance01: +# - External Secrets Operator (namespace external-secrets) +# - ClusterSecretStore openbao-activity-core (railiance-platform addon) +# - Secret external-secrets/openbao-activity-core-eso-token with policies: +# workload-kv-read-issue-core-runtime +# workload-kv-read-forgejo-admin +# (scripts/openbao-eso-token-apply.sh) +# +# OpenBao path: platform/workloads/forgejo/forgejo-admin +# property: API_TOKEN → secret key FORGEJO_TOKEN +# Catalog: warden route show forgejo-admin-api-token +# +# Merge keeps bootstrap-owned keys (ACTCORE_DB_URL, webhook secrets) and the +# ISSUE_CORE_API_KEY ExternalSecret merge intact. Retires one-shot +# `kubectl patch secret` bootstrap for the weekly package prune (WP-0023-T05). +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: actcore-forgejo-admin + namespace: activity-core + labels: + app.kubernetes.io/name: activity-core + app.kubernetes.io/part-of: activity-core + annotations: + argocd.argoproj.io/sync-wave: "0" +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: openbao-activity-core + target: + name: actcore-runtime-secret + creationPolicy: Merge + deletionPolicy: Retain + data: + - secretKey: FORGEJO_TOKEN + remoteRef: + key: platform/workloads/forgejo/forgejo-admin + property: API_TOKEN diff --git a/k8s/railiance/README.md b/k8s/railiance/README.md index d0a079f..5ef1dda 100644 --- a/k8s/railiance/README.md +++ b/k8s/railiance/README.md @@ -12,6 +12,8 @@ name and access policy. NATS JetStream, Temporal, and Temporal UI - `15-externalsecret-issue-core.yaml`: OpenBao → `ISSUE_CORE_API_KEY` merge into `actcore-runtime-secret` via External Secrets +- `15-externalsecret-forgejo-admin.yaml`: OpenBao → `FORGEJO_TOKEN` merge for + weekly package prune (ACTIVITY-WP-0023-T05) - `20-runtime.yaml`: migrate/sync jobs plus API, worker, and event-router - `bootstrap-secrets.sh`: idempotently creates generated Kubernetes secrets @@ -28,15 +30,20 @@ provisions the Inter-Hub ops-hub key. `ISSUE_SINK_TYPE` defaults to **`state-hub`** (ACTIVITY-WP-0022; no silent Forgejo issues). Set `rest` only for intentional issue-core projection when the backend -is healthy. `ISSUE_CORE_API_KEY` -is synced from OpenBao into `actcore-runtime-secret` by ExternalSecret -`actcore-issue-core-runtime` (same path as issue-core: -`platform/workloads/issue-core/issue-core/issue-core-runtime`). Prereqs: -`ClusterSecretStore/openbao-activity-core` and ESO token bootstrap -(`OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token ./scripts/openbao-eso-token-apply.sh`). -Roll back to audit mode by setting -`ISSUE_SINK_TYPE=null` and restarting worker and event-router deployments. -See `docs/issue-core-emission-boundary.md`. +is healthy. `ISSUE_CORE_API_KEY` and `FORGEJO_TOKEN` are synced from OpenBao into +`actcore-runtime-secret` by ExternalSecrets: + +| ExternalSecret | OpenBao path | Secret key | +| --- | --- | --- | +| `actcore-issue-core-runtime` | `platform/workloads/issue-core/issue-core/issue-core-runtime` | `ISSUE_CORE_API_KEY` | +| `actcore-forgejo-admin` | `platform/workloads/forgejo/forgejo-admin` (`API_TOKEN`) | `FORGEJO_TOKEN` | + +Prereqs: `ClusterSecretStore/openbao-activity-core` and ESO token bootstrap with +both read policies (`OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token +./scripts/openbao-eso-token-apply.sh` — attaches +`workload-kv-read-issue-core-runtime` + `workload-kv-read-forgejo-admin`). +Roll back to audit mode by setting `ISSUE_SINK_TYPE=null` and restarting worker +and event-router deployments. See `docs/issue-core-emission-boundary.md`. The same runtime projection now includes the active `daily-statehub-wsjf-triage.md` ActivityDefinition plus its JSON output schema @@ -71,11 +78,13 @@ ssh railiance01 cd ~/activity-core bash k8s/railiance/bootstrap-secrets.sh kubectl apply -f k8s/railiance/10-infrastructure.yaml -# Bootstrap OpenBao ESO token + apply ExternalSecret (once per cluster): +# Bootstrap OpenBao ESO token + apply ExternalSecrets (once per cluster): OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token ./scripts/openbao-eso-token-apply.sh kubectl apply -f ~/railiance-platform/argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml kubectl apply -f k8s/railiance/15-externalsecret-issue-core.yaml +kubectl apply -f k8s/railiance/15-externalsecret-forgejo-admin.yaml kubectl -n activity-core wait --for=condition=Ready externalsecret/actcore-issue-core-runtime --timeout=120s +kubectl -n activity-core wait --for=condition=Ready externalsecret/actcore-forgejo-admin --timeout=120s kubectl -n activity-core wait --for=condition=ready pod -l app.kubernetes.io/name=actcore-app-db --timeout=180s kubectl -n activity-core wait --for=condition=ready pod -l app.kubernetes.io/name=actcore-temporal-db --timeout=180s kubectl -n activity-core wait --for=condition=ready pod -l app.kubernetes.io/name=actcore-nats --timeout=180s diff --git a/scripts/openbao-eso-token-apply.sh b/scripts/openbao-eso-token-apply.sh index f9609fd..5fbf6b5 100755 --- a/scripts/openbao-eso-token-apply.sh +++ b/scripts/openbao-eso-token-apply.sh @@ -1,11 +1,18 @@ #!/usr/bin/env bash # Mint a read-limited OpenBao token on coulombcore (bao.coulomb.social) and store -# it on railiance01 for ClusterSecretStore openbao-activity-core / ExternalSecret -# actcore-issue-core-runtime. +# it on railiance01 for ClusterSecretStore openbao-activity-core / ExternalSecrets: +# - actcore-issue-core-runtime (ISSUE_CORE_API_KEY) +# - actcore-forgejo-admin (FORGEJO_TOKEN for package prune) +# +# Default policies (already present in OpenBao; files re-written when present): +# workload-kv-read-issue-core-runtime +# workload-kv-read-forgejo-admin set -euo pipefail -POLICY_NAME="${OPENBAO_ACTIVITY_CORE_POLICY:-workload-kv-read-issue-core-runtime}" -POLICY_FILE="${OPENBAO_ACTIVITY_CORE_POLICY_FILE:-$HOME/railiance-platform/openbao/policies/workload-kv-read-issue-core-runtime.hcl}" +# Space-separated policy names. Override with OPENBAO_ACTIVITY_CORE_POLICIES. +DEFAULT_POLICIES="workload-kv-read-issue-core-runtime workload-kv-read-forgejo-admin" +POLICIES="${OPENBAO_ACTIVITY_CORE_POLICIES:-$DEFAULT_POLICIES}" +POLICY_DIR="${OPENBAO_POLICY_DIR:-$HOME/railiance-platform/openbao/policies}" BAO_ADDR="${BAO_ADDR:-https://bao.coulomb.social}" RAILIANCE01_KUBECONFIG="${RAILIANCE01_KUBECONFIG:-$HOME/.kube/config-hosteurope}" SECRET_NAME="${OPENBAO_ACTIVITY_CORE_ESO_SECRET:-openbao-activity-core-eso-token}" @@ -16,13 +23,8 @@ if ! command -v bao >/dev/null 2>&1; then exit 1 fi -if [[ ! -f "$POLICY_FILE" ]]; then - echo "ERROR: policy file missing: $POLICY_FILE" >&2 - exit 1 -fi - echo "OpenBao addr: $BAO_ADDR" -echo "Policy: $POLICY_NAME" +echo "Policies: $POLICIES" if [[ -n "${BAO_TOKEN:-}" ]]; then : @@ -46,8 +48,21 @@ if echo "$health" | grep -q '"sealed":true'; then exit 1 fi -bao policy write "$POLICY_NAME" "$POLICY_FILE" -token_json="$(bao token create -policy="$POLICY_NAME" -display-name="eso-activity-core-issue-sink" -period=720h -format=json)" +policy_flags=() +for policy in $POLICIES; do + policy_file="$POLICY_DIR/${policy}.hcl" + if [[ -f "$policy_file" ]]; then + bao policy write "$policy" "$policy_file" + else + echo "WARN: policy file missing ($policy_file); using existing OpenBao policy '$policy'" >&2 + fi + policy_flags+=(-policy="$policy") +done + +token_json="$(bao token create "${policy_flags[@]}" \ + -display-name="eso-activity-core-runtime" \ + -period=720h \ + -format=json)" token="$(python3 -c "import json,sys; print(json.load(sys.stdin)['auth']['client_token'])" <<<"$token_json")" KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl create namespace "$SECRET_NS" --dry-run=client -o yaml | KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl apply -f - @@ -57,4 +72,5 @@ KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl create secret generic "$SECRET_NAME --dry-run=client -o yaml | KUBECONFIG="$RAILIANCE01_KUBECONFIG" kubectl apply -f - unset BAO_TOKEN token token_json -echo "ok: applied $SECRET_NS/$SECRET_NAME on railiance01 (token not printed)" \ No newline at end of file +echo "ok: applied $SECRET_NS/$SECRET_NAME on railiance01 (token not printed)" +echo "ok: policies attached: $POLICIES" diff --git a/workplans/ACTIVITY-WP-0023-intent-scope-gap-closure.md b/workplans/ACTIVITY-WP-0023-intent-scope-gap-closure.md index bdb0c29..1c8ff04 100644 --- a/workplans/ACTIVITY-WP-0023-intent-scope-gap-closure.md +++ b/workplans/ACTIVITY-WP-0023-intent-scope-gap-closure.md @@ -4,7 +4,7 @@ type: workplan title: "Intent–scope gap closure and WP-0020/0021 operational follow-ups" domain: infotech repo: activity-core -status: active +status: finished owner: codex topic_slug: activity-core created: "2026-07-21" @@ -144,7 +144,7 @@ runbook links it from the prune section. ```task id: ACTIVITY-WP-0023-T05 -status: todo +status: done priority: medium state_hub_task_id: "bf21fa27-a872-44a6-a87b-1fe69018d090" ``` @@ -158,11 +158,19 @@ state_hub_task_id: "bf21fa27-a872-44a6-a87b-1fe69018d090" **Done when:** ExternalSecret Ready=True and worker restarts still see a working token without manual patch. +**Done 2026-07-21:** +- `scripts/openbao-eso-token-apply.sh` attaches + `workload-kv-read-issue-core-runtime` + `workload-kv-read-forgejo-admin`. +- ExternalSecret `actcore-forgejo-admin` → `FORGEJO_TOKEN` (API_TOKEN from + `platform/workloads/forgejo/forgejo-admin`); Ready=SecretSynced. +- Worker rollout after ESO merge: `FORGEJO_TOKEN_len=40`, Forgejo + `GET /api/v1/user` ok. + ## Task: issue-core path A (optional rest sink) ```task id: ACTIVITY-WP-0023-T06 -status: todo +status: done priority: low state_hub_task_id: "7232d5a3-c7a8-4fdf-aaef-5daab0ea92b1" ``` @@ -178,11 +186,17 @@ Non-blocking follow-up from ACTIVITY-WP-0021. **Done when:** documented smoke 201 for one opt-in definition, or explicit wontfix with issue-core owning the backend fix only. +**Done 2026-07-21 (external disposition):** reconfirmed from actcore-worker +`POST /issues/` → **503** forgejo-inbox. Global default remains **`state-hub`** +(WP-0022). Path A backend token is **issue-core-owned**; activity-core will not +flip rest until issue-core proves 201. Documented in +`docs/issue-core-emission-boundary.md`. + ## Task: ROS MarkiTect completeness ```task id: ACTIVITY-WP-0023-T07 -status: todo +status: done priority: medium state_hub_task_id: "eeb4beb9-8184-4ae3-95cd-d27dc88e7973" ``` @@ -197,6 +211,13 @@ state_hub_task_id: "eeb4beb9-8184-4ae3-95cd-d27dc88e7973" **Done when:** one clean ROS hourly context snapshot without missing-template or MarkiTect-missing errors for configured domains. +**Done 2026-07-21 (external disposition):** activity-core side complete +(template mount from WP-0021). Live state-hub image +`forgejo.coulomb.social/coulomb/state-hub:main-d8808bf` has **no** +`markitect_tool` module. Clean ROS digests require a **state-hub** image +roll that ships MarkiTect / `STATE_HUB_MARKITECT_CLI_PATH`. Owner: state-hub; +message filed. SCOPE G6 notes residual external. + ## Task: Retire or quarantine TaskExecutorWorkflow ```task @@ -248,12 +269,12 @@ guessing hub topology. ## Success criteria -- [ ] G1 closed via WP-0022 + T01 doc alignment -- [ ] G2 has a proven Binky executor path or explicit external workplan -- [ ] G3 apply-without-protection is impossible -- [ ] G6–G8 operational follow-ups shipped or scheduled with owners -- [ ] G5/G9/G10 dispositioned -- [ ] `SCOPE.md` re-checked after close; no stale “rest default / WP-0006 gate” +- [x] G1 closed via WP-0022 + T01 doc alignment +- [x] G2 has a proven Binky executor path or explicit external workplan +- [x] G3 apply-without-protection is impossible +- [x] G6–G8 operational follow-ups shipped or scheduled with owners +- [x] G5/G9/G10 dispositioned +- [x] `SCOPE.md` re-checked after close; no stale “rest default / WP-0006 gate” language ## References @@ -272,9 +293,12 @@ guessing hub topology. | T02 | done | Consumer contract doc; Binky completion path documented (harness external) | | T03 | done | apply=true hard-fails without non-empty live_images_file | | T04 | done | `scripts/refresh_live_images.sh` + make target | -| T05 | todo | OpenBao ESO policy still 403 on forgejo-admin | -| T06 | todo | issue-core GITEA_BACKEND_TOKEN rotation (external) | -| T07 | todo | MarkiTect in state-hub image (external) | +| T05 | done | ESO `actcore-forgejo-admin` SecretSynced; worker restart keeps FORGEJO_TOKEN | +| T06 | done | Disposition: issue-core owns GITEA_BACKEND_TOKEN; rest stays opt-in; 503 reconfirmed | +| T07 | done | Disposition: state-hub image needs MarkiTect; templates already mounted | | T08 | done | TaskExecutor stub disabled by default | | T09 | done | review_required documented as metadata-only | | T10 | done | Edge vs workstation evidence in runbook | + +**Closeout 2026-07-21:** all tasks done; workplan `finished`. Residual external: +issue-core path A token, state-hub MarkiTect image.