Document recovered ESO lanes and recognize explicit invalid-token responses

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883

docs/argocd-gitops.md

@@ -196,15 +196,18 @@ The GitOps contract uses:
 `ClusterSecretStore/openbao` is limited to the `issue-core` namespace on
 coulombcore.
 
-**Railiance01 interim stores** (read coulombcore OpenBao at
-`https://bao.coulomb.social` until Wave 7) are **not** managed by coulombcore
-ArgoCD. Apply per workload:
+**Railiance01 stores** use the private railiance01 OpenBao service and are
+**not** managed by coulombcore ArgoCD. Apply per workload:
 
 | Store | Bootstrap |
 | --- | --- |
-| `openbao-forgejo` | `railiance-apps` `make forgejo-openbao-eso-token-apply forgejo-openbao-store-deploy` |
+| `openbao-forgejo` | Dedicated Kubernetes auth; `docs/eso-auth-recovery.md` |
 | `openbao-activity-core` | `activity-core` `make openbao-eso-token-apply` + `kubectl apply -f …/openbao-activity-core.clustersecretstore.yaml` |
-| `openbao-reuse` | `railiance-apps` `make reuse-openbao-eso-token-apply reuse-openbao-store-deploy` |
+| `openbao-reuse` | Dedicated Kubernetes auth; `docs/eso-auth-recovery.md` |
+| `openbao-target-revenue` | Dedicated Kubernetes auth; `docs/eso-auth-recovery.md` |
+
+The three recovered stores no longer use the old `*-eso-token-apply` bootstrap
+targets. Do not recreate static delivery tokens for them.
 
 Manifests live under `argocd/platform-addons/openbao-secretstore/` but are
 excluded from the coulombcore kustomization. Broaden or add stores only with

docs/eso-auth-recovery.md

@@ -0,0 +1,48 @@
+# Three-lane ESO authentication recovery
+
+RPF-WP-0037 repairs Forgejo SMTP, reuse-surface and target-revenue delivery.
+All three consumers were active; their old static-token stores failed with 403.
+Before resync, attended comparison proved existing OpenBao values identical to
+deployed workload Secrets. This is an authentication migration, not an
+application credential rotation.
+
+| Store | Dedicated service account | OpenBao role |
+| --- | --- | --- |
+| openbao-forgejo | forgejo/forgejo-mailer-eso | forgejo-mailer-eso |
+| openbao-reuse | reuse/reuse-surface-eso | reuse-surface-runtime-eso |
+| openbao-target-revenue | target-revenue/target-revenue-eso | target-revenue-runtime-eso |
+
+Source: `openbao/eso-auth-recovery/lanes.json`, `serviceaccounts.yaml`, the three
+`*-eso.hcl` policies and existing ClusterSecretStore manifests under
+`argocd/platform-addons/openbao-secretstore/`. These stores remain outside the
+coulombcore kustomization. No broad RBAC is added; ESO's existing TokenRequest
+authority is used with exact namespace/SA/audience OpenBao bindings.
+
+Roles use audience `openbao`, 15-minute TTL/max/explicit max, no default policy,
+and exact data reads. ESO also needs token lookup-self and revoke-self. No KV
+metadata/list/write, token creation or sibling access is granted. Existing
+caller-facing policies and application credentials remain in place.
+
+Commit and push source before running the attended command:
+
+```sh
+warden access openbao-platform-admin-login --exec -- \
+  /usr/bin/python3 /home/worsch/railiance-platform/scripts/repair_eso_kubernetes_auth.py \
+  apply --kubeconfig /home/worsch/.kube/config-railiance01 \
+  --receipt /tmp/eso-auth-apply.json --confirm 'APPLY RPF-WP-0037'
+```
+
+The helper pins cluster identity, compares custody before mutation, refuses
+scope drift, validates bounded positive and negative authentication, and patches
+only each store's auth using a JSON Patch test of the observed spec. `verify`
+forces another refresh and requires its timestamp to advance. `cleanup` performs
+the same verification, proves the old token is unreferenced, revokes it or proves
+invalidity, and deletes only its obsolete Secret with UID/resourceVersion
+preconditions. All receipts are exclusively created mode 0600; values and tokens
+stay in captured memory. Choose a fresh receipt path for each invocation.
+
+On failure, inspect only the receipt and safe metadata. Completed store repairs
+can be verified independently; retry apply only against matching reviewed source.
+Do not roll back to rejected static credentials. Correct forward to the accepted
+Kubernetes role; application Secrets are retained and no workload restart is
+part of this procedure. Custody disagreement needs its own owner-led resolution.

docs/workload-kv-access-lanes.md

@@ -242,11 +242,16 @@ are OpenBao-custodied and delivered to Railiance01 via External Secrets.
 | Fields | `REUSE_SURFACE_TOKEN`, `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` |
 | Read policy | `workload-kv-read-reuse-surface-runtime` |
 | Policy file | `openbao/policies/workload-kv-read-reuse-surface-runtime.hcl` |
-| K8s auth role | `external-secrets-reuse-surface` (ESO delivery path) |
+| K8s auth role | `reuse-surface-runtime-eso`, SA `reuse/reuse-surface-eso`, audience `openbao` |
 | Primary consumer | ExternalSecret `reuse/reuse-surface-runtime` → Secret `reuse-surface-env` (Railiance01, 1h refresh) |
-| Interim ClusterSecretStore | `openbao-reuse` (token auth to `https://bao.coulomb.social`, namespace `reuse`) |
+| ClusterSecretStore | `openbao-reuse` (Kubernetes auth to private railiance01 OpenBao, namespace `reuse`) |
 | ops-warden catalog | `reuse-surface-hub-write-token` |
 
+RPF-WP-0037 replaced static ESO authentication without rotating workload values.
+ESO uses the separate `workload-kv-read-reuse-surface-runtime-eso` policy: exact
+data read and self-token lifecycle only. The caller-facing policy/catalog above
+remains separate. See `docs/eso-auth-recovery.md`.
+
 Fetch hub write token (do not log the value):
 
 ```bash

scripts/repair_eso_kubernetes_auth.py

@@ -30,6 +30,13 @@ def check_role(actual, lane):
     require(not actual.get('token_period') and not actual.get('bound_service_account_namespace_selector'), 'role_drift')
 
 
+def invalid_token_lookup(result):
+    error = result.stderr.lower()
+    return (result.returncode != 0 and (b'400' in error or b'403' in error)
+            and (b'bad token' in error or b'invalid token' in error)
+            and b'permission denied' not in error)
+
+
 def compare_custody(kube, lane):
     es = data(command(kube + ['-n', lane['namespace'], 'get', 'externalsecret', lane['name'], '-o', 'json']))
     require(es['spec']['secretStoreRef'] == {'kind': 'ClusterSecretStore', 'name': lane['store']}, 'external_secret_store_drift')
@@ -153,7 +160,7 @@ def run(args, receipt):
                 revoke(old_token)
                 row['retired_token_revoked'] = True
             else:
-                require(b'400' in lookup.stderr and (b'bad token' in lookup.stderr.lower() or b'invalid token' in lookup.stderr.lower()), 'retired_token_invalidity_unproven')
+                require(invalid_token_lookup(lookup), 'retired_token_invalidity_unproven')
                 row['retired_token_already_invalid'] = True
             check = bao(['token', 'lookup', '-format=json'], token=old_token, allow_failure=True)
             require(check.returncode != 0 and b'403' in check.stderr, 'retired_token_denial_unproven')

tests/test_eso_auth_recovery.py

@@ -39,6 +39,12 @@ class EsoRecoveryTests(unittest.TestCase):
             self.assertEqual(policy.count('path "'),3)
             self.assertIn('path "'+lane['kv_path']+'"',policy)
 
+    def test_expired_token_response_is_distinct_from_authority_failure(self):
+        for code in [400,403]:
+            self.assertTrue(repair.invalid_token_lookup(SimpleNamespace(returncode=2,stderr=f'Code: {code}. Errors: bad token'.encode())))
+        for error in [b'Code: 403. permission denied',b'Code: 500. bad token',b'connection refused']:
+            self.assertFalse(repair.invalid_token_lookup(SimpleNamespace(returncode=2,stderr=error)))
+
 
 if __name__=='__main__':
     unittest.main()

tests/test_state_hub_preflight_lane.py

@@ -96,6 +96,7 @@ class SigningLaneTests(unittest.TestCase):
         import tempfile, yaml
         ccr = copy.deepcopy(self.ccr)
         ccr['status'] = 'approved'
+        ccr['access_frontdoor']['resolvable'] = False
         ccr['openbao']['auth']['bound_claims_confirmed'] = True
         ccr['openbao']['auth']['audience'] = 'kubernetes'
         with tempfile.TemporaryDirectory() as tmp: