ops-mason/plans/rein-openweights-openrouter-approle.md
tegwick 9bcb5819de Fix KV v2 policy path shape; live end-to-end verification succeeded
Two more real bugs found completing this lane for real:

1. platform-admin's own policy had no entry for the new reins/ mount --
   the founder's paste-once-provision write 403'd because the admin
   identity that created the mount was never granted access to operate
   on it. Fixed live (added path "reins/*" matching every other mount
   already in that policy).

2. _policy_hcl wrote the bare KV-v1-shaped path
   (reins/rein-openweights/openrouter) instead of KV v2's data/+metadata/
   sub-paths -- bao token capabilities on the bare path even reported
   full access, but the actual kv get still 403'd, because OpenBao
   evaluates the real request against the data/-prefixed path. Caught
   when the AppRole's own read failed during live verification. Fixed
   in code (now emits both data/ and metadata/ paths), locked in with a
   dedicated unit test, and re-applied to the live policy.

Live end-to-end verification succeeded after both fixes: real AppRole
login, real KV v2 read via the corrected policy, real OpenRouter call,
real commit -- with OPENROUTER_API_KEY unset the whole time. Plan status:
catalogued. glas-harness/GLAS-WP-0002-T02 is closed by this.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-27 01:50:18 +02:00

14 KiB

id demand_source consumer_repo credential_type status approved_by approved_at created updated
rein-openweights-openrouter-approle glas-harness/workplans/GLAS-WP-0002-T02 rein-openweights openbao-approle-kv catalogued Bernd Worsch 2026-07-27 2026-07-27 2026-07-27

Construction plan: rein-openweights OpenBao AppRole

1. Demand

rein-openweights (the OpenRouter-driven rein in the glas-harness family) needs to authenticate to OpenBao non-interactively — no operator OIDC session, no human in the loop at run time — to fetch its OpenRouter API key when it runs unattended (e.g. via a future scheduled trigger, or a glas-harness gateway invocation with no operator present). This was scoped as glas-harness/GLAS-WP-0002-T02 ("Option B" — see that workplan's discussion) and is currently blocked: the only credential path exercised so far is the OPENROUTER_API_KEY env-var short-circuit in rein_openweights/credentials.py; the AppRole/vault branch (_acquire_token, bao kv get) is real code, live-verifiable, but has never actually talked to OpenBao.

2. Existing-structure survey

Checked: ops-warden/registry/routing/catalog.yaml (readable directly — no OpenBao session needed for this). Not checked: live bao policy list / bao auth list / bao secrets list — this session has no valid OpenBao token (bao token lookup returns 403). Whoever executes phase 4 must re-verify against live state before applying anything below; this survey is current as of what's on disk, not as of live OpenBao ACL state.

Relevant existing catalog entry: openrouter-llm-connect (ops-warden/registry/routing/catalog.yaml:283) — an OpenRouter key already exists at platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets (field OPENROUTER_API_KEY), gated by policy workload-kv-read-llm-connect-provider-secrets, auth_method: caller's own OpenBao token (operator OIDC via key-cape, or a token carrying [that policy]), marked risk: high ("provider API key with spend impact"). rein-aharness's own mail-triage/brief-daily already reuse this exact lane (see binky-control/integrations/executor-worker-secrets.md, Lane 1 — "REUSE (verified)").

Does this existing lane satisfy the demand? Partially, by design, not fully — reuse considered and declined:

  • The existing lane's auth_method is interactive-caller-shaped (OIDC / "a token carrying the policy") — it was never scoped for a bare AppRole login the way agent-harness-binky-mail's Lane 3 was. Binding a new AppRole to the existing policy is mechanically possible (Option A from the original discussion) but would give rein-openweights standing, non-interactive access to a policy explicitly flagged risk: high for spend impact, shared with activity-core's broader usage — a wider blast radius than a single rein needs.
  • Recommendation: do not reuse. Build a dedicated, narrowly-scoped AppRole + KV path for rein-openweights specifically (matches credentials.py's existing default reins/rein-openweights/openrouter path already in the code) — same reasoning agent-harness-binky-mail used for its own dedicated lane rather than widening an existing one. This keeps a leak scoped to exactly one consumer, one credential.

3. Proposed changes

# Action Object Reuse-vs-new rationale
1 create KV v2 secret path reins/rein-openweights/openrouter (field api_key) New, narrowly scoped — matches credentials.py's existing default path; declined reuse of the shared activity-core path (see §2)
2 create Policy workload-kv-read-rein-openweights-openrouter — read-only, scoped to exactly path #1 New — mirrors the shape of workload-kv-read-llm-connect-provider-secrets but scoped to one path, not shared
3 create AppRole rein-openweights, bound to policy #2 New — mirrors agent-harness-binky-mail's AppRole shape: token_ttl=15m, token_max_ttl=30m, bounded token_num_uses, secret_id_ttl per current posture (build-phase; see ops-warden/wiki/WorkloadSecurityPosture.md)
4 deliver role_id/secret_id files, 0600, to wherever the live verification runs (workstation or a future Railiance deployment) New — matches agent-harness-binky-mail's delivery pattern (~/.local/agent-harness/approle-binky-mail), analogous path for this consumer
5 propose ops-warden catalog entry rein-openweights-openrouter-approle — pointer-only, warden_executes: false, status: draft New — no existing entry covers this AppRole; openrouter-llm-connect entry stays as-is, untouched

No tear-down in this plan — no existing lane is being retired or compacted; the existing openrouter-llm-connect lane is explicitly kept as-is for its current interactive-caller consumers.

4. Review notes (phase 2)

  • Naming convention check: follows <consumer>-<credential-purpose> shape used by agent-harness-binky-mail — proposed rein-openweights-openrouter-approle for the AppRole/catalog id, workload-kv-read-rein-openweights-openrouter for the policy (matches the workload-kv-read-<path-slug> convention already visible in the existing catalog).
  • TTL/scoping check: token_ttl=15m/token_max_ttl=30m/bounded token_num_uses matches agent-harness-binky-mail exactly — no reason to diverge, this consumer's access pattern (one-shot credential fetch per run) is identical in shape.
  • Redundancy check: confirmed no second existing AppRole or policy already scoped to reins/rein-openweights/* — this is genuinely new structure, not a duplicate.
  • Compaction opportunity: none identified — nothing existing becomes redundant once this lands; the shared activity-core lane keeps its own consumers.
  • Ease of use for the consumer: matches directly — credentials.py already reads REIN_OPENWEIGHTS_APPROLE_DIR (defaults to a role_id/secret_id file pair) and a KV path override via REIN_OPENWEIGHTS_OPENROUTER_KV_PATH (defaulting to exactly reins/rein-openweights/openrouter, field api_key) — this plan's path #1 and delivery #4 need zero code changes in rein-openweights to consume; the code was already written expecting this exact shape.
  • Posture check: build phase (one founder-operator, pre-revenue) per ops-warden/wiki/WorkloadSecurityPosture.mdagent-harness-binky-mail used secret_id_ttl=0 (no expiry) at this same posture; carrying that forward here rather than adding a rotation schedule neither agent-harness-binky-mail nor current posture requires. Explicit assumption, not a silent default — flagged again in the executive summary below for the approval decision to confirm or override.

5. Executive summary (phase 3)

One-line ask: create a narrowly-scoped, non-interactive credential lane so rein-openweights can fetch its own OpenRouter API key at run time without an operator present.

Who/what gets access: a new OpenBao AppRole named rein-openweights. Nothing else — no existing role, human, or service gains anything new.

To what: read-only access to exactly one KV path, reins/rein-openweights/openrouter (field api_key) — a path that does not exist today and holds nothing else. The AppRole cannot read the existing shared activity-core/llm-connect secret or any other path.

For how long: each login is a 15-minute token, renewable up to a 30-minute cap, bounded number of uses per token — matches the existing agent-harness-binky-mail lane exactly. The AppRole credential itself (role_id/secret_id) does not expire on a schedule at current (build-phase) posture, same choice already made for agent-harness-binky-mail.

Blast radius if the credential leaks: whoever holds a valid role_id/secret_id pair can mint a short-lived token that reads one OpenRouter API key — nothing else in OpenBao. Actual damage from the OpenRouter key itself is real but bounded (API spend on that key, no data-store or infra access) and independent of this plan (same exposure openrouter-llm-connect's existing consumers already carry).

Cost to reverse: delete one AppRole, one policy, one KV path — three bao commands, no other lane or consumer affected. Fully isolated blast/reversal radius by construction (that was the point of declining reuse in §2).

Decision needed: approve as proposed, reject, or send back to phase 1 with feedback (e.g. different TTL, different path name, or "actually reuse the shared lane instead").

Decision: approved as proposed (Bernd Worsch, 2026-07-27). Ready for phase 4 — still blocked on a real OpenBao session existing in whatever environment executes the build (bao token lookup from this workstation returns 403; unchanged since GLAS-WP-0002-T02 first flagged it).

6. Build result (phase 4)

Built 2026-07-27, executed via ops_mason.executor.build_approle_kv_lane against real OpenBao (bao login -method=oidc -path=netkingdom role=platform-admin, platform-admin policy, ~1h token). Object names only below — no secret material.

  • Mount created: reins/ (KV v2) — did not exist; bao secrets list showed only platform/, secret/, tenants/ at execution time. Per §2's own caveat ("survey current as of what's on disk, not live OpenBao ACL state"), this was a real, live discrepancy from what the approved plan assumed. Confirmed with the founder before mounting (a new top-level secrets engine is a bigger action than "one KV path" implied in the executive summary's blast-radius framing) — approved to proceed exactly as originally planned rather than substitute an existing mount.
  • Policy created: workload-kv-read-rein-openweights-openrouter — read-only, scoped to exactly reins/rein-openweights/openrouter.
  • AppRole created: rein-openweightstoken_ttl=15m, token_max_ttl=30m, secret_id_ttl=0.
  • Correction made post-build: token_num_uses was built as 0 (OpenBao's own default) instead of the plan's own stated 8 (mirroring agent-harness-binky-mail) — 0 means unlimited in OpenBao, the opposite of "bounded." Caught by checking the live AppRole config against the reference lane immediately after building, fixed with bao write auth/approle/role/rein-openweights token_num_uses=8 (verified). Root cause fixed in code, not just patched live: AppRoleKVSpec.token_num_uses no longer has a default in ops_mason/executor.py — it must be passed explicitly every time, so this specific mistake can't recur silently on a future plan.
  • role_id/secret_id delivered to ~/.local/rein-openweights/approle/ (mode 0600), matching credentials.py's existing REIN_OPENWEIGHTS_APPROLE_DIR default.
  • Catalog entry proposed and merged: ops-warden commit c0a50bc, rein-openweights-openrouter-approle, status: draft (promotes to active once the founder completes paste-once-provision and GLAS-WP-0002-T02's live verification succeeds). Playbook: ops-warden/wiki/playbooks/rein-openweights-openrouter-approle.md. ops-warden's full test suite (326 tests) verified green after the addition.

Remaining, not part of this build: the founder still needs to paste the real OpenRouter API key into reins/rein-openweights/openrouter (field api_key) through ops-warden's paste_once_provision desk — ops-mason built structure only, per INTENT.md.

7. Post-build corrections (found completing phases 4-onward for real)

Two more real bugs surfaced only by actually exercising the built lane, both fixed at the root cause, not just patched live:

  • platform-admin's own policy had no entry for the new reins/ mount. The founder's paste_once_provision desk write failed (403, "preflight capability check") because the admin identity that created the mount was never granted access to operate on it — every other KV mount (platform/, secret/, tenants/, ...) is explicitly listed in platform-admin's policy; reins/* wasn't, since the policy predates the mount. Fixed by adding a path "reins/*" {...} block matching the existing entries exactly. Process gap, not just a code bug: build_approle_kv_lane creates the consumer's policy/AppRole but never checks whether the founder's own admin policy can reach a brand-new mount — worth a follow-up task to check this automatically for any future plan that creates a new mount, not just this one.
  • The generated policy used the wrong path shape for a KV v2 mount. _policy_hcl wrote path "reins/rein-openweights/openrouter" — the bare, KV-v1-shaped path. KV v2 routes actual reads/writes through <mount>/data/<path> (with a separate <mount>/metadata/<path> for version operations); a policy against the bare path silently denies everything on a v2 mount. bao token capabilities on the bare path had even reported full access earlier in this build — misleading, since OpenBao evaluates the real request against the data/-prefixed path, not whatever a caller checks capabilities against. Caught when the AppRole's own kv get 403'd during the live GLAS-WP-0002-T02 verification. Fixed in ops_mason/executor.py::_policy_hcl (now emits both data/ and metadata/ sub-paths) and re-applied to the live policy; a dedicated unit test (test_policy_hcl_uses_kv_v2_data_and_metadata_paths) locks the correct shape in for every future plan.

Live end-to-end verification succeeded (2026-07-27) after both fixes: real AppRole login → real KV v2 read via the corrected policy → real OpenRouter API call → real 2-turn tool-calling loop → real commit (e30d520c79a3493aeea5e8ea4d5ec331512b781d), with OPENROUTER_API_KEY unset the whole time — the vault round trip, not the env-var short-circuit, is what ran. glas-harness/GLAS-WP-0002-T02 is closed by this.

Plan status: catalogued — ops-warden entry promoted draft -> active (see that repo's commit history).