secrets-engine/workplans/SECRETS-WP-0015-original-keycape-recovery-selector.md

294 lines
19 KiB
Markdown
Raw Permalink Normal View History

---
id: SECRETS-WP-0015
type: workplan
title: "Prepare original KeyCape service selector and isolated consumer continuity"
domain: infotech
repo: secrets-engine
status: active
owner: codex
topic_slug: custodian
created: "2026-10-17"
updated: "2026-10-23"
flavor: implementation
depends_on: []
related: [CUST-WP-0082, NK-WP-0049, SECRETS-WP-0012, SECRETS-WP-0008, SECRETS-WP-0013, SECRETS-WP-0014, GEXEC-WP-0001]
state_hub_workstream_id: "68a0642e-4908-5260-89c9-3adce60e6a4d"
---
Needed source-owner work for NK-WP-0049-T36/T18. Reviewed against current
Lifecycle INTENT, explicit consumer constants/custody source and dated historical
proof. Active records the delivered source plan and remaining native owner wait;
it is not native assent. Finished0012 remains finished; functional adoption/backend
entitlement stay0008T06/0013. No new grant,
secret fetch, provider login, production redirect or external message occurred.
2026-10-21 review against gated-executor (GEXEC-WP-0001, accepted on railiance01
2026-10-20): the T03 plan requires "surviving offline founder authorization" bound
to the exact source/issuer/case/deadline, with "no new auth or approval service".
The gexec approval pattern (founder `ssh-keygen -Y` signature, runtime-unwritable
`allowed_signers`) is that existing mechanism, and it verifies offline during an
outage. T05 makes it a checkable precondition of the selector preflight, and T04
now requires it. CUST-WP-0080 finished; its remaining native gates live in CUST-WP-0082.
## Establish exact current intended source selector
```task
id: SECRETS-WP-0015-T01
status: done
priority: high
assignee: codex
state_hub_task_id: "a3e0e243-edac-5ade-a3d5-f58dac41633a"
```
Completed 2026-10-17 as source review. Current consumer constants and config guards
agree with secrets-engine-openbao client/audience, service:secrets-engine,
tenant:platform, role secrets-engine and only openbao:login. Producer env selector
and historical denied openbao:admin case are independently referenced, without
accepting the producer's native role. Prior native issuance7Oct and consumer
probe10Oct predate reviewed archive16Oct; they neither prove exact archived bytes
nor current native assent. Recorded docs/keycape-isolated-recovery-selector.md.
Evidence: Custodian recovery-owner-handoffs-r1 and prior17-source review.
## Deliver bounded offline public source selector preflight
```task
id: SECRETS-WP-0015-T02
status: done
priority: high
assignee: codex
state_hub_task_id: "eab3b758-10ce-5464-b5f7-3a152ee1327a"
```
Completed 2026-10-17 as isolated source preparation. Prepared AST-only source-hash
and all-eight-selector-field checker with bounded strict JSON, source drift,
duplicate, oversize and symlink negatives. It imports no backend/credential client
and never exchanges a token. Tests use public/synthetic metadata only. Canonical
integration/publication/projection must be recorded separately; successful output
is source correspondence, not native acceptance or a new secret source.
## Prepare outage-compatible original consumer proof without custody bypass
```task
id: SECRETS-WP-0015-T03
status: done
priority: high
assignee: codex
state_hub_task_id: "6262a915-7655-50be-b9f7-481f816d7335"
```
Completed 2026-10-17 as source-plan preparation only. The reviewed owner plan
is [keycape-isolated-consumer-continuity-plan.md](../docs/keycape-isolated-consumer-continuity-plan.md),
bound to the frozen Custodian `consumer-outage-r1` packet. It distinguishes the
restored original issuer check, exact Secrets consumer claim compatibility and
actual original OpenBao consumer continuity, preserving the no-file/no-fallback
client-secret source contract. Offline founder authority, original TLS/private
admission, exact source/runtime/receipt interfaces, positive/negative/equality
oracles, timeout, abort and cleanup requirements are reviewable. The plan's done
criterion is met; no native or private custody dependency is represented as done.
The packet includes an unexecuted-on-real-input one-shot claim observer candidate;
it is not wired into recovery or ordinary service authentication. Five synthetic
tests pass, including seven explicitly noncryptographic claim-parser negatives
and source/input refusal checks. The existing exact-source selector preflight
passes. These source checks establish neither token signature/native provenance
nor client-secret acquisition, OpenBao login, backend entitlement or downstream
reconnect. No private archive, original token/key, OpenBao endpoint or native
approval was accessed. Source integration/publication/projection are separate
coordinator follow-through; the immutable packet retains source hashes and limits.
Remaining actual owner dependencies are explicit in the plan and T04: NK49T33
complete original TLS/archive/private admission; an independently recovered
original OpenBao and its custody/OIDC/MFA graph for full consumer continuity;
exact native action approval and resource-server negative/revocation receipts.
Protected backend operation entitlement stays in existing0008T06/0013.
## Require founder-signed exact attempt admission
```task
id: SECRETS-WP-0015-T05
status: done
priority: high
assignee: claude
depends_on: [SECRETS-WP-0015-T03]
state_hub_task_id: "74fedf9e-3f7d-56b6-a031-64012d9b011a"
```
Completed 2026-10-21 as source. `tools/check-keycape-recovery-selector.py --admission`
verifies a `secrets-engine.recovery-attempt-admission.v1` document. It binds the
exact selector-file SHA-256, the consumer source hash, the issuer, the case id and
a window of at most 24 hours. The founder signs it in namespace
`secrets-engine-recovery-attempt` against the shared runtime-unwritable anchor
(`src/secrets_engine/founder_admission.py`, `docs/founder-admission.md`; same
anchor as SECRETS-WP-0014). A pass reports
`founder_admitted_attempt_not_native_acceptance`. Synthetic tests cover every
binding mismatch, the expired/early/overlong window, edits after signing, the
trust/gexec namespaces, an empty signature, a changed selector and suppressed CLI
diagnostics. No private archive, token, endpoint or real founder signature was
touched. Single-use consumption and cleanup remain in the bounded NK49 execution.
## Verify the separately admitted actual isolated consumer result
```task
id: SECRETS-WP-0015-T04
status: wait
priority: high
assignee: codex
needs_human: true
decision_id: "d1ab097d-cc79-4612-9c17-bb9b364081ac"
depends_on: [SECRETS-WP-0015-T03, SECRETS-WP-0015-T05, NK-WP-0049-T33]
blocking_reason: "Original OpenBao restored in isolation 2026-10-24 (RPF-WP-0064-T07, receipt 4e048dee). Await RPF-WP-0064 phase B (internal-only network with the isolated NK49 KeyCape) and a separately founder-admitted consumer run: KeyCape exchange, OpenBao JWT login, openbao:admin refusal, session revocation."
state_hub_task_id: "aeec6eb0-39e2-5f7c-8f5d-778133edfc73"
```
2026-10-24 (RPF-WP-0064-T08 handoff): the original OpenBao was restored from the 2026-10-14 off-host snapshot in a
networkless disposable target, founder attended, and independently verified (railiance-platform `7084b57`,
receipt `sha256:4e048dee…579d`, kept outside Git). Cluster id `fd28df5d…` matched; this consumer's objects were
present and equal to source: `keycape-services` JWT config, role `secrets-engine-openbao`, policy
`secrets-engine-login-self`, and KV metadata `platform/workloads/keycape/secrets-engine-openbao` (no data read).
This fills the "original OpenBao/custody recovery" dependency at configuration level. Consumer continuity is still
open: a live JWT login needs RPF-WP-0064 phase B (internal network with the isolated NK49 KeyCape) and its own
founder admission.
Before any key prompt, the selector preflight must pass with `--admission` on the
founder-signed attempt document for this exact case (T05).
2026-10-21: decision d1ab097d resolved by the founder as approved in principle.
The remaining gate is the actual signed attempt approval once NK-WP-0049-T33 is ready.
T04 also owns the admission anchor on the attempt host. Install root-owned
`/etc/secrets-engine/admission/allowed_signers` there, with the founder approver
key, per `docs/founder-admission.md`. If the attempt host is the secrets-engine
runtime host, one anchor serves both this task and SECRETS-WP-0014-T04. NK-WP-0049
pinned secrets-engine 0f44fbe and the selector and service_auth.py hashes from that
commit, recorded in net-kingdom
`docs/evidence/2026-10-21-nk49-t33-readiness/readiness-observation.json`. Those pins
were confirmed 2026-10-21. T04 consumes the NK49 T33 packet unchanged.
Only a separately admitted owner action can establish actual archived registration,
issuer trust, native negative checks and consumer continuity. Source selector and
historical provider proof do not fill this gate. No production consumer redirect,
new grant, live OpenBao read, backend write or retained-factor MFA is authorized.
Return exact bounded evidence to NK49T36/T18; MFA remains NK49T19. Human gate is
never a worker wakeup. Cross-owner dependency is attached to derived case
secrets15-nk49-readiness in the packet; do not strand the wait or invent a driver.
2026-10-21 upstream path (read-only survey). NK-WP-0049-T33 waits on:
- T35 kc-tls owner metadata (railiance-cluster RCLUSTER-WP-0018-T02, todo).
- Founder decision 9f9d0d88 on T39 (ACME reissue vs. the original key).
- The attempt-host anchor (this task).
- NK49 attempt-assembly metadata: deadline, receipt location, verifier.
NK49-T40 carries the founder signature after T33. There is no pin drift: 0f44fbe..0aeb5a2
changed only this workplan, and the selector, service_auth.py and check tool hashes are
unchanged. Do not edit those files before the attempt, because any edit forces an NK49 re-pin.
2026-10-21: the founder resolved 9f9d0d88 as A. kc-tls is recovered by ACME
letsencrypt-prod reissue, chained to ISRG Root X1, with no self-signed substitute.
NK49 should rehearse on LE staging first. T33 no longer needs original kc-tls key
custody. net-kingdom and railiance-cluster were notified; railiance-cluster was asked
to re-scope RCLUSTER-WP-0018-T02 to the outage prerequisites. Remaining for T33:
the T35 metadata and the NK49 attempt assembly (deadline, receipt location, verifier).
2026-10-23: the founder installed the anchor on the attempt host, the workstation. `/etc/secrets-engine/admission/allowed_signers`
is root:root 0644 with a single entry: principal `bernd`, namespaces=`secrets-engine-recovery-attempt`, key
`SHA256:NekDABfw8K8IBYznjREg7jPpcK/BzCgO1pH1ebT/FXM`. A self-test signature verified Good. This is not
the runtime host anchor for SECRETS-WP-0014-T04 (different namespace and host). No selector, service_auth.py or
check-tool file was edited, so the NK49 pins stand.
2026-10-23 evening, consumer-owner reading of the NK49 attempt (read-only; no live action by this owner).
Inputs: NK-WP-0049-T40 done (net-kingdom a20dcfe), founder-signed admission sha256 `f8297954…c157`,
window 2026-10-23T17:00Z to 2026-10-24T05:00Z; selector preflight `--admission` exit 0,
`founder_admitted_attempt_not_native_acceptance`, principal bernd. NK-WP-0049-T18 attended attempt
(net-kingdom a09bf20, `docs/evidence/2026-10-23-nk49-t18-keycape-attempt/verifier-output.json`):
production LE reissue in an isolated namespace (railiance-cluster 88b360b), launcher passed, independent
verifier `verified` with no findings for receipt sha256 `16d72f7c…1651`. The receipt is retained outside
Git by founder decision and is not copied here. It records `network_mode` none, `production_contacted` false,
`downstream_consumer_reconnect_proven` false and `human_mfa_proven` false.
Filled, for this exact selector only (plan claim 1, original issuer). The pinned checker
`isolated-keycape.py` (sha256 `86f37add…`) selects exactly one archived client `secrets-engine-openbao`
with `secretRef` env `KEYCAPE_SECRETS_ENGINE_OPENBAO_CLIENT_SECRET`, confidential, client_credentials.
It then runs native `keycape verify-client` with all eight selector fields:
- archived registration: `native_service_registration_verified` and
`original_config_registry_policy_receipts_equal`. Exchange with the archived secret, exact sub, tenant,
roles and scope, and no excess grant.
- issuer trust: `original_https_issuer_and_trust_preserved` and `original_public_signing_key_in_live_jwks`
(SPKI sha256 `e803a3e2…3a3d`), on the reissued chain to ISRG Root X1.
- native negative checks at the issuer: `denied_scopes_refused` (`openbao:admin`) and
`wrong_unrelated_secret_explicitly_refused`.
- isolation and cleanup: plaintext and container cleanup verified.
Not filled, and not claimed. The verifier's `claims_not_made` (consumer continuity, retained-factor MFA,
owner acceptance) remain unclaimed here.
- Plan claim 2, consumer claim compatibility. The callback that hands an original-issuer token to the
Secrets claim observer is not implemented, and the observer has never run on real material.
- Plan claim 3, actual consumer continuity. The secrets-engine client has not authenticated natively
against restored dependencies: no OpenBao JWT login, no resource-server negative, no session
revocation. This needs an independently recovered original OpenBao, plus its custody, KV
registration, OIDC role and operator auth. NK49's identity archive does not carry it
(`identity-service-restore-packet.md` excludes OpenBao restoration).
- MFA stays NK49 T19. Owner acceptance is not given by this note.
2026-10-23 read-only survey. No owner holds a prepared, bounded, isolated OpenBao recovery for this
consumer. railiance-platform RPF-WP-0036-T03 inventories OpenBao snapshot/isolated restore and waits on
the activity-core scheduler. Its RPF-WP-0015 Raft snapshot receipt authorizes no restore. rapp-openbao
WP-0001/0002 contain no restore task.
Next owner action: railiance-platform, with rapp-openbao, accepts or declines a bounded isolated original
OpenBao restore with the secrets-engine JWT role and KV path, as a dependency of this task. The
consumer-continuity run (plan claim 3) stays a separately admitted attempt after that. Status stays `wait`.
No selector, service_auth.py, checker or founder_admission.py file was edited, so the NK49 pins stand.
2026-10-27 proposal for NK-WP-0049-T43 / phase B (message 765f6d7b from net-kingdom; proposal only, no live action,
no secret value read, nothing authorized; founder decision 2026-10-27: this workload consumer satisfies T18 scope,
no Authelia in phase B).
Why the ordinary path cannot be used. `login_service_jwt` takes the client secret from
`OpenBaoOidcClientSecretSource`, an attended OIDC operator login to OpenBao followed by an in-memory KV read. Phase B
has no Authelia/OIDC operator, and the plan forbids injecting the recovered issuer secret into ordinary
`login_service_jwt` or creating a persistent consumer copy. `client_secret_file` is the approval-profile path and
needs a 0600 file; phase B must not use it. So phase B needs a new, separately reviewed secret source.
Proposed custody: founder-attended pipe into an inherited descriptor, in-memory only.
1. New file, not an edit of any pinned one (`service_auth.py`, selector, checker and `founder_admission.py` stay
byte-identical, so the NK49 pins stand): `tools/phase-b-consumer-run.py` (name suggested) plus tests. It builds a
`KeyCapeServiceAuthProvider(kcfg, secret_source=<callable>)` itself; the callable reads fd N once.
2. The founder starts the runner in their own terminal. The secret is produced by the founder's pipeline from the
backup's SOPS material and reaches the runner only as a pipe: `<founder decrypt of the one field> | runner
--secret-fd 0`-style, or an explicitly opened fd 3 (`3< <(...)`) so stdin stays free for the founder's own
prompts. The value never appears in argv, environment, a file, a tmpfs path or shell history, and the runner
refuses `--secret` / `--secret-file` / `KEYCAPE_*_SECRET` env if present. The agent never holds the terminal,
the pipeline or the output; it only receives the value-free result.
3. Runner reads at most 4 KiB, until EOF, requires one non-empty token without whitespace (same rule as
`read_client_secret_file`), closes the fd, uses the value for exactly one client-credentials exchange, then
zeroes its buffer reference and does not retry or fall back. fd must be a pipe (S_ISFIFO), not a regular file or
tty, so a file copy or an echoed prompt is refused. Core dumps disabled (RLIMIT_CORE 0, PR_SET_DUMPABLE 0), no
swap-visible logging, no `repr` of the value.
4. Runner runs inside the admitted network (member of the internal network, no default route), only after
`check-keycape-recovery-selector.py --admission` passes for the founder-signed attempt for this exact case; the
run is single use inside the signed window.
5. Result is value-free JSON only: exchange accepted, subject/tenant/roles/scope claims match, `openbao:login`
accepted, `openbao:admin` refused, wrong-secret refusal (founder supplies a second unrelated token through the
same fd mechanism), session `revoke-self` attempted and succeeded, secret source name
`founder_attended_pipe_fd`, cleanup facts.
6. The decrypt step is the founder's: SOPS/age key custody is NetKingdom's and the founder's. Which exact backup field
and the decrypt command are for NetKingdom to name; secrets-engine will not name a path to the value and will not
run the decrypt. Not a bypass of the existing "no persistent consumer copy" rule: nothing persists.
7. Alternative considered and not preferred: the issuer's own `service-token` command mints a token in the issuer
tmpfs and hands only that token over private stdin to the runner (plan claim 2). That needs a new NK49 source
closure and review and hands a token, not the client secret, so it does not exercise the client's own secret and
would not prove the secret in the backup is the one the consumer uses. Offered as a fallback only.
Gaps and owner items: a new reviewed runner file (this task, after founder approval of the design and the
SECRETS-WP-0015 attempt admission for phase B); NK49 names the backup field and decrypt command; the founder signs a
new admission, since the 2026-10-23 window ended 2026-10-24T05:00Z. Status stays `wait`.
ISRG Root X1 trust of restored OpenBao for /jwks: UNPROVEN in this repository. What the evidence does show: the
KeyCape side served a production letsencrypt-prod chain to ISRG Root X1 (NK49 T18 verifier, receipt 16d72f7c) and
the restored OpenBao's `keycape-services` JWT config and role were present and equal to source (RPF-WP-0064-T07,
receipt 4e048dee), with no data read and no JWKS fetch, because both ran `--network none`. No rehearsal here ever
made OpenBao fetch `/jwks`; the only JWKS fetches in this repo are the consumer-side Python tests
(`tools/exercise_approval_identity.py`, synthetic CA via SSL_CERT_FILE), which say nothing about OpenBao's trust
store. Whether it trusts the chain depends on facts only the OpenBao owner can read: the OpenBao image's system CA
bundle, and whether the restored JWT config sets `jwks_ca_pem` / `oidc_discovery_ca_pem` (a pinned CA would
override or restrict system roots). Requested of railiance-platform in its phase B rehearsal: show those two
settings as present/absent (value-free), and a bounded probe from inside the restored OpenBao network namespace
that OpenBao-side validation of a JWT against `https://kc.coulomb.social/jwks` succeeds, using the production
chain and not LE staging (staging roots are not ISRG Root X1). The first live login attempt is the real proof; it
should be preceded by that probe so a trust failure is not misread as a consumer failure.