NK-WP-0055-T01: production readiness ledger; gaps become T10-T15

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 4515@bnt-lap001 Assistant-Session: 0bf96127-6e04-41be-bfda-d11717c3f354
2026-10-27 22:11:57 +01:00 · 2026-10-27 22:11:57 +01:00 · c8f44fa614
commit c8f44fa614
parent 15574a5518
3 changed files with 287 additions and 5 deletions
--- a/WORK-RECORDS.md
+++ b/WORK-RECORDS.md
@ -52,7 +52,7 @@
 | workplan | NK-WP-0052 | finished | — | workplans/NK-WP-0052-wave2-resource-right-sizing.md |
 | workplan | NK-WP-0053 | active | — | workplans/NK-WP-0053-identity-restore-drill-cadence.md |
 | workplan | NK-WP-0054 | active | — | workplans/NK-WP-0054-service-identity-policy.md |
-| workplan | NK-WP-0055 | proposed | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
+| workplan | NK-WP-0055 | active | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
 | task | NK-WP-ADHOC-2026-07-02-T01 | done | — | workplans/ADHOC-2026-07-02.md |
 | task | NK-WP-ADHOC-2026-07-02-T02 | done | — | workplans/ADHOC-2026-07-02.md |
 | task | NK-WP-ADHOC-2026-08-14-T01 | done | — | workplans/ADHOC-2026-08-14.md |
@ -318,7 +318,7 @@
 | task | NK-WP-0054-T06 | wait | — | workplans/NK-WP-0054-service-identity-policy.md |
 | task | NK-WP-0054-T07 | todo | — | workplans/NK-WP-0054-service-identity-policy.md |
 | task | NK-WP-0054-T08 | done | — | workplans/NK-WP-0054-service-identity-policy.md |
-| task | NK-WP-0055-T01 | todo | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
+| task | NK-WP-0055-T01 | done | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
 | task | NK-WP-0055-T02 | wait | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
 | task | NK-WP-0055-T03 | wait | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
 | task | NK-WP-0055-T04 | wait | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
@ -327,6 +327,12 @@
 | task | NK-WP-0055-T07 | todo | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
 | task | NK-WP-0055-T08 | todo | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
 | task | NK-WP-0055-T09 | wait | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
+| task | NK-WP-0055-T10 | todo | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
+| task | NK-WP-0055-T11 | todo | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
+| task | NK-WP-0055-T12 | todo | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
+| task | NK-WP-0055-T13 | todo | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
+| task | NK-WP-0055-T14 | todo | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
+| task | NK-WP-0055-T15 | todo | — | workplans/NK-WP-0055-security-platform-production-readiness.md |
 | intake | NK-IN-0001 | closed | blue | docs/intakes/activity-core-ops-sso-operators.md |
 | intake | NK-IN-0002 | closed | blue | docs/intakes/activity-core-ops-sso-operators.md |
 | intake | NET-IN-0001 | open | — | intakes/intakes.md |
--- a/docs/evidence/2026-10-27-production-readiness-ledger.md
+++ b/docs/evidence/2026-10-27-production-readiness-ledger.md
@ -0,0 +1,187 @@
+# Production readiness ledger — 2026-10-27
+
+**Task:** NK-WP-0055-T01. This is a read-only assessment.
+**Sources:**
+- the net-kingdom repository and State Hub;
+- read-only owner assessments from railiance-platform, access-engine and audit-core, each
+  made from its own repository files and git history the same day;
+- founder-run admin checks recorded in NK-WP-0049.
+
+A check is **met** only if it was observed on the estate that serves production traffic.
+Builder-mode acceptances count as **partial**. The definition is in
+[NK-WP-0055](../../workplans/NK-WP-0055-security-platform-production-readiness.md).
+
+## Summary
+
+| # | Area | Status | Gap carried by |
+|---|---|---|---|
+| 1 | Platform root custody | partial | T10 (decisions), NK-WP-0054-T07 |
+| 2 | MFA and identity | partial | RPF-WP-0065-T04 (SI-X4), NK-WP-0054-T04/T05/T06, T12 |
+| 3 | Bootstrap and recovery | partial | T02–T04, T07 (NK-WP-0053), T08 |
+| 4 | OpenBao runtime secrets | partial | T10, T07 (RPF-WP-0036-T03), T12 |
+| 5 | Secret rotation | partial | T06, T15, RPF-WP-0035, RPF-WP-0055-T01 |
+| 6 | flex-auth (access-engine) policy state | partial | T14, FLEX-WP-0031, FLEX-WP-0024 |
+| 7 | Topaz runtime | open | T10 (is Topaz required?), then T14 |
+| 8 | Tenant onboarding | partial | T11, T12 |
+| 9 | Audit sink | open | T11, AUDIT-WP-0013-T02, AUDIT-WP-0015-T01 |
+| 10 | Break-glass | open | T13 |
+
+None of the ten areas is met. Three are open.
+
+## Per area
+
+### 1. Platform root custody: partial
+
+**Evidence.**
+- The custody model and records are in `docs/platform-root-custody.md` and ADR-0017.
+- The non-secret root-grant record is NK-WP-0045, and railiance-platform keeps its own
+  custody record in RPF-WP-0049.
+- The crisis custody list is `docs/crisis-custody-list.md`.
+
+**Missing.**
+- The two-of-three escrow holders are not named. The custody doc's stage S6 is not reached,
+  so custody is still single-operator, which the architecture allows only before production.
+- The OpenBao unseal share holders are `TO CONFIRM` (crisis list item 3).
+- The offline king path that works with SSO down is `TO CONFIRM` (item 4).
+
+### 2. MFA and identity: partial
+
+**Evidence.**
+- KeyCape issues tokens on the IAM Profile v0.3 contract.
+- Privileged-action MFA guards are delivered for the portal and Vergabe clients (NK-WP-0042).
+- The retained-MFA restore is proven (NK-WP-0049-T19).
+
+**Missing.**
+- The `aal2` binding on OpenBao `platform-admin` is not applied. SI-X4 is still open in
+  `tools/service-identity/declarations/openbao.yaml` and RPF-WP-0065-T04 is waiting.
+- No recorded `tools/iam-profile-conformance/` run against the live issuer (NK-WP-0054-T04
+  is waiting).
+- The bind account is still the human `admin` (NK-WP-0054-T05/T06).
+- Validation runs still lean on `platform-root`.
+
+### 3. Bootstrap and recovery: partial
+
+**Evidence.** NK-WP-0049 is finished:
+- secret-free bootstrap output;
+- daily off-host encrypted backup;
+- retrieval that works with OpenBao and SSO down;
+- isolated owner restores of all four identity services and of OpenBao.
+
+**Missing.**
+- The joint live consumer drill (T02–T04).
+- A recurring drill: NK-WP-0053-T02 waits on its first run, and T04 for an `admin`-operated
+  drill.
+- Undrilled crisis items: `pi-admin` (item 7), the ops bundle location (item 8) and operator
+  access with SSO down (item 9).
+- Receipt custody (T08).
+
+### 4. OpenBao runtime secrets: partial
+
+**Evidence.**
+- Per-workload scoped Kubernetes and JWT auth and policies (`railiance-platform/openbao/policies/`).
+- A file audit device, checked by `make openbao-verify-post-unseal`.
+- Operator-only access (RPF-WP-0025).
+- The isolated original restore (RPF-WP-0064-T07/T08).
+
+**Missing.**
+- Manual Shamir unseal is in use, but it is not recorded as the *approved* production
+  mechanism. Auto-unseal was never verified.
+- No scheduled snapshot job. RPF-WP-0064-T08 says that one restore is not recurring
+  assurance (RPF-WP-0036-T03).
+- No dated live check of the auth methods and mounts against the committed policies.
+
+### 5. Secret rotation: partial
+
+**Evidence.**
+- Dynamic database credentials exist for core-hub.
+- The rotation lanes are designed in RPF-WP-0035 (blocked; T12 in progress).
+- The backup credential default was removed (RPF-WP-0029).
+
+**Missing.**
+- Rotation of the exposed material, held by founder decision `1591daa1` and CUST-WP-0073-T05.
+- KeyCape client-secret rotation (RPF-WP-0055-T01, backlog).
+- No per-lane register of blast radius and verification steps.
+
+### 6. flex-auth (access-engine) policy state: partial
+
+**Evidence.**
+- Policy-as-code with check, list-allowed and explain.
+- Versioned packages: a fail-open fix ships as a new version (FLEX-DEC-2026-008).
+- Tenant-scope coverage (FLEX-WP-0022).
+- Nine declared release units.
+
+**Missing.**
+- Policies are not imported through a pipeline: a change needs a new image plus a helm
+  upgrade.
+- No platform/tenant package split for `tenant:coulomb`.
+- No production explain receipt.
+- Decision-record emission is declared but not delivered (FLEX-WP-0031).
+- The decision envelope is unsigned (FLEX-WP-0024-T07 waits on a founder gate).
+
+### 7. Topaz runtime: open
+
+**Evidence.** Topaz exists only as an adapter shape (`internal/adapters/topaz/`, FLEX-WP-0004).
+Nothing deploys it, and every live decision comes from the Go core.
+
+**Correction to the table.** Fail-closed is declared per consumer (`pep-stance.yaml`), not in
+the PDP.
+
+**Decision needed (T10).** Is a delegated Topaz PDP required for production, or does the Go
+core satisfy this area?
+
+### 8. Tenant onboarding: partial
+
+**Evidence.**
+- The onboarding portal, self-registration and first-login provisioning (NK-WP-0023, 0025,
+  0036).
+- Tenant-scoped identity lifecycle (NK-WP-0038).
+- Tenant-scoped authorization (FLEX-WP-0022).
+
+**Missing.**
+- The architecture's completion test has not been run end to end for `tenant:coulomb`: a
+  workload authenticates, gets a scoped decision, obtains only its secret, enforces locally
+  and produces an auditable record.
+- Its audit and correlation leg depends on area 9.
+
+### 9. Audit sink: open
+
+**Evidence.**
+- audit-core runs on durable PostgreSQL (AUDIT-WP-0005), with a hash chain (AUDIT-WP-0007).
+- A single restore walk on 2026-08-13.
+- Registered senders: user-engine, tenant-engine, approval-engine and informed-decision.
+  The Qonto collector is the only workload source.
+
+**Missing.**
+
+| Source | Status | Owner |
+|---|---|---|
+| Identity providers (KeyCape, Authelia, LLDAP, privacyIDEA) | not senders | — |
+| access-engine | not a sender | AUDIT-WP-0013-T02 waits on access-engine |
+| Topaz | not a sender | — |
+| OpenBao | audit only on a PVC; the docs say "not a durable audit sink" | — |
+| Kubernetes API audit | absent | — |
+
+- No cross-stream correlation id.
+- No recurring restore drill.
+- No offsite chain head (AUDIT-WP-0015-T01).
+
+**Side finding.** The NK-WP-0031 prerequisite (owner and E2 freshness metadata) has been met
+since audit-core commit ca9708e, so NK-WP-0031 can be unblocked.
+
+### 10. Break-glass: open
+
+**Evidence.** The crisis custody list exists, and the king OTP was validated against a
+restored privacyIDEA (isolated, 2026-10-24).
+
+**Missing.**
+- No recorded or drilled path that works when normal identity (SSO) is down: crisis list
+  items 4 and 9 are `TO CONFIRM`.
+- No post-event review record format.
+
+## Stale statements found
+
+These are for their owners to fix. Nothing was changed here.
+- access-engine `SCOPE.md:125` says secrets-engine uses `callerAuth.mode: warn`, but
+  `values/secrets-engine.yaml` says `enforce`.
+- The readiness table row "MFA and identity" names IAM Profile v0.2. The accepted contract is
+  v0.3.
--- a/workplans/NK-WP-0055-security-platform-production-readiness.md
+++ b/workplans/NK-WP-0055-security-platform-production-readiness.md
@ -4,7 +4,7 @@ type: workplan
 title: "Reach and prove security platform production readiness"
 domain: infotech
 repo: net-kingdom
-status: proposed
+status: active
 owner: claude-code
 topic_slug: netkingdom
 created: "2026-10-27"
@ -59,7 +59,7 @@ node.

 ```task
 id: NK-WP-0055-T01
-status: todo
+status: done
 priority: medium
 assignee: claude-code
 state_hub_task_id: "8c91597e-4d3e-5811-ae0d-96ff6b7d9017"
@ -76,6 +76,8 @@ state_hub_task_id: "8c91597e-4d3e-5811-ae0d-96ff6b7d9017"
 Gaps that the tasks below don't cover get new tasks in this plan, or a pointer to an
 existing owner workplan. The assessment is read-only.

+
+2026-10-27: done. [Readiness ledger](../docs/evidence/2026-10-27-production-readiness-ledger.md): 0 areas met, 7 partial, 3 open (Topaz, audit sink, break-glass). Uncovered gaps became T10–T15 below.
 ## Build and rehearse the joint-restore drill (from NK-WP-0049-T43/T44)

 ```task
@ -208,6 +210,8 @@ the current backup generation. It must include:

 Drill results land in the readiness ledger.

+
+2026-10-27: the identity drill already lives in NK-WP-0053 (quarterly isolated identity restore drill; T02 first run, T04 `admin`-operated). This task tracks NK-WP-0053 and RPF-WP-0036-T03 (scheduled OpenBao snapshot) rather than building a second drill; it adds only image lineage, the audit-device mount and the post-archive reconciliation procedure.
 ## Decide durable custody for receipts kept outside Git

 ```task
@ -225,6 +229,91 @@ their public digests.
 **Done when:** an accepted custody location with a second copy holds them, still bound by
 their recorded hashes, or the founder accepts single-copy custody as an exception.

+## Founder production-scope decisions
+
+```task
+id: NK-WP-0055-T10
+status: todo
+priority: high
+assignee: claude-code
+needs_human: true
+```
+
+Decide what production requires:
+(a) whether manual Shamir unseal is the approved production mechanism, or auto-unseal is required;
+(b) whether a delegated Topaz PDP is required, or the access-engine Go core satisfies the PDP area;
+(c) who the two-of-three escrow holders and the OpenBao unseal share holders are (names and places, never values);
+(d) that HA stays out of scope.
+
+**Done when:** each answer is recorded as a hub decision and reflected in the ledger.
+
+## Get every audit source into the durable sink with one correlation id
+
+```task
+id: NK-WP-0055-T11
+status: todo
+priority: high
+assignee: claude-code
+```
+
+Owners are audit-core (receiving), railiance-platform (OpenBao audit forwarding), railiance-cluster (Kubernetes API audit),
+access-engine (AUDIT-WP-0013-T02 registration) and NetKingdom (identity-provider events and the correlation-id contract).
+Topaz joins only if T10 keeps it.
+
+**Done when:** each owner has a task in its own workplan; identity, authorization, OpenBao, Kubernetes and workload records land
+durably; one correlation id is defined and tested across the onboarding chain; audit-core has a recurring restore drill and
+AUDIT-WP-0015-T01's offsite chain head.
+
+## Record live conformance receipts
+
+```task
+id: NK-WP-0055-T12
+status: todo
+priority: medium
+assignee: claude-code
+```
+
+**Done when:** dated receipts exist for the live IAM issuer conformance run (NK-WP-0054-T04), caller-auth enforcement on every
+access-engine release unit (only ops-warden is proven today), the live OpenBao auth methods and mounts against the committed
+policies, and the tenant onboarding completion test for `tenant:coulomb`.
+
+## Record and drill break-glass with SSO down
+
+```task
+id: NK-WP-0055-T13
+status: todo
+priority: high
+assignee: claude-code
+```
+
+**Done when:** the crisis custody list items 4, 7, 8 and 9 are confirmed (kept current by NK-WP-0054-T07); a path that works with
+KeyCape/Authelia down is recorded and drilled by the founder; and use produces a post-event review record.
+
+## Policy import pipeline and platform/tenant package split
+
+```task
+id: NK-WP-0055-T14
+status: todo
+priority: medium
+assignee: claude-code
+```
+
+Owner: access-engine. **Done when:** policy changes are imported without rebuilding the image, `tenant:coulomb` has platform and
+tenant packages split, one production explain receipt exists, and FLEX-WP-0031 (decision records) and FLEX-WP-0024 (signed
+envelope) are delivered or accepted as exceptions.
+
+## Register rotation lanes with blast radius and verification
+
+```task
+id: NK-WP-0055-T15
+status: todo
+priority: medium
+assignee: claude-code
+```
+
+**Done when:** every service, database, OpenBao-issued, client-secret and break-glass credential lane is listed with its blast
+radius and verification step, linked to its owner lane (RPF-WP-0035, RPF-WP-0055-T01, T06).
+
 ## Close readiness and declare production

 ```task
@ -232,7 +321,7 @@ id: NK-WP-0055-T09
 status: wait
 priority: high
 assignee: claude-code
-depends_on: [NK-WP-0055-T01, NK-WP-0055-T03, NK-WP-0055-T04, NK-WP-0055-T05, NK-WP-0055-T06, NK-WP-0055-T07, NK-WP-0055-T08]
+depends_on: [NK-WP-0055-T01, NK-WP-0055-T03, NK-WP-0055-T04, NK-WP-0055-T05, NK-WP-0055-T06, NK-WP-0055-T07, NK-WP-0055-T08, NK-WP-0055-T10, NK-WP-0055-T11, NK-WP-0055-T12, NK-WP-0055-T13, NK-WP-0055-T14, NK-WP-0055-T15]
 blocking_reason: "Every readiness area must be met or an accepted exception."
 state_hub_task_id: "0630b0ea-dec1-5355-8b08-0d317d4a3c9b"
 ```