RCLUSTER-WP-0021: propose k3s API audit policy and shipping to audit-core (NK-WP-0055-T11)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 4515@bnt-lap001
Assistant-Session: 0bf96127-6e04-41be-bfda-d11717c3f354

WORK-RECORDS.md

@@ -24,6 +24,7 @@
 | workplan | RCLUSTER-WP-0018 | active | — | workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md |
 | workplan | RCLUSTER-WP-0019 | active | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md |
 | workplan | RCLUSTER-WP-0020 | active | — | workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md |
+| workplan | RCLUSTER-WP-0021 | proposed | — | workplans/RCLUSTER-WP-0021-k3s-api-audit-to-audit-core.md |
 | task | RCLUSTER-WP-ADHOC-2026-07-27-T01 | done | — | workplans/ADHOC-2026-07-27.md |
 | task | RAIL-BS-WP-0008-T01 | done | — | workplans/RAIL-BS-WP-0008-activity-core-wp0016-triage-output-deploy.md |
 | task | RAIL-BS-WP-0008-T02 | done | — | workplans/RAIL-BS-WP-0008-activity-core-wp0016-triage-output-deploy.md |
@@ -80,3 +81,8 @@
 | task | RCLUSTER-WP-0020-T02 | done | — | workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md |
 | task | RCLUSTER-WP-0020-T03 | wait | — | workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md |
 | task | RCLUSTER-WP-0020-T04 | todo | — | workplans/RCLUSTER-WP-0020-node-budget-knative-caps-and-wave2-allocation.md |
+| task | RCLUSTER-WP-0021-T01 | todo | — | workplans/RCLUSTER-WP-0021-k3s-api-audit-to-audit-core.md |
+| task | RCLUSTER-WP-0021-T02 | todo | — | workplans/RCLUSTER-WP-0021-k3s-api-audit-to-audit-core.md |
+| task | RCLUSTER-WP-0021-T03 | wait | — | workplans/RCLUSTER-WP-0021-k3s-api-audit-to-audit-core.md |
+| task | RCLUSTER-WP-0021-T04 | wait | — | workplans/RCLUSTER-WP-0021-k3s-api-audit-to-audit-core.md |
+| task | RCLUSTER-WP-0021-T05 | wait | — | workplans/RCLUSTER-WP-0021-k3s-api-audit-to-audit-core.md |

workplans/RCLUSTER-WP-0021-k3s-api-audit-to-audit-core.md

@@ -0,0 +1,140 @@
+---
+id: RCLUSTER-WP-0021
+type: workplan
+title: "k3s API-server audit policy, node log backend and durable shipping to audit-core"
+domain: financials
+repo: railiance-cluster
+status: proposed
+owner: claude
+topic_slug: railiance
+created: "2026-10-27"
+updated: "2026-10-27"
+flavor: implementation
+depends_on: []
+related: [NK-WP-0055, AUDIT-WP-0005, AUDIT-WP-0007, AUDIT-WP-0015, RCLUSTER-WP-0019]
+---
+
+Owner source work for NetKingdom NK-WP-0055-T11 (production-readiness ledger,
+area 9 "Audit sink", row "Kubernetes API audit: absent"). The single-node k3s
+server on railiance01 runs with no `audit-policy-file` and no audit log, so no
+Kubernetes API record exists for secrets access, RBAC changes or exec/attach.
+
+Boundary: this repo owns the k3s server flags, the audit policy, the node-local
+log backend and rotation, and the shipper that runs on the node. audit-core owns
+the receiving API, sender registration, schema, hash chain, retention in its
+store and its restore drill. Nothing here is auto-applied; every host change runs
+through an attended playbook. The k3s restart is a founder-run live step.
+
+## Audit policy (source)
+
+```task
+id: RCLUSTER-WP-0021-T01
+status: todo
+priority: high
+assignee: claude
+```
+
+Add `ansible/files/k3s-audit-policy.yaml` (`audit.k8s.io/v1`, rules first-match):
+
+1. `omitStages: [RequestReceived]` globally.
+2. `secrets`, `serviceaccounts/token`, `tokenreviews`, `subjectaccessreviews`:
+   **Metadata only, in every namespace**. The policy must contain no
+   Request/RequestResponse rule that can match `secrets`; a test enforces this.
+3. RBAC (`rbac.authorization.k8s.io`: roles, rolebindings, clusterroles,
+   clusterrolebindings) write verbs: RequestResponse (no secret payloads).
+4. `pods/exec`, `pods/attach`, `pods/portforward`, `nodes/proxy`: Metadata
+   (the request body carries no stream content).
+5. cert-manager (`cert-manager.io`, `acme.cert-manager.io`) writes: Request.
+6. Namespaces `sso`, `mfa`, `openbao` (exact list confirmed against the cluster
+   in T05): all write verbs at Request, reads at Metadata; still never above
+   Metadata for secrets/configmaps (rule 2 precedes; configmaps held at Metadata
+   here because some carry credentials-adjacent config).
+7. Drop noise: `system:kube-proxy` watches, `/healthz*`, `/readyz*`, `/livez*`,
+   `/metrics`, events, leases and node status heartbeats: None.
+8. Catch-all: Metadata for write verbs, None for read verbs elsewhere.
+
+Done when: policy file, a `make audit-policy-check` target (kubeconform-style
+schema check plus a script asserting rule 2 and the absence of any secrets match
+above Metadata), and an offline unit test on synthetic request fixtures.
+
+## Node log backend and rotation (source)
+
+```task
+id: RCLUSTER-WP-0021-T02
+status: todo
+priority: high
+assignee: claude
+```
+
+Extend the existing-host pattern of `ansible/k3s-kubeconfig-mode.yml` into
+`ansible/k3s-api-audit.yml` (and `bootstrap.yml` for fresh installs), writing
+k3s `config.yaml` `kube-apiserver-arg` entries:
+
+- `audit-policy-file=/etc/rancher/k3s/audit-policy.yaml` (root:root 0600)
+- `audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log` (dir 0700)
+- `audit-log-maxage=7`, `audit-log-maxbackup=10`, `audit-log-maxsize=100`
+- `audit-log-format=json`, `audit-log-mode=batch` (buffer sized so a slow disk
+  cannot stall the API server)
+
+Same guards as RCLUSTER-WP-0019: pinned preimage hashes, refuse on drift or
+foreign config.yaml keys, root-only backup, `--check`/`--diff` with content
+suppressed, a rollback action that removes the args. Validate offline only
+(syntax-check, ansible-lint production profile). Disk headroom check: refuse
+if `/var/lib/rancher` has less than 5 GiB free.
+
+## Shipping to audit-core (source; depends on audit-core contract)
+
+```task
+id: RCLUSTER-WP-0021-T03
+status: wait
+priority: high
+assignee: claude
+```
+
+A node-local shipper (Fluent Bit DaemonSet with hostPath read-only on the log
+dir, or a systemd unit on the host; choose after audit-core states its intake)
+tails `audit.log`, keeps a persisted file offset, maps each event to the
+audit-core envelope (`auditID` as idempotency key, `stageTimestamp`, user,
+verb, objectRef, responseStatus, source `kubernetes-api/railiance01`) and posts
+with retry and back-pressure. Sender credential comes from OpenBao via the
+`warden route` owner path; never a static token in Git. Fail mode: the node log
+is the buffer; shipper lag is exported as a metric and alerted.
+
+Waits on audit-core: sender registration for `railiance-cluster` /
+`kubernetes-api`, the intake endpoint and auth method, envelope/schema version,
+expected volume ceiling (estimate in T05), and the correlation-id field from
+NK-WP-0055-T11.
+
+## Retention and restore check (source)
+
+```task
+id: RCLUSTER-WP-0021-T04
+status: wait
+priority: medium
+assignee: claude
+```
+
+Node retention is a buffer only (7 days / 10 x 100 MB). Durable retention
+(proposed: 400 days for Kubernetes API records) lives in audit-core and its
+backups. Add `make audit-k8s-restore-check`: read-only, emits a synthetic
+marker request (a labelled `SelfSubjectAccessReview` by a named operator
+identity), then confirms its `auditID` is present in audit-core and, during
+audit-core's recurring restore drill, in the restored copy with an intact hash
+chain. Receipt goes to `docs/evidence/`. Waits on T03 and audit-core's drill.
+
+## Founder-run live rollout (attended k3s restart)
+
+```task
+id: RCLUSTER-WP-0021-T05
+status: wait
+priority: high
+assignee: founder
+```
+
+Founder runs, attended, after T01/T02 are reviewed:
+`ansible-playbook ansible/k3s-api-audit.yml --check --diff`, then the apply,
+which restarts k3s (single node: brief API outage, workloads keep running).
+Verify `/readyz`, node Ready, `audit.log` growing, a test `kubectl get secret`
+logged at Metadata with no `requestObject`/`responseObject`, then a 24 h
+volume sample to size T03/T04. Rollback is the playbook's rollback action.
+Agents prepare and hand over the exact commands; they do not run them.