railiance-cluster/workplans/RCLUSTER-WP-0021-k3s-api-audit-to-audit-core.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	flavor	depends_on	related	state_hub_workstream_id
RCLUSTER-WP-0021	workplan	k3s API-server audit policy, node log backend and durable shipping to audit-core	financials	railiance-cluster	proposed	claude	railiance	2026-10-27	2026-10-27	implementation		NK-WP-0055 AUDIT-WP-0005 AUDIT-WP-0007 AUDIT-WP-0015 RCLUSTER-WP-0019	f127f44e-aa65-5141-be52-270b71e5b48d

Owner source work for NetKingdom NK-WP-0055-T11 (production-readiness ledger, area 9 "Audit sink", row "Kubernetes API audit: absent"). The single-node k3s server on railiance01 runs with no audit-policy-file and no audit log, so no Kubernetes API record exists for secrets access, RBAC changes or exec/attach.

Boundary: this repo owns the k3s server flags, the audit policy, the node-local log backend and rotation, and the shipper that runs on the node. audit-core owns the receiving API, sender registration, schema, hash chain, retention in its store and its restore drill. Nothing here is auto-applied; every host change runs through an attended playbook. The k3s restart is a founder-run live step.

Audit policy (source)

id: RCLUSTER-WP-0021-T01
status: todo
priority: high
assignee: claude
state_hub_task_id: "591d504d-6ad8-5303-8215-e57ae5b40881"

Add ansible/files/k3s-audit-policy.yaml (audit.k8s.io/v1, rules first-match):

1. omitStages: [RequestReceived] globally.
2. secrets, serviceaccounts/token, tokenreviews, subjectaccessreviews: Metadata only, in every namespace. The policy must contain no Request/RequestResponse rule that can match secrets; a test enforces this.
3. RBAC (rbac.authorization.k8s.io: roles, rolebindings, clusterroles, clusterrolebindings) write verbs: RequestResponse (no secret payloads).
4. pods/exec, pods/attach, pods/portforward, nodes/proxy: Metadata (the request body carries no stream content).
5. cert-manager (cert-manager.io, acme.cert-manager.io) writes: Request.
6. Namespaces sso, mfa, openbao (exact list confirmed against the cluster in T05): all write verbs at Request, reads at Metadata; still never above Metadata for secrets/configmaps (rule 2 precedes; configmaps held at Metadata here because some carry credentials-adjacent config).
7. Drop noise: system:kube-proxy watches, /healthz*, /readyz*, /livez*, /metrics, events, leases and node status heartbeats: None.
8. Catch-all: Metadata for write verbs, None for read verbs elsewhere.

Done when: policy file, a make audit-policy-check target (kubeconform-style schema check plus a script asserting rule 2 and the absence of any secrets match above Metadata), and an offline unit test on synthetic request fixtures.

Node log backend and rotation (source)

id: RCLUSTER-WP-0021-T02
status: todo
priority: high
assignee: claude
state_hub_task_id: "5587c33c-75be-539c-9338-dbc3021ca636"

Extend the existing-host pattern of ansible/k3s-kubeconfig-mode.yml into ansible/k3s-api-audit.yml (and bootstrap.yml for fresh installs), writing k3s config.yaml kube-apiserver-arg entries:

• audit-policy-file=/etc/rancher/k3s/audit-policy.yaml (root:root 0600)
• audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log (dir 0700)
• audit-log-maxage=7, audit-log-maxbackup=10, audit-log-maxsize=100
• audit-log-format=json, audit-log-mode=batch (buffer sized so a slow disk cannot stall the API server)

Same guards as RCLUSTER-WP-0019: pinned preimage hashes, refuse on drift or foreign config.yaml keys, root-only backup, --check/--diff with content suppressed, a rollback action that removes the args. Validate offline only (syntax-check, ansible-lint production profile). Disk headroom check: refuse if /var/lib/rancher has less than 5 GiB free.

Shipping to audit-core (source; depends on audit-core contract)

id: RCLUSTER-WP-0021-T03
status: wait
priority: high
assignee: claude
state_hub_task_id: "74ae7710-91e8-58d4-8765-71d28c72252c"

A node-local shipper (Fluent Bit DaemonSet with hostPath read-only on the log dir, or a systemd unit on the host; choose after audit-core states its intake) tails audit.log, keeps a persisted file offset, maps each event to the audit-core envelope (auditID as idempotency key, stageTimestamp, user, verb, objectRef, responseStatus, source kubernetes-api/railiance01) and posts with retry and back-pressure. Sender credential comes from OpenBao via the warden route owner path; never a static token in Git. Fail mode: the node log is the buffer; shipper lag is exported as a metric and alerted.

Waits on audit-core: sender registration for railiance-cluster / kubernetes-api, the intake endpoint and auth method, envelope/schema version, expected volume ceiling (estimate in T05), and the correlation-id field from NK-WP-0055-T11.

Retention and restore check (source)

id: RCLUSTER-WP-0021-T04
status: wait
priority: medium
assignee: claude
state_hub_task_id: "0f239591-e0d2-5fdf-a64c-27747ae47a54"

Node retention is a buffer only (7 days / 10 x 100 MB). Durable retention (proposed: 400 days for Kubernetes API records) lives in audit-core and its backups. Add make audit-k8s-restore-check: read-only, emits a synthetic marker request (a labelled SelfSubjectAccessReview by a named operator identity), then confirms its auditID is present in audit-core and, during audit-core's recurring restore drill, in the restored copy with an intact hash chain. Receipt goes to docs/evidence/. Waits on T03 and audit-core's drill.

Founder-run live rollout (attended k3s restart)

id: RCLUSTER-WP-0021-T05
status: wait
priority: high
assignee: founder
state_hub_task_id: "ca1d85c5-6fe3-58d5-8a27-0dd79794648e"

Founder runs, attended, after T01/T02 are reviewed: ansible-playbook ansible/k3s-api-audit.yml --check --diff, then the apply, which restarts k3s (single node: brief API outage, workloads keep running). Verify /readyz, node Ready, audit.log growing, a test kubectl get secret logged at Metadata with no requestObject/responseObject, then a 24 h volume sample to size T03/T04. Rollback is the playbook's rollback action. Agents prepare and hand over the exact commands; they do not run them.