net-kingdom/docs/security-bootstrap-age-custody.md

# Bootstrap Age Custody

Status: draft control-surface contract
Date: 2026-05-24

## Purpose

The custodian age keypair is the bootstrap envelope key for NetKingdom security
setup. It is not a login account and it is not the runtime secret manager.

The public key may be stored in the control surface and used by automation to
encrypt generated bootstrap bundles. The private key is supplied only during an
explicit unlock/apply or recovery ceremony.

## Minimal Secret Model

The minimal human-held secret set is:

| Secret | Home | Notes |
| --- | --- | --- |
| Custodian age private key | Password safe or offline custody packet | Used to decrypt bootstrap bundles for an attended apply/recovery ceremony |
| Platform-root password | Password safe or offline custody packet | Dedicated custody identity, not day-to-day account |
| Platform-root OTP/recovery | Authenticator plus offline recovery | Not stored in Git, State Hub, or bootstrap metadata |
| OpenBao recovery/unseal material | OpenBao custody packet | Created only after custody approval |

Everything else should be generated, encrypted to the custodian public key,
applied to the cluster, then shredded or rotated into OpenBao once OpenBao is
available.

## Trial Versus Custody Mode

Trial mode uses throwaway values to document the steps, UI, failure modes, and
operator instructions. Nothing created in trial mode should be trusted as live
security material.

Custody mode uses real generated secrets. The expected lifecycle is:

1. Register the custodian public age recipient in the control surface.
2. Generate or recover bootstrap secrets.
3. Encrypt the bundle to the custodian public key.
4. Apply secrets to the cluster during an attended ceremony.
5. Shred plaintext `sso-mfa/bootstrap/secrets/`.
6. Record only non-secret fingerprints, timestamps, and state transitions.

## Control Surface Rules

The NetKingdom control surface may store:

- custodian public age recipient;
- public-key fingerprint;
- private-key location reference, such as a password-safe entry label;
- encrypted bundle path and file count;
- whether plaintext bootstrap secrets are currently present; and
- non-secret apply/verification status.

It must not store:

- `AGE-SECRET-KEY-1...` private key material;
- generated passwords;
- OTP seeds;
- OpenBao root tokens or unseal shares;
- decrypted bootstrap files; or
- screenshots of secret output.

## Current Tooling Alignment

`sso-mfa/bootstrap/encrypt-secrets.sh` should accept an age public recipient
directly. Encryption does not require the private key.

`sso-mfa/bootstrap/decrypt-secrets.sh` requires the private key path and should
be used only in an explicit unlock/apply ceremony. After apply, plaintext files
must be shredded.

## Custody Unlock Helper

Use `sso-mfa/bootstrap/sops-custody-unlock.sh` when an operator needs to run a
SOPS-backed drill, recovery, or lockdown command but the private key is only in
the password safe or offline custody packet.

The helper disables shell tracing before handling input and refuses conflicting input modes.
Noninteractive callers must select `--from-stdin` or `--from-file` explicitly.

The helper:

- reads the private age key from one explicit source: hidden prompt, stdin, or a source file;
- derives the public age recipient with `age-keygen -y`;
- refuses to continue unless it matches the expected recipient;
- writes a `0600` temporary `SOPS_AGE_KEY_FILE`;
- runs the requested command or opens a temporary custody shell; and
- shreds/removes the temporary key file when the command or shell exits.

It must not be used to install the private key permanently on a workstation or
server. The source of truth remains the password safe/offline custody packet.

### Validate Custody Material

From the NetKingdom repo, validate the supplied private key against `keys/age.pub`
without keeping it on disk:

```bash
make sops-custody-check
```

### Run A One-Shot Recovery Drill

For inter-hub:

```bash
make sops-custody-run COMMAND='make -C /home/worsch/inter-hub recovery-drill'
```

The helper prompts for the `AGE-SECRET-KEY-1...` line from the password safe or
offline custody packet, validates it against the registered public recipient,
sets `SOPS_AGE_KEY_FILE` for the command, and removes the temporary key after
the drill exits.

If using a password-manager CLI, pipe only the private key field:

```bash
op read 'op://Platform/NetKingdom custodian age key/private-key' \
  | sso-mfa/bootstrap/sops-custody-unlock.sh \
      --from-stdin \
      -- make -C /home/worsch/inter-hub recovery-drill
```

### Open An Incident Shell

For multi-step recovery or lockdown work:

```bash
make sops-custody-shell
```

Run the required SOPS, recovery, or lockdown commands inside that shell. Exit
the shell to shred the temporary key file.

## Bootstrap implementation correction — 2026-10-12

The compatibility `creds-agent-init` command now encrypts using `keys/age.pub`.
It does not generate or discover private keys. Live bootstrap requires the
operator terminal; `--dry-run` is a non-mutating plan. For an encrypted resume,
use the explicit custody helper with the independently held matching key. The
bootstrap validates that temporary key before contacting the cluster.

The former emergency-bundle export is a non-secret guide. The current encrypted
ops bundle is handed off by location and SHA256, and a fresh operator
acknowledgment is required before plaintext cleanup/completion. Old delivery
flags retain their historical meaning and do not acknowledge a new artifact.
Generation no longer logs value prefixes and creates private files even under a
permissive caller umask. These controls do not enforce agent runtime isolation.

Recovery verification uses an owner-approved isolated workspace and target. With
the helper's temporary key, decrypt the exact bundle to a private file, verify
its expected contents without printing values, then remove decrypted files and
retain a non-secret receipt. Acknowledgment of a guide is not that verification.
No live root or password was read or rotated during the source correction.
NK-WP-0049 tracks remaining native acceptance and other legacy custody callers.

## Direct helper correction — 2026-10-12

`encrypt-secrets.sh` defaults to the registered `keys/age.pub`. It accepts a public
recipient or public-recipient file and rejects private-key files instead of
loading them to derive metadata. `decrypt-secrets.sh` accepts an explicit key
argument or the custody helper's `NETKINGDOM_CUSTODY_AGE_KEY_FILE`; it no longer
searches HOME. Successful decryption publishes a new private output directory
only after every input decrypts. Failed/empty input removes partial staging;
existing files, directories and dangling symlinks are never replaced.

The custody helper disables shell tracing, rejects repeated/mixed input modes,
does not echo unknown arguments and clears competing inline/key-command SOPS
authentication before the child starts. HUP/INT/TERM now retain failure status
and EXIT cleanup rather than running cleanup and then continuing. Bash may defer
its trap while a foreground child runs; this is not a process-group supervisor,
SIGKILL cleanup guarantee or a runtime authority boundary. Temporary key cleanup
on command return does not certify that an arbitrary child retained no copies.

Remaining concrete caller defects stay in NK-WP-0049-T02, including legacy
diagnostics, key export and coordinated rotation. Its current checklist records
repaired callers and outstanding work. Do not treat narrower helper repairs as
closure of the whole custody surface.

## Make recovery entry points — 2026-10-12

`make sops-setup` now validates only `keys/age.pub`. It never copies or generates
a private key; `creds-init` likewise has no permanent HOME key requirement.
Existing permanent files are not deleted or represented as remediated copies.

`make sops-decrypt FILE=... OUTPUT=...` requires the custody helper's explicit
`NETKINGDOM_CUSTODY_AGE_KEY_FILE`. The owned private regular key file must match
the registered public recipient. Output must be a new file in an owned private
parent outside Git worktrees (including linked-worktree markers and bare repos).
No destination file/symlink is overwritten. Partial tool output is removed on
failure; successful publication is atomic and mode 0600. The operator removes
successful plaintext after the approved operation. This does not promise cleanup
after SIGKILL, hostile same-UID interference or arbitrary child copies.

For the repository's age custody profile, the wrapper accepts SOPS JSON/YAML,
and rejects non-age recipients, age plugins, key groups, unknown metadata and
duplicate keys before decryption.
Other formats/backends require their own reviewed adapter; they are not silently
sent through a different credential source. YAML input requires PyYAML.

SOPS runs with an isolated HOME/XDG configuration and a minimal environment,
using an inherited descriptor for the explicit key. This prevents SOPS from
combining it with HOME/SSH keys, inline keys, key commands, proxies or remote key
services. Raw parser/tool diagnostics and plaintext stdout never enter the
terminal transcript. This wrapper assumes trusted installed `age`, `age-keygen`
and `sops` binaries; it is not an OS sandbox or task-authorization grant.

Use from the operator's approved custody terminal, with a private directory
created outside all repositories (for example using `mktemp -d`):

```bash
# OUTPUT's parent must exist and be owned/private; select a new filename.
make sops-custody-run COMMAND='make sops-decrypt FILE=secrets/example.yaml OUTPUT=/private/recovery/new-example.yaml'
```

The command string carries paths only. Never put a credential in FILE, OUTPUT,
COMMAND, shell history or Make variables. Tests exercise real SOPS 3.13.3 with
disposable keys; no production key or encrypted secret was accessed. The test
binary checksum was checked against the official release checksum; no detached
signature verification or installed-toolchain promotion is claimed.

Source basis: [SOPS age identity loading](https://github.com/getsops/sops/blob/v3.13.3/age/keysource.go)
loads explicit and ambient identities together, so setting `SOPS_AGE_KEY_FILE`
alone is insufficient to exclude ambient credentials.


## SOPS writes — 2026-10-13

The Make wrappers support the registered single-age JSON/YAML profile:

- `make sops-encrypt FILE=...` encrypts a private, owned, untracked plaintext
  mapping in place using only `keys/age.pub`. It needs no private key. Tracked
  plaintext is refused and needs incident review; encryption cannot erase history.
- `make sops-edit FILE=... INPUT=...` replaces content from an owned private
  plaintext file in a private directory outside Git. Use private decryption above
  to prepare that file, edit it in the approved custody environment, then remove
  it after successful replacement. No ambient editor is launched. INPUT must use
  the same JSON/YAML format. The original MAC is verified before replacement and
  its declared field-encryption policy is preserved.
- `make sops-rotate FILE=...` rotates the data key and re-encrypts values. It does
  **not** migrate recipients or apply changes to `.sops.yaml`. A recipient change
  needs its own reviewed custody migration, independent recipient admission and
  recovery verification.

Edit and rotate require the explicit matching temporary custody key. Both refuse
extra document recipients, even when the registered key can decrypt the file:
metadata is not authority to rewrap a data key for another recipient. The new
ciphertext is decrypted for verification with plaintext output discarded.
Inputs reject symlinks, hardlinks, nonregular files and unsafe modes; key inputs
also reject symlink ancestors and FIFOs. Parser and tool diagnostics are captured.
Make passes FILE, INPUT, OUTPUT and custody arguments literally, without evaluating
Make functions or interpolating them into shell code before the custody handoff.
The approved COMMAND is still intentionally executed inside that handoff.

Ciphertext publication uses private staging, a cooperative file lock and a final
source identity/version check followed by atomic rename. The check and rename are
separate operations: this is not protection against hostile concurrent same-UID
writers or ancestor-directory substitution. Directory fsync failure after rename
can report failure after publication; inspect state before retrying. SIGKILL may
leave staging files. Private modes and a temporary key are not credential isolation
or an authorization grant. Current native recovery acceptance remains NK-WP-0049-T03.

Version-specific behavior was exercised with disposable SOPS 3.13.3 and age keys;
see the [upstream SOPS documentation](https://github.com/getsops/sops/blob/v3.13.3/README.rst).

## Rotation output boundary (2026-10-12 source repair)

`creds-rotate.sh` keeps generated values out of stdout/stderr and disables shell
tracing. The selected plaintext secrets directory and service subdirectory must
be owned by the operator with no group/other access; `secrets.env` must be an
owned private regular file with one link and exactly one target assignment.
Symlink paths, missing keys and duplicate target assignments are refused before
cluster work. The helper replaces the file atomically with mode 0600 and never
places the new value in its command arguments. It accepts the generated
alphanumeric credential format. This is file hygiene, not process isolation.

Approved rotations preserve the new candidate in that private service file.
Transfer it through approved custody, verify real consumer authentication and
old-value refusal, refresh the encrypted recovery handoff, then remove plaintext.
A failed apply can leave a new local candidate while some services still use the
old value. Reconcile that partial state; do not infer completion from file contents.
Service-command diagnostics are suppressed because they can contain values;
use approved metadata-only diagnostics to investigate failures.

Database rotation stops before local mutation unless exactly one database pod
is discovered. SQL is sent on stdin with SQL-error failure enabled. Unattended
LLDAP admin-password rotation is refused because the server-update step is
manual. The attended path requires acknowledgment of that step before dependent
Secret updates; cancelling retains the local candidate and requires reconciliation.
Break-glass Secret creation reads the private file instead of a plaintext argv
literal and applies server-side without forcing field conflicts. This does not
remove historical client-side annotations or prove native consumer adoption.

The last-rotated timestamp records successful script steps only. No live rotation
was run for these source tests. CUST-WP-0073-T05 remains founder-deferred. RSA
rotation recovery/rollback, delegated helper argv/annotation handling and
`keycape/create-pi-token.sh` still require source and native acceptance. The
admin-bootstrap source repair below leaves native MFA and legacy-object revocation open.


## Current-profile admin bootstrap (2026-10-12 source repair)

The current KeyCape profile does not consume the old Keycloak trigger-admin
credential. `privacyidea/bootstrap-admin.sh` now creates/verifies only pi-admin
and its explicitly scoped bootstrap policy; it creates no trigger account, token
or Kubernetes Secret. It never deletes existing objects. Owner inventory and
revocation are required for historical credentials, independent of source repair.

Input is one private generated service-file assignment, parsed without sourcing
shell content. The password is sent through kubectl stdin into a version-scoped
privacyIDEA 3.12.2 adapter. That adapter verifies existing passwords and uses
insert-only writes for absent account/policy rows. It refuses broad/changed
legacy policies and existing installations that require policy migration.
The success receipt proves account/password and scoped-policy state only; MFA,
effective privileges, deployed-image behavior and consumer acceptance remain
attended native gates. See the privacyIDEA README for the exact operation scope.