# Phase 0a — Pre-cluster Secret Bootstrap This directory contains tooling for the KeePassXC bootstrap phase (T01 Phase 0a). No secrets are stored here — only scripts and documentation. ## Why KeePassXC first? The single-credential bootstrap principle (Decision D1): one master password unlocks the KeePassXC vault; all other credentials are generated inside it. KeePassXC is the pre-cluster source of truth. After the K3s cluster is running, secrets migrate into HashiCorp Vault (T01 Phase 0b) and KeePassXC becomes the break-glass / dev-local backup. ## Historical KeePassXC layout This early layout is historical; Keycloak is backlog, not a current runtime requirement. Use existing independently held custody and the current asset manifest. Do not generate replacement credentials or enroll a new recovery key merely to make this layout match a failed recovery. Recommended group structure: ``` net-kingdom/ ├── privacyIDEA/ │ ├── pi-admin (username: pi-admin, password: PI_ADMIN_PASSWORD) │ ├── database (username: privacyidea, password: PI_DB_PASSWORD) │ ├── SECRET_KEY (password field only — PI_SECRET_KEY value) │ ├── PI_PEPPER (password field only — PI_PEPPER value) │ └── PI_ENCFILE (binary attachment: pi.enc — generate after deploy) ├── PostgreSQL/ │ ├── postgres root (username: postgres, password: PG_ROOT_PASSWORD) │ ├── keycloak user (username: keycloak, password: PG_KEYCLOAK_PASSWORD) │ └── privacyidea user (username: privacyidea — same password as PI_DB_PASSWORD) ├── Keycloak/ │ ├── admin (username: admin, password: KC_ADMIN_PASSWORD) │ └── database (username: keycloak — same password as PG_KEYCLOAK_PASSWORD) ├── Break-glass/ │ ├── break-glass (username: break-glass, password: BREAKGLASS_PASSWORD) │ └── recovery-otp (TOTP seed — enroll manually after Keycloak is up) └── Vault/ └── (populated in Phase 0b after Vault is deployed) ``` ## Workflow ### Step 1 — Generate secrets ```bash chmod +x gen-secrets.sh ./gen-secrets.sh ./secrets ``` This writes `.env` files to `./secrets/` (gitignored). Handle them only through the private custody procedure; never print them or copy values into chat. ### Step 2 — Confirm independent custody and the registered recipient Use the existing independently registered public recipient in `keys/age.pub`. Encryption needs only that public file. Do not generate, discover or install a permanent private key as part of bootstrap. The matching private key stays in its independently held password-safe/offline custody; temporary attended unlock uses the [custody procedure](../../docs/security-bootstrap-age-custody.md). If it is unavailable, stop and resolve custody through the owner rather than create a new key that cannot decrypt existing backups. ### Step 3 — Create the encrypted ops bundle ```bash chmod +x pack-bundle.sh ./pack-bundle.sh ./secrets "age1..." ops-bundle.tar.age ``` Store `ops-bundle.tar.age` offsite (cloud storage, external drive, separate location). The destination must be new. Inputs must be owned private regular files, without links, special files or age private keys. There is no plaintext tar staging file. Independently verify recovery and service-file coverage before source cleanup; bundle creation alone proves neither current credential validity nor recovery. ### Step 4 — Shred the generated files ```bash find ./secrets -type f -exec shred -u {} \; rm -rf ./secrets ``` ### Step 5 — privacyIDEA encryption and audit keys Follow the current [key recovery procedure](../k8s/privacyidea/KEY-RECOVERY.md). Export the verified existing PVC encryption key and matching audit pair through `enckey-bootstrap.sh` into owned private custody. Missing recovery keys are not regenerated. Creating an audit pair is an explicit first-bootstrap-only action, not recovery. The old unconditional `pi-manage create_enckey`, raw `kubectl cp` and `PI_ENCFILE` Secret recipes are withdrawn; they could destroy continuity or misrepresent the application's actual PVC key path. ## Notes on secret reuse Some secrets appear in multiple components — this is intentional to avoid drift. When adding to KeePassXC, note the cross-references rather than duplicating the value: - `PI_DB_PASSWORD` == PostgreSQL `privacyidea` user password - `KC_DB_PASSWORD` == PostgreSQL `keycloak` user password Use KeePassXC references (`{REF:P@T:UUID}`) to avoid maintaining two copies. ## Phase 0b — HashiCorp Vault (after T02, once K3s is running) Current OpenBao configuration, ESO delivery and custody belong to `railiance-platform`; use its exact reviewed procedure. Legacy bootstrap references do not authorize live initialization, a new root, or replaying old service values into existing workloads. ## Current bootstrap and recovery handoff `make creds-agent-init` is now attended and public-recipient based. Register `keys/age.pub` from independently established custody first. No root private key is generated or discovered by bootstrap. `--dry-run` is a non-mutating plan. `make creds-emergency-reprint` prints only the non-secret recovery guide. For a resume that needs decryption, use the custody unlock helper below. After apply, the operator must store and independently verify the fresh encrypted ops bundle before acknowledging its handoff. Source tests use disposable keys and a simulated cluster; native acceptance remains NK-WP-0049. ## Service Secret creation and explicit rotation The PostgreSQL, privacyIDEA, LLDAP and Authelia `create-secrets.sh` entry points use `apply-service-secrets.py`. Under the current approved bootstrap/maintenance procedure, pass the private generator directory as their first argument. The helper requires Python 3, kubectl and OpenSSL; Authelia also requires Python bcrypt. Input directories/files must be owned, private and free of symlinks or hard-linked files. Generated env files are parsed as literal data, never sourced. Absent Secrets are created. Existing Secrets must match the supplied values; ordinary reapplication cannot rotate them. Objects with owner references or ESO management labels, immutable Secrets, and mismatched target objects are refused. Unknown data keys, labels and unrelated annotations are preserved. Any stored last-applied annotation is removed under the same guarded update to stop retaining duplicate secret values. Writes use stdin and captured diagnostics. Updates test the original object UID, resource version and data; conflicts fail without retry. Returned API state is checked before reporting success. A lost/failed response can leave the remote outcome unknown; inspect through approved metadata-only tooling before retrying. For an already reviewed rotation, the explicit interface is: ```bash bash sso-mfa/k8s/privacyidea/create-secrets.sh /absolute/private/secrets --rotate-key PI_SECRET_KEY ``` Only these component/field pairs are accepted: | Component | Rotation selector | Updated Secret field | | --- | --- | --- | | PostgreSQL | `PI_DB_PASSWORD` | `password` | | privacyIDEA | `PI_DB_PASSWORD` | `PI_SQLALCHEMY_DATABASE_URI` | | privacyIDEA | `PI_SECRET_KEY` | `PI_SECRET_KEY` | | LLDAP | `LLDAP_JWT_SECRET` | `LLDAP_JWT_SECRET` | | Authelia | `AUTHELIA_SESSION_SECRET` | `session_secret` | The selector describes an operation; it grants no authority. All other supplied fields must still match. Rotation requires the target Secret to exist. The guided caller passes these selectors and updates both the CNPG application Secret and privacyIDEA URI during database password rotation. SQL, Secret writes and restarts are separate steps, not an atomic transaction. Native database reconciliation, predecessor denial and consumer acceptance remain required; failure preserves the local candidate and does not record completed rotation. The legacy LDAP/Authelia/KeyCape coordinated replacement path refuses before mutation in both attended and unattended modes, as do KeyCape RSA/client-secret replacement paths. Implement and review retained rollback custody and scoped consumer updates before using a replacement procedure (NK-WP-0049-T02). For first Authelia creation, an explicit `AUTHELIA_OIDC_PRIVATE_KEY_FILE` must be an absolute path to a private owned RSA key. Without it, the helper preserves or atomically creates `authelia/oidc_private_key.pem` under the private input root. A new key survives API failure for recovery. Existing deployments retain their current issuer key; missing local custody never triggers a replacement key. An explicitly supplied differing key is refused. Existing bcrypt hashes are verified against the supplied client value and preserved, not regenerated. Independent signing-key backup remains part of the owner recovery ceremony. `creds-apply.sh` captures delegated output and records a component only after its helper succeeds. Batch application is not atomic: earlier components may already be applied when a later one fails. File status is not native acceptance. ## Custody Age Key Unlock For SOPS-backed drills and incident work, do not copy the custodian age private key into a permanent workstation path. Use the custody helper from the repo root: ```bash make sops-custody-run COMMAND='make -C /home/worsch/inter-hub recovery-drill' ``` The helper prompts for the `AGE-SECRET-KEY-1...` line from the password safe or offline custody packet, validates it against `keys/age.pub`, writes a temporary `SOPS_AGE_KEY_FILE`, runs the command, and removes the temporary file on exit. For multi-step recovery work: ```bash make sops-custody-shell ``` Exit the shell when finished so the temporary key file is removed. For repository SOPS files, use `make sops-encrypt FILE=...` with private untracked JSON/YAML plaintext. Inside the approved custody handoff, `make sops-decrypt FILE=... OUTPUT=...` creates a new private file outside Git, and `make sops-edit FILE=... INPUT=...` replaces encrypted content from a private file outside Git. Remove recovered/replacement plaintext after use. `make sops-rotate FILE=...` rotates only the data key; recipient migration requires a separately reviewed procedure. Edit/rotate require the explicit matching key and exactly the registered recipient. See [the custody contract](../../docs/security-bootstrap-age-custody.md) for input restrictions, failure handling and concurrency limits.