net-kingdom/sso-mfa/bootstrap

Phase 0a — Pre-cluster Secret Bootstrap

This directory contains tooling for the KeePassXC bootstrap phase (T01 Phase 0a). No secrets are stored here — only scripts and documentation.

Why KeePassXC first?

The single-credential bootstrap principle (Decision D1): one master password unlocks the KeePassXC vault; all other credentials are generated inside it. KeePassXC is the pre-cluster source of truth. After the K3s cluster is running, secrets migrate into HashiCorp Vault (T01 Phase 0b) and KeePassXC becomes the break-glass / dev-local backup.

Historical KeePassXC layout

This early layout is historical; Keycloak is backlog, not a current runtime requirement. Use existing independently held custody and the current asset manifest. Do not generate replacement credentials or enroll a new recovery key merely to make this layout match a failed recovery.

Recommended group structure:

net-kingdom/
├── privacyIDEA/
│   ├── pi-admin          (username: pi-admin, password: PI_ADMIN_PASSWORD)
│   ├── database          (username: privacyidea, password: PI_DB_PASSWORD)
│   ├── SECRET_KEY        (password field only — PI_SECRET_KEY value)
│   ├── PI_PEPPER         (password field only — PI_PEPPER value)
│   └── PI_ENCFILE        (binary attachment: pi.enc — generate after deploy)
├── PostgreSQL/
│   ├── postgres root     (username: postgres, password: PG_ROOT_PASSWORD)
│   ├── keycloak user     (username: keycloak, password: PG_KEYCLOAK_PASSWORD)
│   └── privacyidea user  (username: privacyidea — same password as PI_DB_PASSWORD)
├── Keycloak/
│   ├── admin             (username: admin, password: KC_ADMIN_PASSWORD)
│   └── database          (username: keycloak — same password as PG_KEYCLOAK_PASSWORD)
├── Break-glass/
│   ├── break-glass       (username: break-glass, password: BREAKGLASS_PASSWORD)
│   └── recovery-otp      (TOTP seed — enroll manually after Keycloak is up)
└── Vault/
    └── (populated in Phase 0b after Vault is deployed)

Workflow

Step 1 — Generate secrets

chmod +x gen-secrets.sh
./gen-secrets.sh ./secrets

This writes .env files to ./secrets/ (gitignored). Handle them only through the private custody procedure; never print them or copy values into chat.

Step 2 — Confirm independent custody and the registered recipient

Use the existing independently registered public recipient in keys/age.pub. Encryption needs only that public file. Do not generate, discover or install a permanent private key as part of bootstrap. The matching private key stays in its independently held password-safe/offline custody; temporary attended unlock uses the custody procedure. If it is unavailable, stop and resolve custody through the owner rather than create a new key that cannot decrypt existing backups.

Step 3 — Create the encrypted ops bundle

chmod +x pack-bundle.sh
./pack-bundle.sh ./secrets "age1..." ops-bundle.tar.age

Store ops-bundle.tar.age offsite (cloud storage, external drive, separate location). The destination must be new. Inputs must be owned private regular files, without links, special files or age private keys. There is no plaintext tar staging file. Independently verify recovery and service-file coverage before source cleanup; bundle creation alone proves neither current credential validity nor recovery.

Step 4 — Shred the generated files

find ./secrets -type f -exec shred -u {} \;
rm -rf ./secrets

Step 5 — privacyIDEA encryption and audit keys

Follow the current key recovery procedure. Export the verified existing PVC encryption key and matching audit pair through enckey-bootstrap.sh into owned private custody. Missing recovery keys are not regenerated. Creating an audit pair is an explicit first-bootstrap-only action, not recovery. The old unconditional pi-manage create_enckey, raw kubectl cp and PI_ENCFILE Secret recipes are withdrawn; they could destroy continuity or misrepresent the application's actual PVC key path.

Notes on secret reuse

Some secrets appear in multiple components — this is intentional to avoid drift. When adding to KeePassXC, note the cross-references rather than duplicating the value:

• PI_DB_PASSWORD == PostgreSQL privacyidea user password
• KC_DB_PASSWORD == PostgreSQL keycloak user password

Use KeePassXC references ({REF:P@T:UUID}) to avoid maintaining two copies.

Phase 0b — HashiCorp Vault (after T02, once K3s is running)

Current OpenBao configuration, ESO delivery and custody belong to railiance-platform; use its exact reviewed procedure. Legacy bootstrap references do not authorize live initialization, a new root, or replaying old service values into existing workloads.

Current bootstrap and recovery handoff

make creds-agent-init is now attended and public-recipient based. Register keys/age.pub from independently established custody first. No root private key is generated or discovered by bootstrap. --dry-run is a non-mutating plan. make creds-emergency-reprint prints only the non-secret recovery guide.

For a resume that needs decryption, use the custody unlock helper below. After apply, the operator must store and independently verify the fresh encrypted ops bundle before acknowledging its handoff. Source tests use disposable keys and a simulated cluster; native acceptance remains NK-WP-0049.

Service Secret creation and explicit rotation

The PostgreSQL, privacyIDEA, LLDAP and Authelia create-secrets.sh entry points use apply-service-secrets.py. Under the current approved bootstrap/maintenance procedure, pass the private generator directory as their first argument. The helper requires Python 3, kubectl and OpenSSL; Authelia also requires Python bcrypt. Input directories/files must be owned, private and free of symlinks or hard-linked files. Generated env files are parsed as literal data, never sourced.

Absent Secrets are created. Existing Secrets must match the supplied values; ordinary reapplication cannot rotate them. Objects with owner references or ESO management labels, immutable Secrets, and mismatched target objects are refused. Unknown data keys, labels and unrelated annotations are preserved. Any stored last-applied annotation is removed under the same guarded update to stop retaining duplicate secret values. Writes use stdin and captured diagnostics. Updates test the original object UID, resource version and data; conflicts fail without retry. Returned API state is checked before reporting success. A lost/failed response can leave the remote outcome unknown; inspect through approved metadata-only tooling before retrying.

For an already reviewed rotation, the explicit interface is:

bash sso-mfa/k8s/privacyidea/create-secrets.sh /absolute/private/secrets --rotate-key PI_SECRET_KEY

Only these component/field pairs are accepted:

Component	Rotation selector	Updated Secret field
PostgreSQL	PI_DB_PASSWORD	password
privacyIDEA	PI_DB_PASSWORD	PI_SQLALCHEMY_DATABASE_URI
privacyIDEA	PI_SECRET_KEY	PI_SECRET_KEY
LLDAP	LLDAP_JWT_SECRET	LLDAP_JWT_SECRET
Authelia	AUTHELIA_SESSION_SECRET	session_secret

The selector describes an operation; it grants no authority. All other supplied fields must still match. Rotation requires the target Secret to exist. The guided caller passes these selectors and updates both the CNPG application Secret and privacyIDEA URI during database password rotation. SQL, Secret writes and restarts are separate steps, not an atomic transaction. Native database reconciliation, predecessor denial and consumer acceptance remain required; failure preserves the local candidate and does not record completed rotation. The legacy LDAP/Authelia/KeyCape coordinated replacement path refuses before mutation in both attended and unattended modes, as do KeyCape RSA/client-secret replacement paths. Implement and review retained rollback custody and scoped consumer updates before using a replacement procedure (NK-WP-0049-T02).

For first Authelia creation, an explicit AUTHELIA_OIDC_PRIVATE_KEY_FILE must be an absolute path to a private owned RSA key. Without it, the helper preserves or atomically creates authelia/oidc_private_key.pem under the private input root. A new key survives API failure for recovery. Existing deployments retain their current issuer key; missing local custody never triggers a replacement key. An explicitly supplied differing key is refused. Existing bcrypt hashes are verified against the supplied client value and preserved, not regenerated. Independent signing-key backup remains part of the owner recovery ceremony.

creds-apply.sh captures delegated output and records a component only after its helper succeeds. Batch application is not atomic: earlier components may already be applied when a later one fails. File status is not native acceptance.

Custody Age Key Unlock

For SOPS-backed drills and incident work, do not copy the custodian age private key into a permanent workstation path. Use the custody helper from the repo root:

make sops-custody-run COMMAND='make -C /home/worsch/inter-hub recovery-drill'

The helper prompts for the AGE-SECRET-KEY-1... line from the password safe or offline custody packet, validates it against keys/age.pub, writes a temporary SOPS_AGE_KEY_FILE, runs the command, and removes the temporary file on exit.

For multi-step recovery work:

make sops-custody-shell

Exit the shell when finished so the temporary key file is removed.

For repository SOPS files, use make sops-encrypt FILE=... with private untracked JSON/YAML plaintext. Inside the approved custody handoff, make sops-decrypt FILE=... OUTPUT=... creates a new private file outside Git, and make sops-edit FILE=... INPUT=... replaces encrypted content from a private file outside Git. Remove recovered/replacement plaintext after use. make sops-rotate FILE=... rotates only the data key; recipient migration requires a separately reviewed procedure. Edit/rotate require the explicit matching key and exactly the registered recipient. See the custody contract for input restrictions, failure handling and concurrency limits.