net-kingdom/sso-mfa/bootstrap

Phase 0a — Pre-cluster Secret Bootstrap

This directory contains tooling for the KeePassXC bootstrap phase (T01 Phase 0a). No secrets are stored here — only scripts and documentation.

Why KeePassXC first?

The single-credential bootstrap principle (Decision D1): one master password unlocks the KeePassXC vault; all other credentials are generated inside it. KeePassXC is the pre-cluster source of truth. After the K3s cluster is running, secrets migrate into HashiCorp Vault (T01 Phase 0b) and KeePassXC becomes the break-glass / dev-local backup.

KeePassXC database structure

Create a new .kdbx database. Use a strong master password stored in your personal password manager (Bitwarden, 1Password, etc.).

Recommended group structure:

net-kingdom/
├── privacyIDEA/
│   ├── pi-admin          (username: pi-admin, password: PI_ADMIN_PASSWORD)
│   ├── database          (username: privacyidea, password: PI_DB_PASSWORD)
│   ├── SECRET_KEY        (password field only — PI_SECRET_KEY value)
│   ├── PI_PEPPER         (password field only — PI_PEPPER value)
│   └── PI_ENCFILE        (binary attachment: pi.enc — generate after deploy)
├── PostgreSQL/
│   ├── postgres root     (username: postgres, password: PG_ROOT_PASSWORD)
│   ├── keycloak user     (username: keycloak, password: PG_KEYCLOAK_PASSWORD)
│   └── privacyidea user  (username: privacyidea — same password as PI_DB_PASSWORD)
├── Keycloak/
│   ├── admin             (username: admin, password: KC_ADMIN_PASSWORD)
│   └── database          (username: keycloak — same password as PG_KEYCLOAK_PASSWORD)
├── Break-glass/
│   ├── break-glass       (username: break-glass, password: BREAKGLASS_PASSWORD)
│   └── recovery-otp      (TOTP seed — enroll manually after Keycloak is up)
└── Vault/
    └── (populated in Phase 0b after Vault is deployed)

Workflow

Step 1 — Generate secrets

chmod +x gen-secrets.sh
./gen-secrets.sh ./secrets

This writes .env files to ./secrets/ (gitignored). Inspect each file, then copy each value into the appropriate KeePassXC entry.

Step 2 — Create the age encryption key (one-time)

age-keygen -o ~/net-kingdom-ops-bundle.key

The public key prints to stdout. The private key is in the .key file. Store the .key file somewhere safe (NOT in this repo; NOT in the secrets/ dir).

Step 3 — Create the encrypted ops bundle

chmod +x pack-bundle.sh
./pack-bundle.sh ./secrets "age1..." ops-bundle.tar.age

Store ops-bundle.tar.age offsite (cloud storage, external drive, separate location).

Step 4 — Shred the generated files

find ./secrets -type f -exec shred -u {} \;
rm -rf ./secrets

Step 5 — PI_ENCFILE (after privacyIDEA container is running — T04)

# Generate the encryption key inside the running container:
kubectl exec -n mfa <pi-pod-name> -- pi-manage create_enckey

# Extract it:
kubectl cp -n mfa <pi-pod-name>:/etc/privacyidea/enckey ./pi.enc

# Store as a binary attachment in KeePassXC → net-kingdom/privacyIDEA/PI_ENCFILE

# Create the K8s Secret:
kubectl create secret generic privacyidea-enckey \
    --from-file=PI_ENCFILE=./pi.enc \
    --namespace mfa

# Shred the local copy:
shred -u ./pi.enc

Notes on secret reuse

Some secrets appear in multiple components — this is intentional to avoid drift. When adding to KeePassXC, note the cross-references rather than duplicating the value:

• PI_DB_PASSWORD == PostgreSQL privacyidea user password
• KC_DB_PASSWORD == PostgreSQL keycloak user password

Use KeePassXC references ({REF:P@T:UUID}) to avoid maintaining two copies.

Phase 0b — HashiCorp Vault (after T02, once K3s is running)

See ../vault/ (created in T01 Phase 0b) for:

• Vault Helm chart values
• ESO (External Secrets Operator) configuration
• Vault secret path layout
• Migration procedure: KeePassXC → Vault

Current bootstrap and recovery handoff

make creds-agent-init is now attended and public-recipient based. Register keys/age.pub from independently established custody first. No root private key is generated or discovered by bootstrap. --dry-run is a non-mutating plan. make creds-emergency-reprint prints only the non-secret recovery guide.

For a resume that needs decryption, use the custody unlock helper below. After apply, the operator must store and independently verify the fresh encrypted ops bundle before acknowledging its handoff. Source tests use disposable keys and a simulated cluster; native acceptance remains NK-WP-0049.

Service Secret creation and explicit rotation

The PostgreSQL, privacyIDEA, LLDAP and Authelia create-secrets.sh entry points use apply-service-secrets.py. Under the current approved bootstrap/maintenance procedure, pass the private generator directory as their first argument. The helper requires Python 3, kubectl and OpenSSL; Authelia also requires Python bcrypt. Input directories/files must be owned, private and free of symlinks or hard-linked files. Generated env files are parsed as literal data, never sourced.

Absent Secrets are created. Existing Secrets must match the supplied values; ordinary reapplication cannot rotate them. Objects with owner references or ESO management labels, immutable Secrets, and mismatched target objects are refused. Unknown data keys, labels and unrelated annotations are preserved. Any stored last-applied annotation is removed under the same guarded update to stop retaining duplicate secret values. Writes use stdin and captured diagnostics. Updates test the original object UID, resource version and data; conflicts fail without retry. Returned API state is checked before reporting success. A lost/failed response can leave the remote outcome unknown; inspect through approved metadata-only tooling before retrying.

For an already reviewed rotation, the explicit interface is:

bash sso-mfa/k8s/privacyidea/create-secrets.sh /absolute/private/secrets --rotate-key PI_SECRET_KEY

Only these component/field pairs are accepted:

Component	Rotation selector	Updated Secret field
PostgreSQL	PI_DB_PASSWORD	password
privacyIDEA	PI_DB_PASSWORD	PI_SQLALCHEMY_DATABASE_URI
privacyIDEA	PI_SECRET_KEY	PI_SECRET_KEY
LLDAP	LLDAP_JWT_SECRET	LLDAP_JWT_SECRET
Authelia	AUTHELIA_SESSION_SECRET	session_secret

The selector describes an operation; it grants no authority. All other supplied fields must still match. Rotation requires the target Secret to exist. The guided caller passes these selectors and updates both the CNPG application Secret and privacyIDEA URI during database password rotation. SQL, Secret writes and restarts are separate steps, not an atomic transaction. Native database reconciliation, predecessor denial and consumer acceptance remain required; failure preserves the local candidate and does not record completed rotation. The legacy LDAP/Authelia/KeyCape coordinated replacement path refuses before mutation in both attended and unattended modes, as do KeyCape RSA/client-secret replacement paths. Implement and review retained rollback custody and scoped consumer updates before using a replacement procedure (NK-WP-0049-T02).

For first Authelia creation, an explicit AUTHELIA_OIDC_PRIVATE_KEY_FILE must be an absolute path to a private owned RSA key. Without it, the helper preserves or atomically creates authelia/oidc_private_key.pem under the private input root. A new key survives API failure for recovery. Existing deployments retain their current issuer key; missing local custody never triggers a replacement key. An explicitly supplied differing key is refused. Existing bcrypt hashes are verified against the supplied client value and preserved, not regenerated. Independent signing-key backup remains part of the owner recovery ceremony.

creds-apply.sh captures delegated output and records a component only after its helper succeeds. Batch application is not atomic: earlier components may already be applied when a later one fails. File status is not native acceptance.

Custody Age Key Unlock

For SOPS-backed drills and incident work, do not copy the custodian age private key into a permanent workstation path. Use the custody helper from the repo root:

make sops-custody-run COMMAND='make -C /home/worsch/inter-hub recovery-drill'

The helper prompts for the AGE-SECRET-KEY-1... line from the password safe or offline custody packet, validates it against keys/age.pub, writes a temporary SOPS_AGE_KEY_FILE, runs the command, and removes the temporary file on exit.

For multi-step recovery work:

make sops-custody-shell

Exit the shell when finished so the temporary key file is removed.

For repository SOPS files, use make sops-encrypt FILE=... with private untracked JSON/YAML plaintext. Inside the approved custody handoff, make sops-decrypt FILE=... OUTPUT=... creates a new private file outside Git, and make sops-edit FILE=... INPUT=... replaces encrypted content from a private file outside Git. Remove recovered/replacement plaintext after use. make sops-rotate FILE=... rotates only the data key; recipient migration requires a separately reviewed procedure. Edit/rotate require the explicit matching key and exactly the registered recipient. See the custody contract for input restrictions, failure handling and concurrency limits.