tegwick
d9557ba90e
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a13eab-1e6a-7b33-9c37-372e90ada3a2
217 lines
10 KiB
Markdown
217 lines
10 KiB
Markdown
# Phase 0a — Pre-cluster Secret Bootstrap
|
|
|
|
This directory contains tooling for the KeePassXC bootstrap phase (T01 Phase 0a).
|
|
No secrets are stored here — only scripts and documentation.
|
|
|
|
## Why KeePassXC first?
|
|
|
|
The single-credential bootstrap principle (Decision D1): one master password
|
|
unlocks the KeePassXC vault; all other credentials are generated inside it.
|
|
KeePassXC is the pre-cluster source of truth. After the K3s cluster is running,
|
|
secrets migrate into HashiCorp Vault (T01 Phase 0b) and KeePassXC becomes the
|
|
break-glass / dev-local backup.
|
|
|
|
## Historical KeePassXC layout
|
|
|
|
This early layout is historical; Keycloak is backlog, not a current runtime
|
|
requirement. Use existing independently held custody and the current asset
|
|
manifest. Do not generate replacement credentials or enroll a new recovery key
|
|
merely to make this layout match a failed recovery.
|
|
|
|
Recommended group structure:
|
|
|
|
```
|
|
net-kingdom/
|
|
├── privacyIDEA/
|
|
│ ├── pi-admin (username: pi-admin, password: PI_ADMIN_PASSWORD)
|
|
│ ├── database (username: privacyidea, password: PI_DB_PASSWORD)
|
|
│ ├── SECRET_KEY (password field only — PI_SECRET_KEY value)
|
|
│ ├── PI_PEPPER (password field only — PI_PEPPER value)
|
|
│ └── PI_ENCFILE (binary attachment: pi.enc — generate after deploy)
|
|
├── PostgreSQL/
|
|
│ ├── postgres root (username: postgres, password: PG_ROOT_PASSWORD)
|
|
│ ├── keycloak user (username: keycloak, password: PG_KEYCLOAK_PASSWORD)
|
|
│ └── privacyidea user (username: privacyidea — same password as PI_DB_PASSWORD)
|
|
├── Keycloak/
|
|
│ ├── admin (username: admin, password: KC_ADMIN_PASSWORD)
|
|
│ └── database (username: keycloak — same password as PG_KEYCLOAK_PASSWORD)
|
|
├── Break-glass/
|
|
│ ├── break-glass (username: break-glass, password: BREAKGLASS_PASSWORD)
|
|
│ └── recovery-otp (TOTP seed — enroll manually after Keycloak is up)
|
|
└── Vault/
|
|
└── (populated in Phase 0b after Vault is deployed)
|
|
```
|
|
|
|
## Workflow
|
|
|
|
### Step 1 — Generate secrets
|
|
|
|
```bash
|
|
chmod +x gen-secrets.sh
|
|
./gen-secrets.sh ./secrets
|
|
```
|
|
|
|
This writes `.env` files to `./secrets/` (gitignored). Handle them only through
|
|
the private custody procedure; never print them or copy values into chat.
|
|
|
|
### Step 2 — Confirm independent custody and the registered recipient
|
|
|
|
Use the existing independently registered public recipient in `keys/age.pub`.
|
|
Encryption needs only that public file. Do not generate, discover or install a
|
|
permanent private key as part of bootstrap. The matching private key stays in
|
|
its independently held password-safe/offline custody; temporary attended unlock
|
|
uses the [custody procedure](../../docs/security-bootstrap-age-custody.md).
|
|
If it is unavailable, stop and resolve custody through the owner rather than
|
|
create a new key that cannot decrypt existing backups.
|
|
|
|
### Step 3 — Create the encrypted ops bundle
|
|
|
|
```bash
|
|
chmod +x pack-bundle.sh
|
|
./pack-bundle.sh ./secrets "age1..." ops-bundle.tar.age
|
|
```
|
|
|
|
Store `ops-bundle.tar.age` offsite (cloud storage, external drive, separate location).
|
|
The destination must be new. Inputs must be owned private regular files, without
|
|
links, special files or age private keys. There is no plaintext tar staging file.
|
|
Independently verify recovery and service-file coverage before source cleanup;
|
|
bundle creation alone proves neither current credential validity nor recovery.
|
|
|
|
### Step 4 — Shred the generated files
|
|
|
|
```bash
|
|
find ./secrets -type f -exec shred -u {} \;
|
|
rm -rf ./secrets
|
|
```
|
|
|
|
### Step 5 — privacyIDEA encryption and audit keys
|
|
|
|
Follow the current [key recovery procedure](../k8s/privacyidea/KEY-RECOVERY.md).
|
|
Export the verified existing PVC encryption key and matching audit pair through
|
|
`enckey-bootstrap.sh` into owned private custody. Missing recovery keys are not
|
|
regenerated. Creating an audit pair is an explicit first-bootstrap-only action,
|
|
not recovery. The old unconditional `pi-manage create_enckey`, raw `kubectl cp`
|
|
and `PI_ENCFILE` Secret recipes are withdrawn; they could destroy continuity or
|
|
misrepresent the application's actual PVC key path.
|
|
|
|
## Notes on secret reuse
|
|
|
|
Some secrets appear in multiple components — this is intentional to avoid
|
|
drift. When adding to KeePassXC, note the cross-references rather than
|
|
duplicating the value:
|
|
|
|
- `PI_DB_PASSWORD` == PostgreSQL `privacyidea` user password
|
|
- `KC_DB_PASSWORD` == PostgreSQL `keycloak` user password
|
|
|
|
Use KeePassXC references (`{REF:P@T:UUID}`) to avoid maintaining two copies.
|
|
|
|
## Phase 0b — HashiCorp Vault (after T02, once K3s is running)
|
|
|
|
Current OpenBao configuration, ESO delivery and custody belong to
|
|
`railiance-platform`; use its exact reviewed procedure. Legacy bootstrap
|
|
references do not authorize live initialization, a new root, or replaying old
|
|
service values into existing workloads.
|
|
|
|
## Current bootstrap and recovery handoff
|
|
|
|
`make creds-agent-init` is now attended and public-recipient based. Register
|
|
`keys/age.pub` from independently established custody first. No root private key
|
|
is generated or discovered by bootstrap. `--dry-run` is a non-mutating plan.
|
|
`make creds-emergency-reprint` prints only the non-secret recovery guide.
|
|
|
|
For a resume that needs decryption, use the custody unlock helper below. After
|
|
apply, the operator must store and independently verify the fresh encrypted ops
|
|
bundle before acknowledging its handoff. Source tests use disposable keys and a
|
|
simulated cluster; native acceptance remains NK-WP-0049.
|
|
|
|
## Service Secret creation and explicit rotation
|
|
|
|
The PostgreSQL, privacyIDEA, LLDAP and Authelia `create-secrets.sh` entry points
|
|
use `apply-service-secrets.py`. Under the current approved bootstrap/maintenance
|
|
procedure, pass the private generator directory as their first argument. The
|
|
helper requires Python 3, kubectl and OpenSSL; Authelia also requires Python
|
|
bcrypt. Input directories/files must be owned, private and free of symlinks or
|
|
hard-linked files. Generated env files are parsed as literal data, never sourced.
|
|
|
|
Absent Secrets are created. Existing Secrets must match the supplied values;
|
|
ordinary reapplication cannot rotate them. Objects with owner references or ESO management labels, immutable Secrets,
|
|
and mismatched target objects are refused. Unknown data keys, labels
|
|
and unrelated annotations are preserved. Any stored last-applied annotation is
|
|
removed under the same guarded update to stop retaining duplicate secret values.
|
|
Writes use stdin and captured diagnostics. Updates test the original object UID,
|
|
resource version and data; conflicts fail without retry. Returned API state is
|
|
checked before reporting success. A lost/failed response can leave the remote
|
|
outcome unknown; inspect through approved metadata-only tooling before retrying.
|
|
|
|
For an already reviewed rotation, the explicit interface is:
|
|
|
|
```bash
|
|
bash sso-mfa/k8s/privacyidea/create-secrets.sh /absolute/private/secrets --rotate-key PI_SECRET_KEY
|
|
```
|
|
|
|
Only these component/field pairs are accepted:
|
|
|
|
| Component | Rotation selector | Updated Secret field |
|
|
| --- | --- | --- |
|
|
| PostgreSQL | `PI_DB_PASSWORD` | `password` |
|
|
| privacyIDEA | `PI_DB_PASSWORD` | `PI_SQLALCHEMY_DATABASE_URI` |
|
|
| privacyIDEA | `PI_SECRET_KEY` | `PI_SECRET_KEY` |
|
|
| LLDAP | `LLDAP_JWT_SECRET` | `LLDAP_JWT_SECRET` |
|
|
| Authelia | `AUTHELIA_SESSION_SECRET` | `session_secret` |
|
|
|
|
The selector describes an operation; it grants no authority. All other supplied
|
|
fields must still match. Rotation requires the target Secret to exist. The
|
|
guided caller passes these selectors and updates both the CNPG application
|
|
Secret and privacyIDEA URI during database password rotation. SQL, Secret writes
|
|
and restarts are separate steps, not an atomic transaction. Native database
|
|
reconciliation, predecessor denial and consumer acceptance remain required;
|
|
failure preserves the local candidate and does not record completed rotation.
|
|
The legacy LDAP/Authelia/KeyCape coordinated replacement path refuses before
|
|
mutation in both attended and unattended modes, as do KeyCape RSA/client-secret
|
|
replacement paths. Implement and review retained rollback custody and scoped
|
|
consumer updates before using a replacement procedure (NK-WP-0049-T02).
|
|
|
|
For first Authelia creation, an explicit `AUTHELIA_OIDC_PRIVATE_KEY_FILE` must be
|
|
an absolute path to a private owned RSA key. Without it, the helper preserves or
|
|
atomically creates `authelia/oidc_private_key.pem` under the private input root.
|
|
A new key survives API failure for recovery. Existing deployments retain their
|
|
current issuer key; missing local custody never triggers a replacement key.
|
|
An explicitly supplied differing key is refused. Existing bcrypt hashes are
|
|
verified against the supplied client value and preserved, not regenerated.
|
|
Independent signing-key backup remains part of the owner recovery ceremony.
|
|
|
|
`creds-apply.sh` captures delegated output and records a component only after
|
|
its helper succeeds. Batch application is not atomic: earlier components may
|
|
already be applied when a later one fails. File status is not native acceptance.
|
|
|
|
## Custody Age Key Unlock
|
|
|
|
For SOPS-backed drills and incident work, do not copy the custodian age private
|
|
key into a permanent workstation path. Use the custody helper from the repo root:
|
|
|
|
```bash
|
|
make sops-custody-run COMMAND='make -C /home/worsch/inter-hub recovery-drill'
|
|
```
|
|
|
|
The helper prompts for the `AGE-SECRET-KEY-1...` line from the password safe or
|
|
offline custody packet, validates it against `keys/age.pub`, writes a temporary
|
|
`SOPS_AGE_KEY_FILE`, runs the command, and removes the temporary file on exit.
|
|
|
|
For multi-step recovery work:
|
|
|
|
```bash
|
|
make sops-custody-shell
|
|
```
|
|
|
|
Exit the shell when finished so the temporary key file is removed.
|
|
|
|
|
|
For repository SOPS files, use `make sops-encrypt FILE=...` with private untracked
|
|
JSON/YAML plaintext. Inside the approved custody handoff, `make sops-decrypt
|
|
FILE=... OUTPUT=...` creates a new private file outside Git, and `make sops-edit
|
|
FILE=... INPUT=...` replaces encrypted content from a private file outside Git.
|
|
Remove recovered/replacement plaintext after use. `make sops-rotate FILE=...`
|
|
rotates only the data key; recipient migration requires a separately reviewed
|
|
procedure. Edit/rotate require the explicit matching key and exactly the registered
|
|
recipient. See [the custody contract](../../docs/security-bootstrap-age-custody.md)
|
|
for input restrictions, failure handling and concurrency limits.
|