diff --git a/sso-mfa/bootstrap/README.md b/sso-mfa/bootstrap/README.md index d868ea1..cf0d924 100644 --- a/sso-mfa/bootstrap/README.md +++ b/sso-mfa/bootstrap/README.md @@ -128,6 +128,66 @@ apply, the operator must store and independently verify the fresh encrypted ops bundle before acknowledging its handoff. Source tests use disposable keys and a simulated cluster; native acceptance remains NK-WP-0049. +## Service Secret creation and explicit rotation + +The PostgreSQL, privacyIDEA, LLDAP and Authelia `create-secrets.sh` entry points +use `apply-service-secrets.py`. Under the current approved bootstrap/maintenance +procedure, pass the private generator directory as their first argument. The +helper requires Python 3, kubectl and OpenSSL; Authelia also requires Python +bcrypt. Input directories/files must be owned, private and free of symlinks or +hard-linked files. Generated env files are parsed as literal data, never sourced. + +Absent Secrets are created. Existing Secrets must match the supplied values; +ordinary reapplication cannot rotate them. Objects with owner references or ESO management labels, immutable Secrets, +and mismatched target objects are refused. Unknown data keys, labels +and unrelated annotations are preserved. Any stored last-applied annotation is +removed under the same guarded update to stop retaining duplicate secret values. +Writes use stdin and captured diagnostics. Updates test the original object UID, +resource version and data; conflicts fail without retry. Returned API state is +checked before reporting success. A lost/failed response can leave the remote +outcome unknown; inspect through approved metadata-only tooling before retrying. + +For an already reviewed rotation, the explicit interface is: + +```bash +bash sso-mfa/k8s/privacyidea/create-secrets.sh /absolute/private/secrets --rotate-key PI_SECRET_KEY +``` + +Only these component/field pairs are accepted: + +| Component | Rotation selector | Updated Secret field | +| --- | --- | --- | +| PostgreSQL | `PI_DB_PASSWORD` | `password` | +| privacyIDEA | `PI_DB_PASSWORD` | `PI_SQLALCHEMY_DATABASE_URI` | +| privacyIDEA | `PI_SECRET_KEY` | `PI_SECRET_KEY` | +| LLDAP | `LLDAP_JWT_SECRET` | `LLDAP_JWT_SECRET` | +| Authelia | `AUTHELIA_SESSION_SECRET` | `session_secret` | + +The selector describes an operation; it grants no authority. All other supplied +fields must still match. Rotation requires the target Secret to exist. The +guided caller passes these selectors and updates both the CNPG application +Secret and privacyIDEA URI during database password rotation. SQL, Secret writes +and restarts are separate steps, not an atomic transaction. Native database +reconciliation, predecessor denial and consumer acceptance remain required; +failure preserves the local candidate and does not record completed rotation. +The legacy LDAP/Authelia/KeyCape coordinated replacement path refuses before +mutation in both attended and unattended modes, as do KeyCape RSA/client-secret +replacement paths. Implement and review retained rollback custody and scoped +consumer updates before using a replacement procedure (NK-WP-0049-T02). + +For first Authelia creation, an explicit `AUTHELIA_OIDC_PRIVATE_KEY_FILE` must be +an absolute path to a private owned RSA key. Without it, the helper preserves or +atomically creates `authelia/oidc_private_key.pem` under the private input root. +A new key survives API failure for recovery. Existing deployments retain their +current issuer key; missing local custody never triggers a replacement key. +An explicitly supplied differing key is refused. Existing bcrypt hashes are +verified against the supplied client value and preserved, not regenerated. +Independent signing-key backup remains part of the owner recovery ceremony. + +`creds-apply.sh` captures delegated output and records a component only after +its helper succeeds. Batch application is not atomic: earlier components may +already be applied when a later one fails. File status is not native acceptance. + ## Custody Age Key Unlock For SOPS-backed drills and incident work, do not copy the custodian age private diff --git a/sso-mfa/bootstrap/apply-service-secrets.py b/sso-mfa/bootstrap/apply-service-secrets.py new file mode 100644 index 0000000..d7db595 --- /dev/null +++ b/sso-mfa/bootstrap/apply-service-secrets.py @@ -0,0 +1,252 @@ +#!/usr/bin/env python3 +"""Private service Secret creation, exact-match verification and named rotation. + +No credential is accepted on argv, evaluated as shell code, or printed. This +helper is not an authority grant or a multi-service rotation transaction. +""" +import base64 +import contextlib +import json +import os +from pathlib import Path +import re +import secrets +import stat +import subprocess +import sys + + +class Refused(Exception): + pass + + +TARGETS = { + 'postgresql': ('databases', 'net-kingdom-pg-privacyidea-app', ('privacyidea',)), + 'privacyidea': ('mfa', 'privacyidea-config', ('privacyidea',)), + 'lldap': ('sso', 'lldap-secrets', ('lldap',)), + 'authelia': ('sso', 'authelia-secrets', ('authelia', 'lldap')), +} +ROTATIONS = { + 'postgresql': {'PI_DB_PASSWORD': 'password'}, + 'privacyidea': {'PI_SECRET_KEY': 'PI_SECRET_KEY', 'PI_DB_PASSWORD': 'PI_SQLALCHEMY_DATABASE_URI'}, + 'lldap': {'LLDAP_JWT_SECRET': 'LLDAP_JWT_SECRET'}, + 'authelia': {'AUTHELIA_SESSION_SECRET': 'session_secret'}, +} +ANNOTATION = 'kubectl.kubernetes.io/last-applied-configuration' + + +def private(info, directory=False): + kind = stat.S_ISDIR if directory else stat.S_ISREG + if not kind(info.st_mode) or info.st_uid != os.getuid() or info.st_mode & 0o077: + raise Refused() + if not directory and info.st_nlink != 1: + raise Refused() + + +@contextlib.contextmanager +def custody(root, components): + root = Path(os.path.abspath(root)) + if any(p.is_symlink() for p in (root, *root.parents)): raise Refused() + descriptors = [] + try: + flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW + fd = os.open(root, flags); descriptors.append(fd); private(os.fstat(fd), True) + children = {} + for component in components: + child = os.open(component, flags, dir_fd=fd) + descriptors.append(child); private(os.fstat(child), True) + children[component] = child + yield children + finally: + for fd in reversed(descriptors): os.close(fd) + + +def read_private(parent, name): + fd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=parent) + try: + private(os.fstat(fd)) + raw = os.read(fd, 65537) + if len(raw) > 65536: raise Refused() + return raw + finally: + os.close(fd) + + +def read_env(parent): + values = {} + for line in read_private(parent, 'secrets.env').decode().splitlines(): + if not line.strip() or line.lstrip().startswith('#'): continue + name, separator, value = line.partition('=') + if not separator or not re.fullmatch(r'[A-Z][A-Z0-9_]*', name) or name in values: + raise Refused() + values[name] = value + return values + + +def literal(values, name): + value = values.get(name, '') + if not re.fullmatch(r'[A-Za-z0-9]{32,128}', value): raise Refused() + return value + + +def run(args, payload=None): + completed = subprocess.run(args, input=payload, stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, timeout=60, check=False) + if completed.returncode or len(completed.stdout) > 2 * 1024 * 1024: raise Refused() + return completed.stdout + + +def decode_secret(raw, namespace, name): + obj = json.loads(raw) + if not isinstance(obj, dict) or obj.get('kind') != 'Secret' or obj.get('apiVersion') != 'v1': + raise Refused() + meta = obj.get('metadata') + if not isinstance(meta, dict) or meta.get('name') != name or meta.get('namespace') != namespace: + raise Refused() + if (not isinstance(meta.get('uid'), str) or not isinstance(meta.get('resourceVersion'), str) + or not re.fullmatch(r'[a-zA-Z0-9-]{1,128}', meta['uid']) + or not re.fullmatch(r'[0-9]{1,128}', meta['resourceVersion'])): + raise Refused() + if (meta.get('ownerReferences') or obj.get('immutable') + or obj.get('type') not in (('Opaque', 'kubernetes.io/basic-auth') + if name == 'net-kingdom-pg-privacyidea-app' else ('Opaque',))): + raise Refused() + annotations = meta.get('annotations', {}) + labels = meta.get('labels', {}) + if not isinstance(annotations, dict) or not isinstance(labels, dict): raise Refused() + if any(not isinstance(v, str) for v in (*annotations.values(), *labels.values())): raise Refused() + if any(k.startswith(('external-secrets.io/', 'reconcile.external-secrets.io/')) for k in labels): raise Refused() + if not isinstance(obj.get('data'), dict) or not obj['data']: raise Refused() + for key, value in obj['data'].items(): + if not isinstance(key, str) or not isinstance(value, str): raise Refused() + base64.b64decode(value, validate=True) + return obj + + +def read_key(parent, configured, existing): + if configured: + path = Path(configured) + if not path.is_absolute() or any(p.is_symlink() for p in (path, *path.parents)): + raise Refused() + fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) + try: + private(os.fstat(fd), True) + key = read_private(fd, path.name) + finally: + os.close(fd) + else: + try: + key = read_private(parent, 'oidc_private_key.pem') + except FileNotFoundError: + if existing is not None: + key = base64.b64decode(existing['data']['oidc_issuer_private_key'], validate=True) + else: + key = run(['openssl', 'genrsa', '2048']) + run(['openssl', 'rsa', '-check', '-noout'], key) + temp = '.issuer-' + secrets.token_hex(12) + fd = os.open(temp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600, dir_fd=parent) + try: + with os.fdopen(fd, 'wb') as file: + file.write(key); file.flush(); os.fsync(file.fileno()) + os.link(temp, 'oidc_private_key.pem', src_dir_fd=parent, dst_dir_fd=parent, follow_symlinks=False) + os.fsync(parent) + finally: + os.unlink(temp, dir_fd=parent) + run(['openssl', 'rsa', '-check', '-noout'], key) + return key.decode() + + +def service_values(component, values): + # Validate all required literals before any cluster call or local key creation. + if component == 'postgresql': + return {'username': 'privacyidea', 'password': literal(values['privacyidea'], 'PI_DB_PASSWORD')} + if component == 'privacyidea': + env = values['privacyidea']; password = literal(env, 'PI_DB_PASSWORD') + return {'PI_SECRET_KEY': literal(env, 'PI_SECRET_KEY'), 'PI_PEPPER': literal(env, 'PI_PEPPER'), + 'PI_SQLALCHEMY_DATABASE_URI': 'postgresql://privacyidea:' + password + + '@net-kingdom-pg-rw.databases.svc.cluster.local:5432/privacyidea_db'} + if component == 'lldap': + return {key: literal(values['lldap'], key) for key in ('LLDAP_JWT_SECRET', 'LLDAP_LDAP_USER_PASS')} + env = values['authelia'] + return {'jwt_secret': literal(env, 'AUTHELIA_JWT_SECRET'), + 'session_secret': literal(env, 'AUTHELIA_SESSION_SECRET'), + 'storage_encryption_key': literal(env, 'AUTHELIA_STORAGE_ENCRYPTION_KEY'), + 'oidc_hmac_secret': literal(env, 'AUTHELIA_OIDC_HMAC_SECRET'), + 'ldap_password': literal(values['lldap'], 'LLDAP_LDAP_USER_PASS')} + + +def publish(namespace, name, desired, old, allowed): + encoded = {k: base64.b64encode(v.encode()).decode() for k, v in desired.items()} + if old is None: + if allowed: raise Refused() # Rotation may not silently create missing state. + obj = {'apiVersion': 'v1', 'kind': 'Secret', 'type': 'Opaque', + 'metadata': {'namespace': namespace, 'name': name}, 'data': encoded} + result = run(['kubectl', 'create', '-o', 'json', '-f', '-'], json.dumps(obj).encode()) + expected = encoded; outcome = 'created' + else: + if not set(encoded).issubset(old['data']): raise Refused() + different = {k for k, value in encoded.items() if old['data'][k] != value} + if different - allowed: raise Refused() + annotations = old['metadata'].get('annotations', {}) + if not different and ANNOTATION not in annotations: return 'verified' + patch = [{'op': 'test', 'path': '/metadata/uid', 'value': old['metadata']['uid']}, + {'op': 'test', 'path': '/metadata/resourceVersion', 'value': old['metadata']['resourceVersion']}, + {'op': 'test', 'path': '/data', 'value': old['data']}] + patch.extend({'op': 'replace', 'path': '/data/' + key, 'value': encoded[key]} for key in sorted(different)) + if ANNOTATION in annotations: + patch.append({'op': 'remove', 'path': '/metadata/annotations/kubectl.kubernetes.io~1last-applied-configuration'}) + result = run(['kubectl', 'patch', 'secret', name, '-n', namespace, '--type=json', + '--patch-file=/dev/stdin', '-o', 'json'], json.dumps(patch).encode()) + expected = {**old['data'], **{k: encoded[k] for k in different}} + outcome = 'updated' + readback = decode_secret(result, namespace, name) + if readback['data'] != expected or ANNOTATION in readback['metadata'].get('annotations', {}): raise Refused() + if old is not None: + if readback['metadata']['uid'] != old['metadata']['uid'] or readback['type'] != old['type']: raise Refused() + expected_annotations = {k: v for k, v in old['metadata'].get('annotations', {}).items() if k != ANNOTATION} + if readback['metadata'].get('annotations', {}) != expected_annotations: raise Refused() + if readback['metadata'].get('labels', {}) != old['metadata'].get('labels', {}): raise Refused() + return outcome + + +def main(): + if len(sys.argv) not in (2, 3, 5): raise Refused() + component = sys.argv[1] + if component not in TARGETS: raise Refused() + namespace, name, inputs = TARGETS[component] + root = sys.argv[2] if len(sys.argv) >= 3 else Path(__file__).resolve().parent / 'secrets' + rotation = None + if len(sys.argv) == 5: + if sys.argv[3] != '--rotate-key' or sys.argv[4] not in ROTATIONS[component]: raise Refused() + rotation = ROTATIONS[component][sys.argv[4]] + with custody(root, inputs) as directories: + values = {c: read_env(directories[c]) for c in inputs} + desired = service_values(component, values) + if component == 'authelia': + import bcrypt + client = literal(values['authelia'], 'AUTHELIA_KEYCAPE_CLIENT_SECRET').encode() + if len(client) > 72: raise Refused() + raw = run(['kubectl', 'get', 'secret', name, '-n', namespace, '--ignore-not-found', '-o', 'json']) + old = decode_secret(raw, namespace, name) if raw.strip() else None + if rotation and old is None: raise Refused() + if component == 'authelia': + if old is None: + hashed = bcrypt.hashpw(client, bcrypt.gensalt(rounds=12)) + else: + hashed = base64.b64decode(old['data']['keycape_client_secret_hash'], validate=True) + if not bcrypt.checkpw(client, hashed): raise Refused() + desired['keycape_client_secret_hash'] = hashed.decode() + desired['oidc_issuer_private_key'] = read_key(directories['authelia'], + values['authelia'].get('AUTHELIA_OIDC_PRIVATE_KEY_FILE', ''), old) + outcome = publish(namespace, name, desired, old, {rotation} if rotation else set()) + print('Service Secret ' + outcome + '; native consumer acceptance remains required.') + + +if __name__ == '__main__': + try: + main() + except (Refused, OSError, ValueError, KeyError, TypeError, ImportError, subprocess.SubprocessError): + print('ERROR: service Secret operation refused or failed. Check private inputs, current ownership ' + 'and the named rotation through approved metadata-only diagnostics. A failed write has an ' + 'unverified remote outcome; reconcile before retry. Local recovery keys are retained.', file=sys.stderr) + sys.exit(1) diff --git a/sso-mfa/bootstrap/creds-apply.sh b/sso-mfa/bootstrap/creds-apply.sh index 7e81c82..66a0798 100755 --- a/sso-mfa/bootstrap/creds-apply.sh +++ b/sso-mfa/bootstrap/creds-apply.sh @@ -13,13 +13,20 @@ # 5. keycape SKIPPED — requires owner-established RPF-WP-0040 factor delivery. # create-secrets.sh is first-creation only, never live replacement. # -# After each successful component, creds-state.yaml is updated. +# After each successful component create/verification, creds-state.yaml is updated. +# Differing existing values require a separate named rotation; this is not a batch rotation. +set +x set -euo pipefail +umask 077 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -SECRETS_DIR="${1:-$SCRIPT_DIR/secrets}" +if (( $# > 1 )); then + echo "ERROR: expected one private secrets directory." >&2 + exit 1 +fi +SECRETS_DIR="$(realpath -sm -- "${1:-$SCRIPT_DIR/secrets}")" STATE_FILE="$SCRIPT_DIR/creds-state.yaml" K8S_DIR="$REPO_ROOT/sso-mfa/k8s" @@ -44,7 +51,7 @@ fi echo "=== creds-apply — net-kingdom SSO/MFA secrets ===" echo "Secrets dir : $SECRETS_DIR" -echo "Cluster : $(kubectl config current-context 2>/dev/null || echo '(unknown)')" +echo "Cluster connectivity checked; use only the context admitted by the current procedure." echo "" # ── Helper: run a component's create-secrets.sh ─────────────────────────────── @@ -62,7 +69,11 @@ run_component() { exit 1 fi - (cd "$script_dir" && bash create-secrets.sh "$SECRETS_DIR") + if ! (cd "$script_dir" && bash create-secrets.sh "$SECRETS_DIR") >/dev/null 2>&1; then + echo "ERROR: component create/verification failed; earlier components may already be applied." >&2 + echo "Use metadata-only reconciliation before retry; no native acceptance is claimed." >&2 + exit 1 + fi # Update state file if [[ -f "$STATE_FILE" ]]; then diff --git a/sso-mfa/bootstrap/creds-rotate.sh b/sso-mfa/bootstrap/creds-rotate.sh index 4e79f14..f03259f 100755 --- a/sso-mfa/bootstrap/creds-rotate.sh +++ b/sso-mfa/bootstrap/creds-rotate.sh @@ -134,7 +134,7 @@ PI_SECRET_KEY) echo "" store_value privacyidea echo " 1. Updating K8s Secret privacyidea-config (namespace: mfa)..." - (cd "$K8S_DIR/privacyidea" && quiet bash create-secrets.sh "$SECRETS_DIR") + (cd "$K8S_DIR/privacyidea" && quiet bash create-secrets.sh "$SECRETS_DIR" --rotate-key "$SECRET") echo " 2. Restarting privacyIDEA pod..." quiet kubectl rollout restart deployment privacyidea -n mfa quiet kubectl rollout status deployment privacyidea -n mfa @@ -160,10 +160,10 @@ PI_PEPPER) PI_DB_PASSWORD) header echo "" - echo "Impact: privacyIDEA database password — must be updated atomically in:" + echo "Impact: privacyIDEA database password — coordinated updates are required in:" echo " 1. PostgreSQL (CNPG) — ALTER USER privacyidea WITH PASSWORD '...';" - echo " 2. K8s Secret privacyidea-config (PI_SQLALCHEMY_DATABASE_URI)" - echo " Privacyidea pod must be restarted after both are updated." + echo " 2. CNPG application Secret and privacyidea-config (PI_SQLALCHEMY_DATABASE_URI)" + echo " These are separate steps; preserve the candidate and reconcile any partial failure." echo "" prepare_value privacyidea b64 confirm "Apply rotation?" || exit 0 @@ -183,8 +183,9 @@ PI_DB_PASSWORD) quiet kubectl exec -i -n databases "$PG_POD" -- psql -U postgres -v ON_ERROR_STOP=1 echo " PostgreSQL password update command succeeded." echo "" - echo " 2. Updating K8s Secret privacyidea-config..." - (cd "$K8S_DIR/privacyidea" && quiet bash create-secrets.sh "$SECRETS_DIR") + echo " 2. Updating PostgreSQL application credential Secret and privacyidea-config..." + (cd "$K8S_DIR/postgresql" && quiet bash create-secrets.sh "$SECRETS_DIR" --rotate-key "$SECRET") + (cd "$K8S_DIR/privacyidea" && quiet bash create-secrets.sh "$SECRETS_DIR" --rotate-key "$SECRET") echo "" echo " 3. Restarting privacyIDEA pod..." quiet kubectl rollout restart deployment privacyidea -n mfa @@ -202,7 +203,7 @@ LLDAP_JWT_SECRET) confirm "Apply rotation?" || exit 0 store_value lldap echo " Updating K8s Secret lldap-secrets (namespace: sso)..." - (cd "$K8S_DIR/lldap" && quiet bash create-secrets.sh "$SECRETS_DIR") + (cd "$K8S_DIR/lldap" && quiet bash create-secrets.sh "$SECRETS_DIR" --rotate-key "$SECRET") echo " Restarting LLDAP pod..." quiet kubectl rollout restart deployment lldap -n sso quiet kubectl rollout status deployment lldap -n sso @@ -211,38 +212,9 @@ LLDAP_JWT_SECRET) ;; LLDAP_LDAP_USER_PASS) - if [[ "$NON_INTERACTIVE" == true || ! -t 0 ]]; then - echo "ERROR: LDAP password rotation requires an attended manual server update; automation is unavailable." >&2 - exit 1 - fi - header - echo "" - echo "Impact: LLDAP admin + LDAP bind password — used by Authelia and KeyCape." - echo " All three must be updated atomically:" - echo " 1. LLDAP admin password (via LLDAP web UI or API)" - echo " 2. Authelia K8s Secret (ldap_password)" - echo " 3. KeyCape K8s Secret (config.yaml lldap.bindPW)" - echo " Restart all three pods after updating." - echo "" - prepare_value lldap b64 - confirm "Apply rotation?" || exit 0 - store_value lldap - echo " 1. Manually updating the LLDAP admin password..." - echo " WARN: Automated LLDAP password update not implemented." - echo " Use the candidate in the private lldap/secrets.env file to update the server." - echo " If cancelled, reconcile that local candidate with the unchanged server before reuse." - confirm "Confirm you have updated the LLDAP admin password?" || { echo "Aborting; local candidate retained for recovery."; exit 1; } - echo " 2. Updating Authelia secrets..." - (cd "$K8S_DIR/authelia" && quiet bash create-secrets.sh "$SECRETS_DIR") - echo " 3. Updating KeyCape secrets..." - (cd "$K8S_DIR/keycape" && quiet bash create-secrets.sh "$SECRETS_DIR") - echo " 4. Restarting Authelia and KeyCape pods..." - quiet kubectl rollout restart deployment authelia -n sso - quiet kubectl rollout restart deployment keycape -n sso - quiet kubectl rollout status deployment authelia -n sso - quiet kubectl rollout status deployment keycape -n sso - update_last_rotated "LLDAP_LDAP_USER_PASS" - post_rotation_reminder + echo "ERROR: coordinated LDAP/Authelia/KeyCape rotation requires an owner-reviewed procedure with rollback and native acceptance (NK-WP-0049-T02)." >&2 + echo "The legacy manual-server/full-config-replacement path is retired before any credential changes." >&2 + exit 1 ;; AUTHELIA_SESSION_SECRET) @@ -254,7 +226,7 @@ AUTHELIA_SESSION_SECRET) prepare_value authelia hex confirm "Apply rotation?" || exit 0 store_value authelia - (cd "$K8S_DIR/authelia" && quiet bash create-secrets.sh "$SECRETS_DIR") + (cd "$K8S_DIR/authelia" && quiet bash create-secrets.sh "$SECRETS_DIR" --rotate-key "$SECRET") quiet kubectl rollout restart deployment authelia -n sso quiet kubectl rollout status deployment authelia -n sso update_last_rotated "AUTHELIA_SESSION_SECRET" diff --git a/sso-mfa/k8s/authelia/README.md b/sso-mfa/k8s/authelia/README.md index bb5f012..fa7304b 100644 --- a/sso-mfa/k8s/authelia/README.md +++ b/sso-mfa/k8s/authelia/README.md @@ -92,9 +92,13 @@ apps — the edge already calls Authelia `/api/verify`. | | `keycape_client_secret_hash` | Bcrypt hash of `AUTHELIA_KEYCAPE_CLIENT_SECRET` | `create-secrets.sh` reads plaintext values from `secrets/authelia/secrets.env` and -`secrets/lldap/secrets.env`. It generates the bcrypt hash on the fly (requires -`python3+bcrypt` or `apache2-utils`). The RSA OIDC private key is generated -automatically if `AUTHELIA_OIDC_PRIVATE_KEY_FILE` is not set. +`secrets/lldap/secrets.env` as private literal input (requires Python 3 + bcrypt). +First creation stores a new signing key privately in `authelia/oidc_private_key.pem` +unless an explicit private absolute key path is supplied. Existing deployments +preserve their issuer key and matching bcrypt hash. Normal reapplication verifies +matching values; session rotation must explicitly select `AUTHELIA_SESSION_SECRET`. +See [service Secret custody and rotation](../../bootstrap/README.md#service-secret-creation-and-explicit-rotation) +for conflict handling, recovery custody and native acceptance limits. ## Storage diff --git a/sso-mfa/k8s/authelia/create-secrets.sh b/sso-mfa/k8s/authelia/create-secrets.sh index 9be7fa8..393fbae 100644 --- a/sso-mfa/k8s/authelia/create-secrets.sh +++ b/sso-mfa/k8s/authelia/create-secrets.sh @@ -1,113 +1,7 @@ #!/usr/bin/env bash -# create-secrets.sh — create the authelia-secrets K8s Secret -# -# Usage: -# ./create-secrets.sh [secrets-dir] -# -# is the output directory from sso-mfa/bootstrap/gen-secrets.sh -# (default: ../../bootstrap/secrets). -# -# Creates ONE Secret in the sso namespace: -# authelia-secrets — all Authelia sensitive values as named files: -# jwt_secret ← AUTHELIA_JWT_SECRET_FILE -# session_secret ← AUTHELIA_SESSION_SECRET_FILE -# storage_encryption_key ← AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE -# ldap_password ← AUTHELIA_AUTHENTICATION_BACKEND_LDAP_PASSWORD_FILE -# oidc_hmac_secret ← AUTHELIA_IDENTITY_PROVIDERS_OIDC_HMAC_SECRET_FILE -# oidc_issuer_private_key ← AUTHELIA_IDENTITY_PROVIDERS_OIDC_ISSUER_PRIVATE_KEY_FILE -# keycape_client_secret_hash ← AUTHELIA_IDENTITY_PROVIDERS_OIDC_CLIENTS_0_SECRET_FILE -# -# The keycape_client_secret_hash is the bcrypt hash of AUTHELIA_KEYCAPE_CLIENT_SECRET -# from secrets/authelia/secrets.env. The plaintext is stored in KeyCape's secret. -# -# Requires: kubectl, openssl, python3 (for bcrypt hash generation) - +# Private creation or exact-match verification; rotations require --rotate-key. +set +x set -euo pipefail - -SECRETS_DIR="${1:-../../bootstrap/secrets}" -AUTHELIA_ENV="$SECRETS_DIR/authelia/secrets.env" -LLDAP_ENV="$SECRETS_DIR/lldap/secrets.env" - -for f in "$AUTHELIA_ENV" "$LLDAP_ENV"; do - if [[ ! -f "$f" ]]; then - echo "ERROR: $f not found" >&2 - echo "Run sso-mfa/bootstrap/gen-secrets.sh first." >&2 - exit 1 - fi -done - -read_env() { bash -c "source '$1' 2>/dev/null; echo \${$2}"; } - -AUTHELIA_JWT_SECRET=$(read_env "$AUTHELIA_ENV" AUTHELIA_JWT_SECRET) -AUTHELIA_SESSION_SECRET=$(read_env "$AUTHELIA_ENV" AUTHELIA_SESSION_SECRET) -AUTHELIA_STORAGE_ENCRYPTION_KEY=$(read_env "$AUTHELIA_ENV" AUTHELIA_STORAGE_ENCRYPTION_KEY) -AUTHELIA_OIDC_HMAC_SECRET=$(read_env "$AUTHELIA_ENV" AUTHELIA_OIDC_HMAC_SECRET) -AUTHELIA_OIDC_PRIVATE_KEY=$(read_env "$AUTHELIA_ENV" AUTHELIA_OIDC_PRIVATE_KEY_FILE) -AUTHELIA_KEYCAPE_CLIENT_SECRET=$(read_env "$AUTHELIA_ENV" AUTHELIA_KEYCAPE_CLIENT_SECRET) -LLDAP_LDAP_USER_PASS=$(read_env "$LLDAP_ENV" LLDAP_LDAP_USER_PASS) - -# Validate required values -REQUIRED=(AUTHELIA_JWT_SECRET AUTHELIA_SESSION_SECRET AUTHELIA_STORAGE_ENCRYPTION_KEY - AUTHELIA_OIDC_HMAC_SECRET AUTHELIA_KEYCAPE_CLIENT_SECRET LLDAP_LDAP_USER_PASS) -for var in "${REQUIRED[@]}"; do - if [[ -z "${!var}" ]]; then - echo "ERROR: $var is empty — re-run gen-secrets.sh" >&2 - exit 1 - fi -done - -# The OIDC issuer private key is stored as a file path in secrets.env. -# Read the actual PEM content from the path referenced there. -AUTHELIA_OIDC_PRIVATE_KEY_PATH=$(read_env "$AUTHELIA_ENV" AUTHELIA_OIDC_PRIVATE_KEY_FILE) -if [[ -z "$AUTHELIA_OIDC_PRIVATE_KEY_PATH" ]]; then - # Fall back: generate a new RSA key on the fly (dev only) - echo "WARN: AUTHELIA_OIDC_PRIVATE_KEY_FILE not set — generating a new RSA-2048 key." - echo " Store the generated key in KeePassXC as net-kingdom/Authelia/oidc-private-key." - TMP_KEY=$(mktemp) - openssl genrsa -out "$TMP_KEY" 2048 2>/dev/null - AUTHELIA_OIDC_PRIVATE_KEY_CONTENT=$(cat "$TMP_KEY") - shred -u "$TMP_KEY" -elif [[ -f "$AUTHELIA_OIDC_PRIVATE_KEY_PATH" ]]; then - AUTHELIA_OIDC_PRIVATE_KEY_CONTENT=$(cat "$AUTHELIA_OIDC_PRIVATE_KEY_PATH") -else - echo "ERROR: OIDC private key file not found: $AUTHELIA_OIDC_PRIVATE_KEY_PATH" >&2 - exit 1 -fi - -# Hash the Authelia-KeyCape client secret with bcrypt (cost 12). -# Authelia requires the bcrypt hash for OIDC client secrets. -echo "Hashing KeyCape client secret with bcrypt (this may take a moment)..." -if command -v python3 &>/dev/null && python3 -c "import bcrypt" 2>/dev/null; then - KEYCAPE_CLIENT_SECRET_HASH=$(python3 -c " -import bcrypt, sys -pw = sys.argv[1].encode() -print(bcrypt.hashpw(pw, bcrypt.gensalt(rounds=12)).decode()) -" "$AUTHELIA_KEYCAPE_CLIENT_SECRET") -elif command -v htpasswd &>/dev/null; then - KEYCAPE_CLIENT_SECRET_HASH=$(htpasswd -nbBC 12 "" "$AUTHELIA_KEYCAPE_CLIENT_SECRET" | cut -d: -f2) -else - echo "ERROR: bcrypt hash generation requires python3+bcrypt or apache2-utils (htpasswd)" >&2 - echo " Install: pip3 install bcrypt OR apt install apache2-utils" >&2 - exit 1 -fi - -echo "Creating K8s Secret: authelia-secrets (namespace: sso)" -kubectl create secret generic authelia-secrets \ - --namespace=sso \ - --from-literal=jwt_secret="$AUTHELIA_JWT_SECRET" \ - --from-literal=session_secret="$AUTHELIA_SESSION_SECRET" \ - --from-literal=storage_encryption_key="$AUTHELIA_STORAGE_ENCRYPTION_KEY" \ - --from-literal=ldap_password="$LLDAP_LDAP_USER_PASS" \ - --from-literal=oidc_hmac_secret="$AUTHELIA_OIDC_HMAC_SECRET" \ - --from-literal=oidc_issuer_private_key="$AUTHELIA_OIDC_PRIVATE_KEY_CONTENT" \ - --from-literal=keycape_client_secret_hash="$KEYCAPE_CLIENT_SECRET_HASH" \ - --dry-run=client -o yaml | kubectl apply -f - - -echo "" -echo "Done. Secret authelia-secrets created in namespace: sso" -echo "" -echo "Next:" -echo " Apply deployment.yaml, then ingress.yaml." -echo " The plaintext Authelia-KeyCape client secret is in secrets/authelia/secrets.env" -echo " as AUTHELIA_KEYCAPE_CLIENT_SECRET. This value goes into keycape/create-secrets.sh" -echo " as the authelia.clientSecret in the KeyCape config." +umask 077 +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +exec python3 "$SCRIPT_DIR/../../bootstrap/apply-service-secrets.py" authelia "$@" diff --git a/sso-mfa/k8s/lldap/create-secrets.sh b/sso-mfa/k8s/lldap/create-secrets.sh index 132a5e0..d7fa9cc 100644 --- a/sso-mfa/k8s/lldap/create-secrets.sh +++ b/sso-mfa/k8s/lldap/create-secrets.sh @@ -1,57 +1,7 @@ #!/usr/bin/env bash -# create-secrets.sh — create the lldap-secrets K8s Secret -# -# Usage: -# ./create-secrets.sh [secrets-dir] -# -# is the output directory from sso-mfa/bootstrap/gen-secrets.sh -# (default: ../../bootstrap/secrets). -# -# Creates ONE Secret in the sso namespace: -# lldap-secrets — LLDAP_JWT_SECRET, LLDAP_LDAP_USER_PASS -# -# LLDAP_LDAP_USER_PASS is also used as the LDAP bind password -# by Authelia (authelia/create-secrets.sh) and KeyCape (keycape/create-secrets.sh). -# All three read the same value from secrets/lldap/secrets.env. - +# Private creation or exact-match verification; rotations require --rotate-key. +set +x set -euo pipefail - -SECRETS_DIR="${1:-../../bootstrap/secrets}" -LLDAP_ENV="$SECRETS_DIR/lldap/secrets.env" - -if [[ ! -d "$SECRETS_DIR" ]]; then - echo "ERROR: secrets directory not found: $SECRETS_DIR" >&2 - echo "Run sso-mfa/bootstrap/gen-secrets.sh first." >&2 - exit 1 -fi - -if [[ ! -f "$LLDAP_ENV" ]]; then - echo "ERROR: $LLDAP_ENV not found" >&2 - echo "If you ran gen-secrets.sh before the KeyCape migration, re-run it to add LLDAP secrets." >&2 - exit 1 -fi - -LLDAP_JWT_SECRET=$(bash -c "source '$LLDAP_ENV' 2>/dev/null; echo \$LLDAP_JWT_SECRET") -LLDAP_LDAP_USER_PASS=$(bash -c "source '$LLDAP_ENV' 2>/dev/null; echo \$LLDAP_LDAP_USER_PASS") - -if [[ -z "$LLDAP_JWT_SECRET" || -z "$LLDAP_LDAP_USER_PASS" ]]; then - echo "ERROR: could not read LLDAP_JWT_SECRET or LLDAP_LDAP_USER_PASS from $LLDAP_ENV" >&2 - exit 1 -fi - -echo "Creating K8s Secret: lldap-secrets (namespace: sso)" -kubectl create secret generic lldap-secrets \ - --namespace=sso \ - --from-literal=LLDAP_JWT_SECRET="$LLDAP_JWT_SECRET" \ - --from-literal=LLDAP_LDAP_USER_PASS="$LLDAP_LDAP_USER_PASS" \ - --dry-run=client -o yaml | kubectl apply -f - - -echo "" -echo "Done. Secret lldap-secrets created in namespace: sso" -echo "" -echo "Next:" -echo " Apply manifests (see README.md apply order)." -echo " After LLDAP is Running, seed groups:" -echo " ./bootstrap-users.sh" -echo " Groups: net-kingdom-users, net-kingdom-admins, activity-core-operators" -echo " Membership runbook: OPERATOR-GROUPS.md" +umask 077 +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +exec python3 "$SCRIPT_DIR/../../bootstrap/apply-service-secrets.py" lldap "$@" diff --git a/sso-mfa/k8s/postgresql/README.md b/sso-mfa/k8s/postgresql/README.md index 6c19846..98d60e1 100644 --- a/sso-mfa/k8s/postgresql/README.md +++ b/sso-mfa/k8s/postgresql/README.md @@ -141,5 +141,7 @@ chmod +x ../verify-t03.sh Do not rename the cluster without also updating netpol-databases.yaml. - `instances: 1` is intentional for dev/staging. Change to 3 before ThreePhoenix HA production deployment (requires at least 3 schedulable nodes). -- Password rotation: update the K8s Secret values and CNPG's managed.roles reconciler will - apply the change at the next reconciliation cycle (within seconds). +- Password rotation uses the explicit `PI_DB_PASSWORD` selector under the reviewed + procedure. CNPG's managed role and the privacyIDEA consumer URI must converge + on the same candidate; verify actual reconciliation and authentication before + calling it complete. See [service Secret custody](../../bootstrap/README.md#service-secret-creation-and-explicit-rotation). diff --git a/sso-mfa/k8s/postgresql/create-secrets.sh b/sso-mfa/k8s/postgresql/create-secrets.sh index ca4645b..793bda7 100755 --- a/sso-mfa/k8s/postgresql/create-secrets.sh +++ b/sso-mfa/k8s/postgresql/create-secrets.sh @@ -1,57 +1,7 @@ #!/usr/bin/env bash -# create-secrets.sh — create K8s Secrets for PostgreSQL from gen-secrets.sh output -# -# Usage: -# ./create-secrets.sh -# -# is the output directory produced by sso-mfa/bootstrap/gen-secrets.sh -# (default: ../../bootstrap/secrets). -# -# Creates one K8s Secret in the databases namespace: -# net-kingdom-pg-privacyidea-app — privacyIDEA DB credentials -# -# Note: net-kingdom-pg-keycloak-app removed — Keycloak replaced by Authelia+LLDAP+KeyCape (T05). -# -# These secrets must exist before applying cluster.yaml. -# Re-run this script whenever you rotate passwords in KeePassXC / gen-secrets.sh. - +# Private creation or exact-match verification; rotations require --rotate-key. +set +x set -euo pipefail - -SECRETS_DIR="${1:-../../bootstrap/secrets}" - -if [[ ! -d "$SECRETS_DIR" ]]; then - echo "ERROR: secrets directory not found: $SECRETS_DIR" >&2 - echo "Run sso-mfa/bootstrap/gen-secrets.sh first, then re-run this script." >&2 - exit 1 -fi - -PI_SECRETS="$SECRETS_DIR/privacyidea/secrets.env" - -if [[ ! -f "$PI_SECRETS" ]]; then - echo "ERROR: $PI_SECRETS not found" >&2 - exit 1 -fi - -# Source the generated env file (KEY=VALUE pairs, no export) -# Use a subshell to avoid polluting the current environment. -PI_DB_PASS=$(bash -c "source $PI_SECRETS 2>/dev/null; echo \$PI_DB_PASSWORD") - -if [[ -z "$PI_DB_PASS" ]]; then - echo "ERROR: could not read PI_DB_PASSWORD from $PI_SECRETS" >&2 - echo "Check that gen-secrets.sh ran successfully and the file is intact." >&2 - exit 1 -fi - -echo "Creating K8s Secret: net-kingdom-pg-privacyidea-app" -kubectl create secret generic net-kingdom-pg-privacyidea-app \ - --namespace=databases \ - --from-literal=username=privacyidea \ - --from-literal=password="$PI_DB_PASS" \ - --dry-run=client -o yaml | kubectl apply -f - - -echo "" -echo "Done. Secret created in namespace: databases" -echo "" -echo "Verify:" -echo " kubectl get secrets -n databases" -echo " kubectl describe secret net-kingdom-pg-privacyidea-app -n databases" +umask 077 +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +exec python3 "$SCRIPT_DIR/../../bootstrap/apply-service-secrets.py" postgresql "$@" diff --git a/sso-mfa/k8s/privacyidea/README.md b/sso-mfa/k8s/privacyidea/README.md index 5ef5d3b..864c716 100644 --- a/sso-mfa/k8s/privacyidea/README.md +++ b/sso-mfa/k8s/privacyidea/README.md @@ -57,7 +57,7 @@ kubectl logs -n mfa -l app.kubernetes.io/name=privacyidea --previous Common causes: - `privacyidea-config` Secret missing → run `create-secrets.sh` - PostgreSQL not reachable → verify T03, check NetworkPolicies -- Wrong DB password → re-run `create-secrets.sh` with corrected secrets +- Wrong DB password → reconcile the database, CNPG application Secret and privacyIDEA URI through the reviewed named `PI_DB_PASSWORD` rotation; ordinary reapplication refuses differing values. See [service Secret custody](../../bootstrap/README.md#service-secret-creation-and-explicit-rotation). --- diff --git a/sso-mfa/k8s/privacyidea/create-secrets.sh b/sso-mfa/k8s/privacyidea/create-secrets.sh index 59eb21d..2dd5f96 100755 --- a/sso-mfa/k8s/privacyidea/create-secrets.sh +++ b/sso-mfa/k8s/privacyidea/create-secrets.sh @@ -1,65 +1,7 @@ #!/usr/bin/env bash -# create-secrets.sh — create the privacyidea-config K8s Secret -# -# Usage: -# ./create-secrets.sh [secrets-dir] -# -# is the output directory from sso-mfa/bootstrap/gen-secrets.sh -# (default: ../../bootstrap/secrets). -# -# Creates ONE Secret in the mfa namespace: -# privacyidea-config — PI_SECRET_KEY, PI_PEPPER, PI_SQLALCHEMY_DATABASE_URI -# -# This secret must exist before applying deployment.yaml. -# -# The enckey and auditkey Secrets (privacyidea-enckey, privacyidea-auditkeys) -# are created separately by enckey-bootstrap.sh AFTER the first pod start, -# because those keys are auto-generated by the container on first run. -# -# Re-run this script if you rotate PI_SECRET_KEY or PI_PEPPER in KeePassXC. - +# Private creation or exact-match verification; rotations require --rotate-key. +set +x set -euo pipefail - -SECRETS_DIR="${1:-../../bootstrap/secrets}" -PI_ENV="$SECRETS_DIR/privacyidea/secrets.env" - -if [[ ! -d "$SECRETS_DIR" ]]; then - echo "ERROR: secrets directory not found: $SECRETS_DIR" >&2 - echo "Run sso-mfa/bootstrap/gen-secrets.sh first." >&2 - exit 1 -fi - -if [[ ! -f "$PI_ENV" ]]; then - echo "ERROR: $PI_ENV not found" >&2 - exit 1 -fi - -# Read values from the generated env file in a subshell to avoid polluting env. -PI_SECRET_KEY=$(bash -c "source '$PI_ENV' 2>/dev/null; echo \$PI_SECRET_KEY") -PI_PEPPER=$(bash -c "source '$PI_ENV' 2>/dev/null; echo \$PI_PEPPER") -PI_DB_PASSWORD=$(bash -c "source '$PI_ENV' 2>/dev/null; echo \$PI_DB_PASSWORD") - -if [[ -z "$PI_SECRET_KEY" || -z "$PI_PEPPER" || -z "$PI_DB_PASSWORD" ]]; then - echo "ERROR: could not read PI_SECRET_KEY, PI_PEPPER, or PI_DB_PASSWORD from $PI_ENV" >&2 - echo "Check that gen-secrets.sh ran successfully." >&2 - exit 1 -fi - -# Construct the SQLAlchemy database URI. -# CloudNativePG read-write service: net-kingdom-pg-rw.databases.svc.cluster.local -PI_DB_URI="postgresql://privacyidea:${PI_DB_PASSWORD}@net-kingdom-pg-rw.databases.svc.cluster.local:5432/privacyidea_db" - -echo "Creating K8s Secret: privacyidea-config (namespace: mfa)" -kubectl create secret generic privacyidea-config \ - --namespace=mfa \ - --from-literal=PI_SECRET_KEY="$PI_SECRET_KEY" \ - --from-literal=PI_PEPPER="$PI_PEPPER" \ - --from-literal=PI_SQLALCHEMY_DATABASE_URI="$PI_DB_URI" \ - --dry-run=client -o yaml | kubectl apply -f - - -echo "" -echo "Done. Secret privacyidea-config created in namespace: mfa" -echo "" -echo "Next:" -echo " Apply manifests (see README.md apply order)." -echo " After the pod is Running, run: ./enckey-bootstrap.sh" +umask 077 +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +exec python3 "$SCRIPT_DIR/../../bootstrap/apply-service-secrets.py" privacyidea "$@" diff --git a/tests/test_rotation_output.py b/tests/test_rotation_output.py index c846ad3..6706e9f 100644 --- a/tests/test_rotation_output.py +++ b/tests/test_rotation_output.py @@ -26,7 +26,7 @@ def rotation(tmp_path): shutil.copyfile(ROOT / 'Makefile', repo / 'Makefile') (boot / 'creds-state.yaml').write_text('version: 2\n') secrets = tmp_path / 'private'; secrets.mkdir(mode=0o700) - for component in ('privacyidea', 'lldap', 'authelia', 'keycape', 'breakglass'): + for component in ('postgresql', 'privacyidea', 'lldap', 'authelia', 'keycape', 'breakglass'): directory = secrets / component; directory.mkdir(mode=0o700) entries = [f'{key}=old-value' for key, value in CASES.items() if value == component] if component == 'lldap': entries.append('LLDAP_LDAP_USER_PASS=old-value') @@ -34,7 +34,10 @@ def rotation(tmp_path): env_file.write_text('\n'.join(entries) + '\n'); env_file.chmod(0o600) helper = repo / f'sso-mfa/k8s/{component}'; helper.mkdir(parents=True) (helper / 'create-secrets.sh').write_text( - '#!/bin/bash\nprintf "%s\\n" "$SYNTHETIC" >&2\nexit "${HELPER_FAIL:-0}"\n') + '#!/bin/bash\npython3 - "$@" <<\'PYTRACE\'\n' + 'import json,os,sys\nfrom pathlib import Path\n' + 'with open(os.environ["HELPERS"],"a") as f:f.write(json.dumps({"component":Path.cwd().name,"args":sys.argv[1:]})+"\\n")\n' + 'PYTRACE\nprintf "%s\\n" "$SYNTHETIC" >&2\nexit "${HELPER_FAIL:-0}"\n') binary = tmp_path / 'bin'; binary.mkdir() (binary / 'openssl').write_text('#!/bin/sh\nprintf "%s\\n" "$SYNTHETIC"\n') (binary / 'kubectl').write_text('''#!/usr/bin/python3 @@ -52,7 +55,7 @@ else: for command in binary.iterdir(): command.chmod(0o700) env = {'PATH': str(binary) + ':' + os.environ['PATH'], 'HOME': str(tmp_path), 'KUBECONFIG': str(tmp_path / 'nonexistent'), 'CALLS': str(tmp_path / 'calls'), - 'SYNTHETIC': SYNTHETIC} + 'SYNTHETIC': SYNTHETIC, 'HELPERS': str(tmp_path / 'helpers')} return repo, boot, secrets, env @@ -185,7 +188,7 @@ def test_declining_rotation_preserves_custody_file_and_cluster(rotation): assert bool(SYNTHETIC[:40] in result.stdout + result.stderr) is False -@pytest.mark.parametrize('key',['AUTHELIA_KEYCAPE_CLIENT_SECRET','KEYCAPE_RSA_KEY']) +@pytest.mark.parametrize('key',['AUTHELIA_KEYCAPE_CLIENT_SECRET','KEYCAPE_RSA_KEY','LLDAP_LDAP_USER_PASS']) @pytest.mark.parametrize('attended',[False,True]) def test_retired_keycape_replacement_refuses_before_any_credential_change(rotation,key,attended): repo,boot,secrets,env=rotation @@ -199,3 +202,25 @@ def test_retired_keycape_replacement_refuses_before_any_credential_change(rotati assert {p:p.read_bytes() for p in secrets.rglob('*') if p.is_file()}==before assert 'last_rotated_' not in (boot/'creds-state.yaml').read_text() assert 'NK-WP-0049-T02' in result.stderr + + +@pytest.mark.parametrize('key,components',[ + ('PI_SECRET_KEY',['privacyidea']), + ('PI_DB_PASSWORD',['postgresql','privacyidea']), + ('LLDAP_JWT_SECRET',['lldap']), + ('AUTHELIA_SESSION_SECRET',['authelia']), +]) +def test_rotation_forwards_exact_selector_to_every_secret_consumer(rotation,key,components): + result=run(rotation,key) + assert result.returncode==0 + observed=[json.loads(line) for line in Path(rotation[3]['HELPERS']).read_text().splitlines()] + assert observed==[{'component':component,'args':[str(rotation[2]),'--rotate-key',key]} for component in components] + + +def test_database_secret_failure_stops_before_other_consumer_or_restart(rotation): + result=run(rotation,'PI_DB_PASSWORD',HELPER_FAIL='1') + assert result.returncode!=0 + assert [json.loads(line)['component'] for line in Path(rotation[3]['HELPERS']).read_text().splitlines()]==['postgresql'] + assert [call['args'][0] for call in calls(rotation)]==['get','exec'] + assert 'last_rotated_' not in (rotation[1]/'creds-state.yaml').read_text() + assert 'PI_DB_PASSWORD='+SYNTHETIC[:40] in (rotation[2]/'privacyidea/secrets.env').read_text() diff --git a/tests/test_service_apply_caller.py b/tests/test_service_apply_caller.py new file mode 100644 index 0000000..3d83107 --- /dev/null +++ b/tests/test_service_apply_caller.py @@ -0,0 +1,41 @@ +"""Batch transport tests use synthetic noisy helpers and no real cluster.""" +import json +import os +from pathlib import Path +import shutil +import subprocess + +import pytest + +ROOT=Path(__file__).resolve().parents[1] +MARKER='SyntheticChildOutputNeverInParentTranscript' + + +@pytest.mark.parametrize('failed',[None,'authelia']) +def test_batch_captures_children_and_only_records_successful_components(tmp_path,failed): + repo=tmp_path/'repo';boot=repo/'sso-mfa/bootstrap';boot.mkdir(parents=True) + shutil.copyfile(ROOT/'sso-mfa/bootstrap/creds-apply.sh',boot/'creds-apply.sh') + state=boot/'creds-state.yaml';state.write_text('secrets_applied:\n postgres: false\n lldap: false\n authelia: false\n privacyidea: false\n') + private=tmp_path/'private with spaces';private.mkdir(mode=0o700) + for name in ('lldap','privacyidea'): + p=private/name;p.mkdir(mode=0o700);(p/'secrets.env').write_text('# no actual value needed for transport fixture\n') + binary=tmp_path/'bin';binary.mkdir();(binary/'kubectl').write_text('#!/bin/sh\nprintf "%s\\n" "$MARKER"\nexit 0\n');(binary/'kubectl').chmod(0o700) + for component in ('postgresql','lldap','authelia','privacyidea'): + p=repo/f'sso-mfa/k8s/{component}';p.mkdir(parents=True) + (p/'create-secrets.sh').write_text('''#!/bin/bash +printf '%s\\n' "$MARKER" +printf '%s\\n' "$MARKER" >&2 +python3 - "$1" <<'PYTHON' +import json,os,sys +with open(os.environ['ARGS'],'a') as file:file.write(json.dumps(sys.argv[1:])+'\\n') +PYTHON +'''+ ('exit 1\n' if component==failed else 'exit 0\n')) + env={'PATH':str(binary)+':'+os.environ['PATH'],'HOME':str(tmp_path),'KUBECONFIG':str(tmp_path/'unused'),'ARGS':str(tmp_path/'args'),'MARKER':MARKER} + result=subprocess.run(['bash','-x',str(boot/'creds-apply.sh'),private.name],cwd=tmp_path,env=env,capture_output=True,text=True,timeout=10) + assert (result.returncode==0)==(failed is None) + assert MARKER not in result.stdout+result.stderr + args=[json.loads(line) for line in Path(env['ARGS']).read_text().splitlines()] + assert args==[[str(private)]]*(4 if failed is None else 3) + recorded=state.read_text();assert 'postgres: true' in recorded and 'lldap: true' in recorded + assert ('authelia: true' in recorded)==(failed is None) + assert ('privacyidea: true' in recorded)==(failed is None) diff --git a/tests/test_service_secret_custody.py b/tests/test_service_secret_custody.py new file mode 100644 index 0000000..ebceb63 --- /dev/null +++ b/tests/test_service_secret_custody.py @@ -0,0 +1,276 @@ +"""Service helpers against synthetic files and a recording Kubernetes API stub.""" +import base64 +import json +import os +from pathlib import Path +import subprocess + +import bcrypt +import pytest + +ROOT = Path(__file__).resolve().parents[1] +PASSWORD = 'SyntheticServiceCredential12345678901234567890' +CHANGED = 'ReplacementSyntheticCredential12345678901234567890' +VALUES = { + 'privacyidea': ['PI_SECRET_KEY', 'PI_PEPPER', 'PI_DB_PASSWORD'], + 'lldap': ['LLDAP_JWT_SECRET', 'LLDAP_LDAP_USER_PASS'], + 'authelia': ['AUTHELIA_JWT_SECRET', 'AUTHELIA_SESSION_SECRET', 'AUTHELIA_STORAGE_ENCRYPTION_KEY', + 'AUTHELIA_OIDC_HMAC_SECRET', 'AUTHELIA_KEYCAPE_CLIENT_SECRET'], +} +COMPONENTS = ['postgresql', 'privacyidea', 'lldap', 'authelia'] +ROTATIONS = [('postgresql','PI_DB_PASSWORD','password'), ('privacyidea','PI_SECRET_KEY','PI_SECRET_KEY'), + ('privacyidea','PI_DB_PASSWORD','PI_SQLALCHEMY_DATABASE_URI'), + ('lldap','LLDAP_JWT_SECRET','LLDAP_JWT_SECRET'), ('authelia','AUTHELIA_SESSION_SECRET','session_secret')] + + +@pytest.fixture +def case(tmp_path): + private=tmp_path/'private';private.mkdir(mode=0o700) + for component,names in VALUES.items(): + directory=private/component;directory.mkdir(mode=0o700) + env=directory/'secrets.env';env.write_text(''.join(f'{name}={PASSWORD}\n' for name in names));env.chmod(0o600) + binary=tmp_path/'bin';binary.mkdir() + (binary/'kubectl').write_text('''#!/usr/bin/python3 +import json,os,sys +from pathlib import Path +args=sys.argv[1:] +raw=sys.stdin.read() if '-f' in args or '--patch-file=/dev/stdin' in args else '' +with open(os.environ['CALLS'],'a') as f:f.write(json.dumps({'args':args,'stdin':raw})+'\\n') +if os.environ.get('FAIL')==args[0]: + print(os.environ['MARKER']);print(os.environ['MARKER'],file=sys.stderr);sys.exit(1) +p=Path(os.environ['STATE']) +if args[0]=='get': + if os.environ.get('BAD_GET'):print(os.environ['BAD_GET']) + elif p.exists():print(p.read_text()) + sys.exit(0) +if args[0]=='create' and raw: + if p.exists() or os.environ.get('RACE'):sys.exit(1) + state=json.loads(raw);state['metadata'].update(uid='fixture-uid',resourceVersion='1') +elif args[0]=='patch': + if os.environ.get('RACE'):sys.exit(1) + state=json.loads(p.read_text()) + for op in json.loads(raw): + parts=[s.replace('~1','/').replace('~0','~') for s in op['path'].split('/')[1:]] + parent=state + for part in parts[:-1]:parent=parent[part] + if op['op']=='test': + if parent.get(parts[-1])!=op['value']:sys.exit(1) + elif op['op']=='remove':del parent[parts[-1]] + elif op['op']=='replace':parent[parts[-1]]=op['value'] + else:sys.exit(1) + state['metadata']['resourceVersion']=str(int(state['metadata']['resourceVersion'])+1) +else:sys.exit(2) +p.write_text(json.dumps(state)) +if os.environ.get('BAD_RESPONSE'):state['data']['unexpected']='wrong' +print(json.dumps(state)) +''') + (binary/'kubectl').chmod(0o700) + env={'PATH':str(binary)+':'+os.environ['PATH'],'HOME':str(tmp_path), + 'CALLS':str(tmp_path/'calls'),'STATE':str(tmp_path/'state'),'MARKER':PASSWORD} + return private,env + + +def invoke(case,component,rotate=None,**extra): + args=['bash','-x',str(ROOT/f'sso-mfa/k8s/{component}/create-secrets.sh'),str(case[0])] + if rotate:args+=['--rotate-key',rotate] + return subprocess.run(args,env={**case[1],**extra},capture_output=True,text=True,timeout=20) + + +def calls(case): + p=Path(case[1]['CALLS']);return [json.loads(line) for line in p.read_text().splitlines()] if p.exists() else [] + + +def state(case):return json.loads(Path(case[1]['STATE']).read_text()) + +def put_state(case,obj):Path(case[1]['STATE']).write_text(json.dumps(obj)) + +def check_private_result(result): + assert PASSWORD not in result.stdout+result.stderr + assert CHANGED not in result.stdout+result.stderr + assert 'BEGIN PRIVATE KEY' not in result.stdout+result.stderr + + +def source(component):return 'privacyidea' if component=='postgresql' else component + + +@pytest.mark.parametrize('component',COMPONENTS) +def test_create_uses_literal_inputs_stdin_and_no_apply_annotation(case,component): + result=invoke(case,component);assert result.returncode==0,result.stderr;check_private_result(result) + assert PASSWORD not in json.dumps([c['args'] for c in calls(case)]) + writes=[c for c in calls(case) if c['args'][0]!='get'] + assert len(writes)==1 and writes[0]['args']==['create','-o','json','-f','-'] + obj=state(case);assert 'annotations' not in obj['metadata'] + values={k:base64.b64decode(v).decode() for k,v in obj['data'].items()} + if component=='postgresql':assert values=={'username':'privacyidea','password':PASSWORD} + elif component=='privacyidea': + assert values['PI_PEPPER']==PASSWORD and values['PI_SECRET_KEY']==PASSWORD + assert values['PI_SQLALCHEMY_DATABASE_URI']==f'postgresql://privacyidea:{PASSWORD}@net-kingdom-pg-rw.databases.svc.cluster.local:5432/privacyidea_db' + elif component=='lldap':assert values=={k:PASSWORD for k in VALUES['lldap']} + else: + key=case[0]/'authelia/oidc_private_key.pem' + assert key.stat().st_mode & 0o077==0 and values['oidc_issuer_private_key']==key.read_text() + assert bcrypt.checkpw(PASSWORD.encode(),values['keycape_client_secret_hash'].encode()) + assert values['ldap_password']==PASSWORD + + +@pytest.mark.parametrize('component',COMPONENTS) +def test_repeated_apply_is_noop_and_preserves_other_data(case,component): + assert invoke(case,component).returncode==0 + old=state(case);old['data']['owner-added']='c3ludGhldGlj';old['metadata']['annotations']={'owner':'retained'};put_state(case,old) + count=len(calls(case));result=invoke(case,component) + assert result.returncode==0;check_private_result(result) + assert state(case)==old + assert all(c['args'][0]=='get' for c in calls(case)[count:]) + + +@pytest.mark.parametrize('component,key,field',ROTATIONS) +def test_explicit_rotation_is_conditional_and_changes_only_named_field(case,component,key,field): + assert invoke(case,component).returncode==0 + old=state(case);old['data']['owner-added']='c3ludGhldGlj';old['metadata']['annotations']={'owner':'retained','kubectl.kubernetes.io/last-applied-configuration':'synthetic-old-copy'};put_state(case,old) + file=case[0]/source(component)/'secrets.env';file.write_text(file.read_text().replace(f'{key}={PASSWORD}',f'{key}={CHANGED}')) + result=invoke(case,component,rotate=key);assert result.returncode==0,result.stderr;check_private_result(result) + updated=state(case) + assert updated['metadata']['annotations']=={'owner':'retained'} + assert updated['data'][field]!=old['data'][field] + assert {k:v for k,v in updated['data'].items() if k!=field}=={k:v for k,v in old['data'].items() if k!=field} + patch=next(c for c in reversed(calls(case)) if c['args'][0]=='patch') + tests={op['path']:op['value'] for op in json.loads(patch['stdin']) if op['op']=='test'} + assert tests['/metadata/uid']==old['metadata']['uid'] and tests['/metadata/resourceVersion']==old['metadata']['resourceVersion'] + assert PASSWORD not in json.dumps(patch['args']) and CHANGED not in json.dumps(patch['args']) + + +@pytest.mark.parametrize('component',COMPONENTS) +def test_differing_values_require_explicit_rotation(case,component): + assert invoke(case,component).returncode==0;old=state(case) + file=case[0]/source(component)/'secrets.env';file.write_text(file.read_text().replace(PASSWORD,CHANGED)) + count=len(calls(case));result=invoke(case,component) + assert result.returncode!=0;check_private_result(result);assert state(case)==old + assert all(c['args'][0]=='get' for c in calls(case)[count:]) + + +@pytest.mark.parametrize('component',COMPONENTS) +@pytest.mark.parametrize('defect',['shell','duplicate','public','symlink','hardlink','public-parent']) +def test_bad_private_input_never_reaches_cluster(case,component,defect): + file=case[0]/source(component)/'secrets.env';marker=case[0]/'executed' + if defect=='shell':file.write_text(file.read_text()+f'$(touch {marker})\n') + elif defect=='duplicate':file.write_text(file.read_text()*2) + elif defect=='public':file.chmod(0o644) + elif defect=='public-parent':file.parent.chmod(0o755) + elif defect=='hardlink':os.link(file,file.with_name('alias')) + else: + target=file.with_name('other');file.rename(target);file.symlink_to(target) + result=invoke(case,component);assert result.returncode!=0;check_private_result(result) + assert not marker.exists() and not calls(case) + + +@pytest.mark.parametrize('component',COMPONENTS) +@pytest.mark.parametrize('failure',['get','create']) +def test_failed_child_diagnostics_never_escape(case,component,failure): + result=invoke(case,component,FAIL=failure);assert result.returncode!=0;check_private_result(result) + assert not Path(case[1]['STATE']).exists() + + +@pytest.mark.parametrize('defect',['owner','eso-label','identity','type','basic-auth','data','immutable','missing-uid']) +def test_wrong_or_controller_owned_existing_secret_is_refused(case,defect): + assert invoke(case,'lldap').returncode==0;old=state(case) + if defect=='owner':old['metadata']['ownerReferences']=[{'kind':'ExternalSecret','name':'fixture'}] + elif defect=='eso-label':old['metadata']['labels']={'reconcile.external-secrets.io/managed':'true'} + elif defect=='missing-uid':old['metadata']['uid']=None + elif defect=='identity':old['metadata']['namespace']='other' + elif defect=='type':old['type']='kubernetes.io/tls' + elif defect=='basic-auth':old['type']='kubernetes.io/basic-auth' + elif defect=='data':old['data']['LLDAP_JWT_SECRET']='invalid-base64' + else:old['immutable']=True + put_state(case,old);count=len(calls(case));result=invoke(case,'lldap') + assert result.returncode!=0 and state(case)==old + assert all(c['args'][0]=='get' for c in calls(case)[count:]) + + +@pytest.mark.parametrize('component,key,field',ROTATIONS) +def test_concurrent_change_stops_rotation_without_retry(case,component,key,field): + assert invoke(case,component).returncode==0;old=state(case) + file=case[0]/source(component)/'secrets.env';file.write_text(file.read_text().replace(f'{key}={PASSWORD}',f'{key}={CHANGED}')) + count=len(calls(case));result=invoke(case,component,rotate=key,RACE='1') + assert result.returncode!=0 and state(case)==old;check_private_result(result) + assert sum(c['args'][0]=='patch' for c in calls(case)[count:])==1 + + +def test_create_race_never_switches_to_overwrite(case): + result=invoke(case,'lldap',RACE='1');assert result.returncode!=0 + assert [c['args'][0] for c in calls(case)]==['get','create'] + + +def test_authelia_failure_retains_same_key_on_retry(case): + result=invoke(case,'authelia',FAIL='create');assert result.returncode!=0 + key=case[0]/'authelia/oidc_private_key.pem';original=key.read_bytes() + assert invoke(case,'authelia').returncode==0 and key.read_bytes()==original + + +def test_rotating_session_never_generates_or_replaces_existing_signing_key(case): + assert invoke(case,'authelia').returncode==0;old=state(case) + (case[0]/'authelia/oidc_private_key.pem').unlink() + file=case[0]/'authelia/secrets.env';file.write_text(file.read_text().replace(f'AUTHELIA_SESSION_SECRET={PASSWORD}',f'AUTHELIA_SESSION_SECRET={CHANGED}')) + assert invoke(case,'authelia',rotate='AUTHELIA_SESSION_SECRET').returncode==0 + assert state(case)['data']['oidc_issuer_private_key']==old['data']['oidc_issuer_private_key'] + assert not (case[0]/'authelia/oidc_private_key.pem').exists() + + +def test_wrong_response_is_not_reported_as_success(case): + result=invoke(case,'lldap',BAD_RESPONSE='1');assert result.returncode!=0;check_private_result(result) + + +@pytest.mark.parametrize('annotation',['', 'synthetic-old-copy']) +def test_annotation_cleanup_preserves_secret_values_and_other_annotations(case,annotation): + assert invoke(case,'lldap').returncode==0;old=state(case) + old['metadata']['annotations']={'owner':'retained','kubectl.kubernetes.io/last-applied-configuration':annotation};put_state(case,old) + result=invoke(case,'lldap');assert result.returncode==0 + assert state(case)['data']==old['data'] + assert state(case)['metadata']['annotations']=={'owner':'retained'} + + +@pytest.mark.parametrize('component,key,field',ROTATIONS) +def test_rotation_refuses_missing_secret_before_create(case,component,key,field): + result=invoke(case,component,rotate=key);assert result.returncode!=0 + assert all(c['args'][0]=='get' for c in calls(case)) + assert not (case[0]/'authelia/oidc_private_key.pem').exists() + + +def test_rotation_cannot_change_an_unrelated_field(case): + assert invoke(case,'privacyidea').returncode==0;old=state(case) + file=case[0]/'privacyidea/secrets.env';file.write_text(file.read_text().replace(PASSWORD,CHANGED)) + result=invoke(case,'privacyidea',rotate='PI_SECRET_KEY');assert result.returncode!=0 + assert state(case)==old + assert not any(c['args'][0]=='patch' for c in calls(case)) + + +@pytest.mark.parametrize('defect',['symlink','hardlink','public','public-parent','invalid']) +def test_explicit_authelia_key_custody_is_validated_without_overwrite(case,defect): + key=case[0]/'authelia/external.pem';key.write_text('synthetic-invalid-key');key.chmod(0o600) + if defect=='symlink': + target=key.with_name('other');key.rename(target);key.symlink_to(target) + elif defect=='hardlink':os.link(key,key.with_name('alias')) + elif defect=='public':key.chmod(0o644) + elif defect=='public-parent':key.parent.chmod(0o755) + file=case[0]/'authelia/secrets.env';file.write_text(file.read_text()+f'AUTHELIA_OIDC_PRIVATE_KEY_FILE={key}\n') + result=invoke(case,'authelia');assert result.returncode!=0 + assert key.read_text()=='synthetic-invalid-key' + assert not any(c['args'][0]!='get' for c in calls(case)) + + +def test_native_kubectl_applies_stdin_patch_and_refuses_stale_version(case,tmp_path): + import shutil + binary=shutil.which('kubectl') + if binary is None:pytest.skip('native kubectl not available') + assert invoke(case,'lldap').returncode==0;old=state(case) + file=case[0]/'lldap/secrets.env';file.write_text(file.read_text().replace(f'LLDAP_JWT_SECRET={PASSWORD}',f'LLDAP_JWT_SECRET={CHANGED}')) + assert invoke(case,'lldap',rotate='LLDAP_JWT_SECRET').returncode==0 + patch=next(c['stdin'] for c in reversed(calls(case)) if c['args'][0]=='patch') + original=tmp_path/'original.json';original.write_text(json.dumps(old));original.chmod(0o600) + env={**os.environ,'KUBECONFIG':str(tmp_path/'no-cluster-config'),'HOME':str(tmp_path)} + command=[binary,'patch','--local','--type=json','--patch-file=/dev/stdin','-f',str(original),'-o','json'] + result=subprocess.run(command,input=patch,env=env,capture_output=True,text=True,timeout=20) + assert result.returncode==0 + assert json.loads(result.stdout)['data']==state(case)['data'] + old['metadata']['resourceVersion']='999';original.write_text(json.dumps(old)) + refused=subprocess.run(command,input=patch,env=env,capture_output=True,text=True,timeout=20) + assert refused.returncode!=0 diff --git a/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md b/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md index afabe21..387cb37 100644 --- a/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md +++ b/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md @@ -83,8 +83,9 @@ Concrete uncovered callers remain (T02 stays progress): - RSA and Authelia-to-KeyCape client-secret rotation need configuration-preserving updates, retained rollback custody and native multi-consumer acceptance. The legacy replacement paths now refuse before mutation; this is not rotation closure. -- Delegated create/apply helpers still need full tracing/error/Secret-annotation - output review; transport capture and native effect evidence remain mandatory. +- PostgreSQL/privacyIDEA/LLDAP/Authelia generators and the batch apply caller + are repaired below. Other diagnostic/bootstrap/key-export callers and direct + SOPS edit/encrypt/rotate remain; native effect/recovery evidence is mandatory. 2026-10-12 Make caller repair: sops-setup now validates only the public recipient; creds-init no longer requires or installs a permanent private key. sops-decrypt @@ -151,6 +152,36 @@ current deployed consumer acceptance and other create/apply/SOPS callers remain open. Source verification and its limits: Custodian `docs/evidence/2026-10-13-netkingdom-security/keycape-caller-reconciliation.json`. +2026-10-13 delegated service caller repair: the PostgreSQL, privacyIDEA, +LLDAP and Authelia generators now share private literal-input parsing, captured +child diagnostics and stdin-only Secret transfer. Ordinary invocation creates +an absent object or verifies matching fields; it cannot silently rotate an +existing object. Named rotation updates only its allowed field. UID/resource +version/data tests guard patches; unknown data and unrelated metadata survive, +and last-applied copies are removed under the same guard. Owner references, +ESO management labels, immutable objects and wrong target identity are refused. +The returned API object is checked before success; write failure remains an +unknown remote outcome requiring reconciliation, never blind retry. + +Authelia first creation retains a private signing key atomically; API failure +keeps it for recovery. Existing issuer keys and matching bcrypt hashes survive +repeat apply and session rotation. Explicit unsafe/differing keys are refused. +The batch caller suppresses child output and records only successful components. +Rotation callers pass exact field selectors; database rotation now updates the +CNPG application Secret as well as the privacyIDEA URI. The attended LDAP path +also depended on retired full KeyCape replacement, so now refuses before any +change. SQL, Secrets and restarts remain non-atomic; coordinated recovery and +native database/consumer acceptance are still required. + +All 63 initial regression cases failed on the old helpers; the replacement and +expanded caller tests pass, including real OpenSSL/bcrypt and actual kubectl +local stdin patch/stale-version refusal with no cluster. Final source-bound +suite results, dependency skips and test limitations are in Custodian +`docs/evidence/2026-10-13-netkingdom-security/service-secret-callers.json`. +No live Secret, signing key, credential or deployment was changed. T02 remains +progress for the explicit callers/lifecycle work above; T03 remains waiting for +its owner-scoped native recovery ceremony. + ## Verify owner-approved recovery and current native callers ```task