fix(security): make service Secret updates private and conditional

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a1346b-72a4-7d40-b8bd-8b8ca346cfa2

sso-mfa/bootstrap/README.md

@@ -128,6 +128,66 @@ apply, the operator must store and independently verify the fresh encrypted ops
 bundle before acknowledging its handoff. Source tests use disposable keys and a
 simulated cluster; native acceptance remains NK-WP-0049.
 
+## Service Secret creation and explicit rotation
+
+The PostgreSQL, privacyIDEA, LLDAP and Authelia `create-secrets.sh` entry points
+use `apply-service-secrets.py`. Under the current approved bootstrap/maintenance
+procedure, pass the private generator directory as their first argument. The
+helper requires Python 3, kubectl and OpenSSL; Authelia also requires Python
+bcrypt. Input directories/files must be owned, private and free of symlinks or
+hard-linked files. Generated env files are parsed as literal data, never sourced.
+
+Absent Secrets are created. Existing Secrets must match the supplied values;
+ordinary reapplication cannot rotate them. Objects with owner references or ESO management labels, immutable Secrets,
+and mismatched target objects are refused. Unknown data keys, labels
+and unrelated annotations are preserved. Any stored last-applied annotation is
+removed under the same guarded update to stop retaining duplicate secret values.
+Writes use stdin and captured diagnostics. Updates test the original object UID,
+resource version and data; conflicts fail without retry. Returned API state is
+checked before reporting success. A lost/failed response can leave the remote
+outcome unknown; inspect through approved metadata-only tooling before retrying.
+
+For an already reviewed rotation, the explicit interface is:
+
+```bash
+bash sso-mfa/k8s/privacyidea/create-secrets.sh /absolute/private/secrets --rotate-key PI_SECRET_KEY
+```
+
+Only these component/field pairs are accepted:
+
+| Component | Rotation selector | Updated Secret field |
+| --- | --- | --- |
+| PostgreSQL | `PI_DB_PASSWORD` | `password` |
+| privacyIDEA | `PI_DB_PASSWORD` | `PI_SQLALCHEMY_DATABASE_URI` |
+| privacyIDEA | `PI_SECRET_KEY` | `PI_SECRET_KEY` |
+| LLDAP | `LLDAP_JWT_SECRET` | `LLDAP_JWT_SECRET` |
+| Authelia | `AUTHELIA_SESSION_SECRET` | `session_secret` |
+
+The selector describes an operation; it grants no authority. All other supplied
+fields must still match. Rotation requires the target Secret to exist. The
+guided caller passes these selectors and updates both the CNPG application
+Secret and privacyIDEA URI during database password rotation. SQL, Secret writes
+and restarts are separate steps, not an atomic transaction. Native database
+reconciliation, predecessor denial and consumer acceptance remain required;
+failure preserves the local candidate and does not record completed rotation.
+The legacy LDAP/Authelia/KeyCape coordinated replacement path refuses before
+mutation in both attended and unattended modes, as do KeyCape RSA/client-secret
+replacement paths. Implement and review retained rollback custody and scoped
+consumer updates before using a replacement procedure (NK-WP-0049-T02).
+
+For first Authelia creation, an explicit `AUTHELIA_OIDC_PRIVATE_KEY_FILE` must be
+an absolute path to a private owned RSA key. Without it, the helper preserves or
+atomically creates `authelia/oidc_private_key.pem` under the private input root.
+A new key survives API failure for recovery. Existing deployments retain their
+current issuer key; missing local custody never triggers a replacement key.
+An explicitly supplied differing key is refused. Existing bcrypt hashes are
+verified against the supplied client value and preserved, not regenerated.
+Independent signing-key backup remains part of the owner recovery ceremony.
+
+`creds-apply.sh` captures delegated output and records a component only after
+its helper succeeds. Batch application is not atomic: earlier components may
+already be applied when a later one fails. File status is not native acceptance.
+
 ## Custody Age Key Unlock
 
 For SOPS-backed drills and incident work, do not copy the custodian age private

sso-mfa/bootstrap/apply-service-secrets.py

@@ -0,0 +1,252 @@
+#!/usr/bin/env python3
+"""Private service Secret creation, exact-match verification and named rotation.
+
+No credential is accepted on argv, evaluated as shell code, or printed. This
+helper is not an authority grant or a multi-service rotation transaction.
+"""
+import base64
+import contextlib
+import json
+import os
+from pathlib import Path
+import re
+import secrets
+import stat
+import subprocess
+import sys
+
+
+class Refused(Exception):
+    pass
+
+
+TARGETS = {
+    'postgresql': ('databases', 'net-kingdom-pg-privacyidea-app', ('privacyidea',)),
+    'privacyidea': ('mfa', 'privacyidea-config', ('privacyidea',)),
+    'lldap': ('sso', 'lldap-secrets', ('lldap',)),
+    'authelia': ('sso', 'authelia-secrets', ('authelia', 'lldap')),
+}
+ROTATIONS = {
+    'postgresql': {'PI_DB_PASSWORD': 'password'},
+    'privacyidea': {'PI_SECRET_KEY': 'PI_SECRET_KEY', 'PI_DB_PASSWORD': 'PI_SQLALCHEMY_DATABASE_URI'},
+    'lldap': {'LLDAP_JWT_SECRET': 'LLDAP_JWT_SECRET'},
+    'authelia': {'AUTHELIA_SESSION_SECRET': 'session_secret'},
+}
+ANNOTATION = 'kubectl.kubernetes.io/last-applied-configuration'
+
+
+def private(info, directory=False):
+    kind = stat.S_ISDIR if directory else stat.S_ISREG
+    if not kind(info.st_mode) or info.st_uid != os.getuid() or info.st_mode & 0o077:
+        raise Refused()
+    if not directory and info.st_nlink != 1:
+        raise Refused()
+
+
+@contextlib.contextmanager
+def custody(root, components):
+    root = Path(os.path.abspath(root))
+    if any(p.is_symlink() for p in (root, *root.parents)): raise Refused()
+    descriptors = []
+    try:
+        flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW
+        fd = os.open(root, flags); descriptors.append(fd); private(os.fstat(fd), True)
+        children = {}
+        for component in components:
+            child = os.open(component, flags, dir_fd=fd)
+            descriptors.append(child); private(os.fstat(child), True)
+            children[component] = child
+        yield children
+    finally:
+        for fd in reversed(descriptors): os.close(fd)
+
+
+def read_private(parent, name):
+    fd = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=parent)
+    try:
+        private(os.fstat(fd))
+        raw = os.read(fd, 65537)
+        if len(raw) > 65536: raise Refused()
+        return raw
+    finally:
+        os.close(fd)
+
+
+def read_env(parent):
+    values = {}
+    for line in read_private(parent, 'secrets.env').decode().splitlines():
+        if not line.strip() or line.lstrip().startswith('#'): continue
+        name, separator, value = line.partition('=')
+        if not separator or not re.fullmatch(r'[A-Z][A-Z0-9_]*', name) or name in values:
+            raise Refused()
+        values[name] = value
+    return values
+
+
+def literal(values, name):
+    value = values.get(name, '')
+    if not re.fullmatch(r'[A-Za-z0-9]{32,128}', value): raise Refused()
+    return value
+
+
+def run(args, payload=None):
+    completed = subprocess.run(args, input=payload, stdout=subprocess.PIPE,
+                               stderr=subprocess.DEVNULL, timeout=60, check=False)
+    if completed.returncode or len(completed.stdout) > 2 * 1024 * 1024: raise Refused()
+    return completed.stdout
+
+
+def decode_secret(raw, namespace, name):
+    obj = json.loads(raw)
+    if not isinstance(obj, dict) or obj.get('kind') != 'Secret' or obj.get('apiVersion') != 'v1':
+        raise Refused()
+    meta = obj.get('metadata')
+    if not isinstance(meta, dict) or meta.get('name') != name or meta.get('namespace') != namespace:
+        raise Refused()
+    if (not isinstance(meta.get('uid'), str) or not isinstance(meta.get('resourceVersion'), str)
+            or not re.fullmatch(r'[a-zA-Z0-9-]{1,128}', meta['uid'])
+            or not re.fullmatch(r'[0-9]{1,128}', meta['resourceVersion'])):
+        raise Refused()
+    if (meta.get('ownerReferences') or obj.get('immutable')
+            or obj.get('type') not in (('Opaque', 'kubernetes.io/basic-auth')
+                                     if name == 'net-kingdom-pg-privacyidea-app' else ('Opaque',))):
+        raise Refused()
+    annotations = meta.get('annotations', {})
+    labels = meta.get('labels', {})
+    if not isinstance(annotations, dict) or not isinstance(labels, dict): raise Refused()
+    if any(not isinstance(v, str) for v in (*annotations.values(), *labels.values())): raise Refused()
+    if any(k.startswith(('external-secrets.io/', 'reconcile.external-secrets.io/')) for k in labels): raise Refused()
+    if not isinstance(obj.get('data'), dict) or not obj['data']: raise Refused()
+    for key, value in obj['data'].items():
+        if not isinstance(key, str) or not isinstance(value, str): raise Refused()
+        base64.b64decode(value, validate=True)
+    return obj
+
+
+def read_key(parent, configured, existing):
+    if configured:
+        path = Path(configured)
+        if not path.is_absolute() or any(p.is_symlink() for p in (path, *path.parents)):
+            raise Refused()
+        fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
+        try:
+            private(os.fstat(fd), True)
+            key = read_private(fd, path.name)
+        finally:
+            os.close(fd)
+    else:
+        try:
+            key = read_private(parent, 'oidc_private_key.pem')
+        except FileNotFoundError:
+            if existing is not None:
+                key = base64.b64decode(existing['data']['oidc_issuer_private_key'], validate=True)
+            else:
+                key = run(['openssl', 'genrsa', '2048'])
+                run(['openssl', 'rsa', '-check', '-noout'], key)
+                temp = '.issuer-' + secrets.token_hex(12)
+                fd = os.open(temp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600, dir_fd=parent)
+                try:
+                    with os.fdopen(fd, 'wb') as file:
+                        file.write(key); file.flush(); os.fsync(file.fileno())
+                    os.link(temp, 'oidc_private_key.pem', src_dir_fd=parent, dst_dir_fd=parent, follow_symlinks=False)
+                    os.fsync(parent)
+                finally:
+                    os.unlink(temp, dir_fd=parent)
+    run(['openssl', 'rsa', '-check', '-noout'], key)
+    return key.decode()
+
+
+def service_values(component, values):
+    # Validate all required literals before any cluster call or local key creation.
+    if component == 'postgresql':
+        return {'username': 'privacyidea', 'password': literal(values['privacyidea'], 'PI_DB_PASSWORD')}
+    if component == 'privacyidea':
+        env = values['privacyidea']; password = literal(env, 'PI_DB_PASSWORD')
+        return {'PI_SECRET_KEY': literal(env, 'PI_SECRET_KEY'), 'PI_PEPPER': literal(env, 'PI_PEPPER'),
+                'PI_SQLALCHEMY_DATABASE_URI': 'postgresql://privacyidea:' + password +
+                '@net-kingdom-pg-rw.databases.svc.cluster.local:5432/privacyidea_db'}
+    if component == 'lldap':
+        return {key: literal(values['lldap'], key) for key in ('LLDAP_JWT_SECRET', 'LLDAP_LDAP_USER_PASS')}
+    env = values['authelia']
+    return {'jwt_secret': literal(env, 'AUTHELIA_JWT_SECRET'),
+            'session_secret': literal(env, 'AUTHELIA_SESSION_SECRET'),
+            'storage_encryption_key': literal(env, 'AUTHELIA_STORAGE_ENCRYPTION_KEY'),
+            'oidc_hmac_secret': literal(env, 'AUTHELIA_OIDC_HMAC_SECRET'),
+            'ldap_password': literal(values['lldap'], 'LLDAP_LDAP_USER_PASS')}
+
+
+def publish(namespace, name, desired, old, allowed):
+    encoded = {k: base64.b64encode(v.encode()).decode() for k, v in desired.items()}
+    if old is None:
+        if allowed: raise Refused()  # Rotation may not silently create missing state.
+        obj = {'apiVersion': 'v1', 'kind': 'Secret', 'type': 'Opaque',
+               'metadata': {'namespace': namespace, 'name': name}, 'data': encoded}
+        result = run(['kubectl', 'create', '-o', 'json', '-f', '-'], json.dumps(obj).encode())
+        expected = encoded; outcome = 'created'
+    else:
+        if not set(encoded).issubset(old['data']): raise Refused()
+        different = {k for k, value in encoded.items() if old['data'][k] != value}
+        if different - allowed: raise Refused()
+        annotations = old['metadata'].get('annotations', {})
+        if not different and ANNOTATION not in annotations: return 'verified'
+        patch = [{'op': 'test', 'path': '/metadata/uid', 'value': old['metadata']['uid']},
+                 {'op': 'test', 'path': '/metadata/resourceVersion', 'value': old['metadata']['resourceVersion']},
+                 {'op': 'test', 'path': '/data', 'value': old['data']}]
+        patch.extend({'op': 'replace', 'path': '/data/' + key, 'value': encoded[key]} for key in sorted(different))
+        if ANNOTATION in annotations:
+            patch.append({'op': 'remove', 'path': '/metadata/annotations/kubectl.kubernetes.io~1last-applied-configuration'})
+        result = run(['kubectl', 'patch', 'secret', name, '-n', namespace, '--type=json',
+                      '--patch-file=/dev/stdin', '-o', 'json'], json.dumps(patch).encode())
+        expected = {**old['data'], **{k: encoded[k] for k in different}}
+        outcome = 'updated'
+    readback = decode_secret(result, namespace, name)
+    if readback['data'] != expected or ANNOTATION in readback['metadata'].get('annotations', {}): raise Refused()
+    if old is not None:
+        if readback['metadata']['uid'] != old['metadata']['uid'] or readback['type'] != old['type']: raise Refused()
+        expected_annotations = {k: v for k, v in old['metadata'].get('annotations', {}).items() if k != ANNOTATION}
+        if readback['metadata'].get('annotations', {}) != expected_annotations: raise Refused()
+        if readback['metadata'].get('labels', {}) != old['metadata'].get('labels', {}): raise Refused()
+    return outcome
+
+
+def main():
+    if len(sys.argv) not in (2, 3, 5): raise Refused()
+    component = sys.argv[1]
+    if component not in TARGETS: raise Refused()
+    namespace, name, inputs = TARGETS[component]
+    root = sys.argv[2] if len(sys.argv) >= 3 else Path(__file__).resolve().parent / 'secrets'
+    rotation = None
+    if len(sys.argv) == 5:
+        if sys.argv[3] != '--rotate-key' or sys.argv[4] not in ROTATIONS[component]: raise Refused()
+        rotation = ROTATIONS[component][sys.argv[4]]
+    with custody(root, inputs) as directories:
+        values = {c: read_env(directories[c]) for c in inputs}
+        desired = service_values(component, values)
+        if component == 'authelia':
+            import bcrypt
+            client = literal(values['authelia'], 'AUTHELIA_KEYCAPE_CLIENT_SECRET').encode()
+            if len(client) > 72: raise Refused()
+        raw = run(['kubectl', 'get', 'secret', name, '-n', namespace, '--ignore-not-found', '-o', 'json'])
+        old = decode_secret(raw, namespace, name) if raw.strip() else None
+        if rotation and old is None: raise Refused()
+        if component == 'authelia':
+            if old is None:
+                hashed = bcrypt.hashpw(client, bcrypt.gensalt(rounds=12))
+            else:
+                hashed = base64.b64decode(old['data']['keycape_client_secret_hash'], validate=True)
+                if not bcrypt.checkpw(client, hashed): raise Refused()
+            desired['keycape_client_secret_hash'] = hashed.decode()
+            desired['oidc_issuer_private_key'] = read_key(directories['authelia'],
+                values['authelia'].get('AUTHELIA_OIDC_PRIVATE_KEY_FILE', ''), old)
+        outcome = publish(namespace, name, desired, old, {rotation} if rotation else set())
+    print('Service Secret ' + outcome + '; native consumer acceptance remains required.')
+
+
+if __name__ == '__main__':
+    try:
+        main()
+    except (Refused, OSError, ValueError, KeyError, TypeError, ImportError, subprocess.SubprocessError):
+        print('ERROR: service Secret operation refused or failed. Check private inputs, current ownership '
+              'and the named rotation through approved metadata-only diagnostics. A failed write has an '
+              'unverified remote outcome; reconcile before retry. Local recovery keys are retained.', file=sys.stderr)
+        sys.exit(1)

sso-mfa/bootstrap/creds-apply.sh

@@ -13,13 +13,20 @@
 #   5. keycape     SKIPPED — requires owner-established RPF-WP-0040 factor delivery.
 #                  create-secrets.sh is first-creation only, never live replacement.
 #
-# After each successful component, creds-state.yaml is updated.
+# After each successful component create/verification, creds-state.yaml is updated.
+# Differing existing values require a separate named rotation; this is not a batch rotation.
 
+set +x
 set -euo pipefail
+umask 077
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
-SECRETS_DIR="${1:-$SCRIPT_DIR/secrets}"
+if (( $# > 1 )); then
+    echo "ERROR: expected one private secrets directory." >&2
+    exit 1
+fi
+SECRETS_DIR="$(realpath -sm -- "${1:-$SCRIPT_DIR/secrets}")"
 STATE_FILE="$SCRIPT_DIR/creds-state.yaml"
 K8S_DIR="$REPO_ROOT/sso-mfa/k8s"
 
@@ -44,7 +51,7 @@ fi
 
 echo "=== creds-apply — net-kingdom SSO/MFA secrets ==="
 echo "Secrets dir : $SECRETS_DIR"
-echo "Cluster     : $(kubectl config current-context 2>/dev/null || echo '(unknown)')"
+echo "Cluster connectivity checked; use only the context admitted by the current procedure."
 echo ""
 
 # ── Helper: run a component's create-secrets.sh ───────────────────────────────
@@ -62,7 +69,11 @@ run_component() {
         exit 1
     fi
 
-    (cd "$script_dir" && bash create-secrets.sh "$SECRETS_DIR")
+    if ! (cd "$script_dir" && bash create-secrets.sh "$SECRETS_DIR") >/dev/null 2>&1; then
+        echo "ERROR: component create/verification failed; earlier components may already be applied." >&2
+        echo "Use metadata-only reconciliation before retry; no native acceptance is claimed." >&2
+        exit 1
+    fi
 
     # Update state file
     if [[ -f "$STATE_FILE" ]]; then

sso-mfa/bootstrap/creds-rotate.sh

@@ -134,7 +134,7 @@ PI_SECRET_KEY)
     echo ""
     store_value privacyidea
     echo "  1. Updating K8s Secret privacyidea-config (namespace: mfa)..."
-    (cd "$K8S_DIR/privacyidea" && quiet bash create-secrets.sh "$SECRETS_DIR")
+    (cd "$K8S_DIR/privacyidea" && quiet bash create-secrets.sh "$SECRETS_DIR" --rotate-key "$SECRET")
     echo "  2. Restarting privacyIDEA pod..."
     quiet kubectl rollout restart deployment privacyidea -n mfa
     quiet kubectl rollout status deployment privacyidea -n mfa
@@ -160,10 +160,10 @@ PI_PEPPER)
 PI_DB_PASSWORD)
     header
     echo ""
-    echo "Impact: privacyIDEA database password — must be updated atomically in:"
+    echo "Impact: privacyIDEA database password — coordinated updates are required in:"
     echo "  1. PostgreSQL (CNPG) — ALTER USER privacyidea WITH PASSWORD '...';"
-    echo "  2. K8s Secret privacyidea-config (PI_SQLALCHEMY_DATABASE_URI)"
-    echo "  Privacyidea pod must be restarted after both are updated."
+    echo "  2. CNPG application Secret and privacyidea-config (PI_SQLALCHEMY_DATABASE_URI)"
+    echo "  These are separate steps; preserve the candidate and reconcile any partial failure."
     echo ""
     prepare_value privacyidea b64
     confirm "Apply rotation?" || exit 0
@@ -183,8 +183,9 @@ PI_DB_PASSWORD)
         quiet kubectl exec -i -n databases "$PG_POD" -- psql -U postgres -v ON_ERROR_STOP=1
     echo "     PostgreSQL password update command succeeded."
     echo ""
-    echo "  2. Updating K8s Secret privacyidea-config..."
-    (cd "$K8S_DIR/privacyidea" && quiet bash create-secrets.sh "$SECRETS_DIR")
+    echo "  2. Updating PostgreSQL application credential Secret and privacyidea-config..."
+    (cd "$K8S_DIR/postgresql" && quiet bash create-secrets.sh "$SECRETS_DIR" --rotate-key "$SECRET")
+    (cd "$K8S_DIR/privacyidea" && quiet bash create-secrets.sh "$SECRETS_DIR" --rotate-key "$SECRET")
     echo ""
     echo "  3. Restarting privacyIDEA pod..."
     quiet kubectl rollout restart deployment privacyidea -n mfa
@@ -202,7 +203,7 @@ LLDAP_JWT_SECRET)
     confirm "Apply rotation?" || exit 0
     store_value lldap
     echo "  Updating K8s Secret lldap-secrets (namespace: sso)..."
-    (cd "$K8S_DIR/lldap" && quiet bash create-secrets.sh "$SECRETS_DIR")
+    (cd "$K8S_DIR/lldap" && quiet bash create-secrets.sh "$SECRETS_DIR" --rotate-key "$SECRET")
     echo "  Restarting LLDAP pod..."
     quiet kubectl rollout restart deployment lldap -n sso
     quiet kubectl rollout status deployment lldap -n sso
@@ -211,38 +212,9 @@ LLDAP_JWT_SECRET)
     ;;
 
 LLDAP_LDAP_USER_PASS)
-    if [[ "$NON_INTERACTIVE" == true || ! -t 0 ]]; then
-        echo "ERROR: LDAP password rotation requires an attended manual server update; automation is unavailable." >&2
-        exit 1
-    fi
-    header
-    echo ""
-    echo "Impact: LLDAP admin + LDAP bind password — used by Authelia and KeyCape."
-    echo "  All three must be updated atomically:"
-    echo "    1. LLDAP admin password (via LLDAP web UI or API)"
-    echo "    2. Authelia K8s Secret (ldap_password)"
-    echo "    3. KeyCape K8s Secret (config.yaml lldap.bindPW)"
-    echo "  Restart all three pods after updating."
-    echo ""
-    prepare_value lldap b64
-    confirm "Apply rotation?" || exit 0
-    store_value lldap
-    echo "  1. Manually updating the LLDAP admin password..."
-    echo "     WARN: Automated LLDAP password update not implemented."
-    echo "           Use the candidate in the private lldap/secrets.env file to update the server."
-    echo "           If cancelled, reconcile that local candidate with the unchanged server before reuse."
-    confirm "Confirm you have updated the LLDAP admin password?" || { echo "Aborting; local candidate retained for recovery."; exit 1; }
-    echo "  2. Updating Authelia secrets..."
-    (cd "$K8S_DIR/authelia" && quiet bash create-secrets.sh "$SECRETS_DIR")
-    echo "  3. Updating KeyCape secrets..."
-    (cd "$K8S_DIR/keycape" && quiet bash create-secrets.sh "$SECRETS_DIR")
-    echo "  4. Restarting Authelia and KeyCape pods..."
-    quiet kubectl rollout restart deployment authelia -n sso
-    quiet kubectl rollout restart deployment keycape  -n sso
-    quiet kubectl rollout status deployment authelia  -n sso
-    quiet kubectl rollout status deployment keycape   -n sso
-    update_last_rotated "LLDAP_LDAP_USER_PASS"
-    post_rotation_reminder
+    echo "ERROR: coordinated LDAP/Authelia/KeyCape rotation requires an owner-reviewed procedure with rollback and native acceptance (NK-WP-0049-T02)." >&2
+    echo "The legacy manual-server/full-config-replacement path is retired before any credential changes." >&2
+    exit 1
     ;;
 
 AUTHELIA_SESSION_SECRET)
@@ -254,7 +226,7 @@ AUTHELIA_SESSION_SECRET)
     prepare_value authelia hex
     confirm "Apply rotation?" || exit 0
     store_value authelia
-    (cd "$K8S_DIR/authelia" && quiet bash create-secrets.sh "$SECRETS_DIR")
+    (cd "$K8S_DIR/authelia" && quiet bash create-secrets.sh "$SECRETS_DIR" --rotate-key "$SECRET")
     quiet kubectl rollout restart deployment authelia -n sso
     quiet kubectl rollout status deployment authelia  -n sso
     update_last_rotated "AUTHELIA_SESSION_SECRET"

sso-mfa/k8s/authelia/README.md

@@ -92,9 +92,13 @@ apps — the edge already calls Authelia `/api/verify`.
 | | `keycape_client_secret_hash` | Bcrypt hash of `AUTHELIA_KEYCAPE_CLIENT_SECRET` |
 
 `create-secrets.sh` reads plaintext values from `secrets/authelia/secrets.env` and
-`secrets/lldap/secrets.env`. It generates the bcrypt hash on the fly (requires
-`python3+bcrypt` or `apache2-utils`). The RSA OIDC private key is generated
-automatically if `AUTHELIA_OIDC_PRIVATE_KEY_FILE` is not set.
+`secrets/lldap/secrets.env` as private literal input (requires Python 3 + bcrypt).
+First creation stores a new signing key privately in `authelia/oidc_private_key.pem`
+unless an explicit private absolute key path is supplied. Existing deployments
+preserve their issuer key and matching bcrypt hash. Normal reapplication verifies
+matching values; session rotation must explicitly select `AUTHELIA_SESSION_SECRET`.
+See [service Secret custody and rotation](../../bootstrap/README.md#service-secret-creation-and-explicit-rotation)
+for conflict handling, recovery custody and native acceptance limits.
 
 ## Storage
 

sso-mfa/k8s/authelia/create-secrets.sh

@@ -1,113 +1,7 @@
 #!/usr/bin/env bash
-# create-secrets.sh — create the authelia-secrets K8s Secret
-#
-# Usage:
-#   ./create-secrets.sh [secrets-dir]
-#
-# <secrets-dir> is the output directory from sso-mfa/bootstrap/gen-secrets.sh
-# (default: ../../bootstrap/secrets).
-#
-# Creates ONE Secret in the sso namespace:
-#   authelia-secrets — all Authelia sensitive values as named files:
-#     jwt_secret                 ← AUTHELIA_JWT_SECRET_FILE
-#     session_secret             ← AUTHELIA_SESSION_SECRET_FILE
-#     storage_encryption_key     ← AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE
-#     ldap_password              ← AUTHELIA_AUTHENTICATION_BACKEND_LDAP_PASSWORD_FILE
-#     oidc_hmac_secret           ← AUTHELIA_IDENTITY_PROVIDERS_OIDC_HMAC_SECRET_FILE
-#     oidc_issuer_private_key    ← AUTHELIA_IDENTITY_PROVIDERS_OIDC_ISSUER_PRIVATE_KEY_FILE
-#     keycape_client_secret_hash ← AUTHELIA_IDENTITY_PROVIDERS_OIDC_CLIENTS_0_SECRET_FILE
-#
-# The keycape_client_secret_hash is the bcrypt hash of AUTHELIA_KEYCAPE_CLIENT_SECRET
-# from secrets/authelia/secrets.env. The plaintext is stored in KeyCape's secret.
-#
-# Requires: kubectl, openssl, python3 (for bcrypt hash generation)
-
+# Private creation or exact-match verification; rotations require --rotate-key.
+set +x
 set -euo pipefail
-
-SECRETS_DIR="${1:-../../bootstrap/secrets}"
-AUTHELIA_ENV="$SECRETS_DIR/authelia/secrets.env"
-LLDAP_ENV="$SECRETS_DIR/lldap/secrets.env"
-
-for f in "$AUTHELIA_ENV" "$LLDAP_ENV"; do
-    if [[ ! -f "$f" ]]; then
-        echo "ERROR: $f not found" >&2
-        echo "Run sso-mfa/bootstrap/gen-secrets.sh first." >&2
-        exit 1
-    fi
-done
-
-read_env() { bash -c "source '$1' 2>/dev/null; echo \${$2}"; }
-
-AUTHELIA_JWT_SECRET=$(read_env "$AUTHELIA_ENV" AUTHELIA_JWT_SECRET)
-AUTHELIA_SESSION_SECRET=$(read_env "$AUTHELIA_ENV" AUTHELIA_SESSION_SECRET)
-AUTHELIA_STORAGE_ENCRYPTION_KEY=$(read_env "$AUTHELIA_ENV" AUTHELIA_STORAGE_ENCRYPTION_KEY)
-AUTHELIA_OIDC_HMAC_SECRET=$(read_env "$AUTHELIA_ENV" AUTHELIA_OIDC_HMAC_SECRET)
-AUTHELIA_OIDC_PRIVATE_KEY=$(read_env "$AUTHELIA_ENV" AUTHELIA_OIDC_PRIVATE_KEY_FILE)
-AUTHELIA_KEYCAPE_CLIENT_SECRET=$(read_env "$AUTHELIA_ENV" AUTHELIA_KEYCAPE_CLIENT_SECRET)
-LLDAP_LDAP_USER_PASS=$(read_env "$LLDAP_ENV" LLDAP_LDAP_USER_PASS)
-
-# Validate required values
-REQUIRED=(AUTHELIA_JWT_SECRET AUTHELIA_SESSION_SECRET AUTHELIA_STORAGE_ENCRYPTION_KEY
-          AUTHELIA_OIDC_HMAC_SECRET AUTHELIA_KEYCAPE_CLIENT_SECRET LLDAP_LDAP_USER_PASS)
-for var in "${REQUIRED[@]}"; do
-    if [[ -z "${!var}" ]]; then
-        echo "ERROR: $var is empty — re-run gen-secrets.sh" >&2
-        exit 1
-    fi
-done
-
-# The OIDC issuer private key is stored as a file path in secrets.env.
-# Read the actual PEM content from the path referenced there.
-AUTHELIA_OIDC_PRIVATE_KEY_PATH=$(read_env "$AUTHELIA_ENV" AUTHELIA_OIDC_PRIVATE_KEY_FILE)
-if [[ -z "$AUTHELIA_OIDC_PRIVATE_KEY_PATH" ]]; then
-    # Fall back: generate a new RSA key on the fly (dev only)
-    echo "WARN: AUTHELIA_OIDC_PRIVATE_KEY_FILE not set — generating a new RSA-2048 key."
-    echo "      Store the generated key in KeePassXC as net-kingdom/Authelia/oidc-private-key."
-    TMP_KEY=$(mktemp)
-    openssl genrsa -out "$TMP_KEY" 2048 2>/dev/null
-    AUTHELIA_OIDC_PRIVATE_KEY_CONTENT=$(cat "$TMP_KEY")
-    shred -u "$TMP_KEY"
-elif [[ -f "$AUTHELIA_OIDC_PRIVATE_KEY_PATH" ]]; then
-    AUTHELIA_OIDC_PRIVATE_KEY_CONTENT=$(cat "$AUTHELIA_OIDC_PRIVATE_KEY_PATH")
-else
-    echo "ERROR: OIDC private key file not found: $AUTHELIA_OIDC_PRIVATE_KEY_PATH" >&2
-    exit 1
-fi
-
-# Hash the Authelia-KeyCape client secret with bcrypt (cost 12).
-# Authelia requires the bcrypt hash for OIDC client secrets.
-echo "Hashing KeyCape client secret with bcrypt (this may take a moment)..."
-if command -v python3 &>/dev/null && python3 -c "import bcrypt" 2>/dev/null; then
-    KEYCAPE_CLIENT_SECRET_HASH=$(python3 -c "
-import bcrypt, sys
-pw = sys.argv[1].encode()
-print(bcrypt.hashpw(pw, bcrypt.gensalt(rounds=12)).decode())
-" "$AUTHELIA_KEYCAPE_CLIENT_SECRET")
-elif command -v htpasswd &>/dev/null; then
-    KEYCAPE_CLIENT_SECRET_HASH=$(htpasswd -nbBC 12 "" "$AUTHELIA_KEYCAPE_CLIENT_SECRET" | cut -d: -f2)
-else
-    echo "ERROR: bcrypt hash generation requires python3+bcrypt or apache2-utils (htpasswd)" >&2
-    echo "  Install: pip3 install bcrypt  OR  apt install apache2-utils" >&2
-    exit 1
-fi
-
-echo "Creating K8s Secret: authelia-secrets (namespace: sso)"
-kubectl create secret generic authelia-secrets \
-    --namespace=sso \
-    --from-literal=jwt_secret="$AUTHELIA_JWT_SECRET" \
-    --from-literal=session_secret="$AUTHELIA_SESSION_SECRET" \
-    --from-literal=storage_encryption_key="$AUTHELIA_STORAGE_ENCRYPTION_KEY" \
-    --from-literal=ldap_password="$LLDAP_LDAP_USER_PASS" \
-    --from-literal=oidc_hmac_secret="$AUTHELIA_OIDC_HMAC_SECRET" \
-    --from-literal=oidc_issuer_private_key="$AUTHELIA_OIDC_PRIVATE_KEY_CONTENT" \
-    --from-literal=keycape_client_secret_hash="$KEYCAPE_CLIENT_SECRET_HASH" \
-    --dry-run=client -o yaml | kubectl apply -f -
-
-echo ""
-echo "Done. Secret authelia-secrets created in namespace: sso"
-echo ""
-echo "Next:"
-echo "  Apply deployment.yaml, then ingress.yaml."
-echo "  The plaintext Authelia-KeyCape client secret is in secrets/authelia/secrets.env"
-echo "  as AUTHELIA_KEYCAPE_CLIENT_SECRET. This value goes into keycape/create-secrets.sh"
-echo "  as the authelia.clientSecret in the KeyCape config."
+umask 077
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+exec python3 "$SCRIPT_DIR/../../bootstrap/apply-service-secrets.py" authelia "$@"

sso-mfa/k8s/lldap/create-secrets.sh

@@ -1,57 +1,7 @@
 #!/usr/bin/env bash
-# create-secrets.sh — create the lldap-secrets K8s Secret
-#
-# Usage:
-#   ./create-secrets.sh [secrets-dir]
-#
-# <secrets-dir> is the output directory from sso-mfa/bootstrap/gen-secrets.sh
-# (default: ../../bootstrap/secrets).
-#
-# Creates ONE Secret in the sso namespace:
-#   lldap-secrets — LLDAP_JWT_SECRET, LLDAP_LDAP_USER_PASS
-#
-# LLDAP_LDAP_USER_PASS is also used as the LDAP bind password
-# by Authelia (authelia/create-secrets.sh) and KeyCape (keycape/create-secrets.sh).
-# All three read the same value from secrets/lldap/secrets.env.
-
+# Private creation or exact-match verification; rotations require --rotate-key.
+set +x
 set -euo pipefail
-
-SECRETS_DIR="${1:-../../bootstrap/secrets}"
-LLDAP_ENV="$SECRETS_DIR/lldap/secrets.env"
-
-if [[ ! -d "$SECRETS_DIR" ]]; then
-    echo "ERROR: secrets directory not found: $SECRETS_DIR" >&2
-    echo "Run sso-mfa/bootstrap/gen-secrets.sh first." >&2
-    exit 1
-fi
-
-if [[ ! -f "$LLDAP_ENV" ]]; then
-    echo "ERROR: $LLDAP_ENV not found" >&2
-    echo "If you ran gen-secrets.sh before the KeyCape migration, re-run it to add LLDAP secrets." >&2
-    exit 1
-fi
-
-LLDAP_JWT_SECRET=$(bash -c "source '$LLDAP_ENV' 2>/dev/null; echo \$LLDAP_JWT_SECRET")
-LLDAP_LDAP_USER_PASS=$(bash -c "source '$LLDAP_ENV' 2>/dev/null; echo \$LLDAP_LDAP_USER_PASS")
-
-if [[ -z "$LLDAP_JWT_SECRET" || -z "$LLDAP_LDAP_USER_PASS" ]]; then
-    echo "ERROR: could not read LLDAP_JWT_SECRET or LLDAP_LDAP_USER_PASS from $LLDAP_ENV" >&2
-    exit 1
-fi
-
-echo "Creating K8s Secret: lldap-secrets (namespace: sso)"
-kubectl create secret generic lldap-secrets \
-    --namespace=sso \
-    --from-literal=LLDAP_JWT_SECRET="$LLDAP_JWT_SECRET" \
-    --from-literal=LLDAP_LDAP_USER_PASS="$LLDAP_LDAP_USER_PASS" \
-    --dry-run=client -o yaml | kubectl apply -f -
-
-echo ""
-echo "Done. Secret lldap-secrets created in namespace: sso"
-echo ""
-echo "Next:"
-echo "  Apply manifests (see README.md apply order)."
-echo "  After LLDAP is Running, seed groups:"
-echo "    ./bootstrap-users.sh"
-echo "    Groups: net-kingdom-users, net-kingdom-admins, activity-core-operators"
-echo "    Membership runbook: OPERATOR-GROUPS.md"
+umask 077
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+exec python3 "$SCRIPT_DIR/../../bootstrap/apply-service-secrets.py" lldap "$@"

sso-mfa/k8s/postgresql/README.md

@@ -141,5 +141,7 @@ chmod +x ../verify-t03.sh
   Do not rename the cluster without also updating netpol-databases.yaml.
 - `instances: 1` is intentional for dev/staging. Change to 3 before ThreePhoenix HA production
   deployment (requires at least 3 schedulable nodes).
-- Password rotation: update the K8s Secret values and CNPG's managed.roles reconciler will
-  apply the change at the next reconciliation cycle (within seconds).
+- Password rotation uses the explicit `PI_DB_PASSWORD` selector under the reviewed
+  procedure. CNPG's managed role and the privacyIDEA consumer URI must converge
+  on the same candidate; verify actual reconciliation and authentication before
+  calling it complete. See [service Secret custody](../../bootstrap/README.md#service-secret-creation-and-explicit-rotation).

sso-mfa/k8s/postgresql/create-secrets.sh

@@ -1,57 +1,7 @@
 #!/usr/bin/env bash
-# create-secrets.sh — create K8s Secrets for PostgreSQL from gen-secrets.sh output
-#
-# Usage:
-#   ./create-secrets.sh <secrets-dir>
-#
-# <secrets-dir> is the output directory produced by sso-mfa/bootstrap/gen-secrets.sh
-# (default: ../../bootstrap/secrets).
-#
-# Creates one K8s Secret in the databases namespace:
-#   net-kingdom-pg-privacyidea-app    — privacyIDEA DB credentials
-#
-# Note: net-kingdom-pg-keycloak-app removed — Keycloak replaced by Authelia+LLDAP+KeyCape (T05).
-#
-# These secrets must exist before applying cluster.yaml.
-# Re-run this script whenever you rotate passwords in KeePassXC / gen-secrets.sh.
-
+# Private creation or exact-match verification; rotations require --rotate-key.
+set +x
 set -euo pipefail
-
-SECRETS_DIR="${1:-../../bootstrap/secrets}"
-
-if [[ ! -d "$SECRETS_DIR" ]]; then
-    echo "ERROR: secrets directory not found: $SECRETS_DIR" >&2
-    echo "Run sso-mfa/bootstrap/gen-secrets.sh first, then re-run this script." >&2
-    exit 1
-fi
-
-PI_SECRETS="$SECRETS_DIR/privacyidea/secrets.env"
-
-if [[ ! -f "$PI_SECRETS" ]]; then
-    echo "ERROR: $PI_SECRETS not found" >&2
-    exit 1
-fi
-
-# Source the generated env file (KEY=VALUE pairs, no export)
-# Use a subshell to avoid polluting the current environment.
-PI_DB_PASS=$(bash -c "source $PI_SECRETS 2>/dev/null; echo \$PI_DB_PASSWORD")
-
-if [[ -z "$PI_DB_PASS" ]]; then
-    echo "ERROR: could not read PI_DB_PASSWORD from $PI_SECRETS" >&2
-    echo "Check that gen-secrets.sh ran successfully and the file is intact." >&2
-    exit 1
-fi
-
-echo "Creating K8s Secret: net-kingdom-pg-privacyidea-app"
-kubectl create secret generic net-kingdom-pg-privacyidea-app \
-    --namespace=databases \
-    --from-literal=username=privacyidea \
-    --from-literal=password="$PI_DB_PASS" \
-    --dry-run=client -o yaml | kubectl apply -f -
-
-echo ""
-echo "Done. Secret created in namespace: databases"
-echo ""
-echo "Verify:"
-echo "  kubectl get secrets -n databases"
-echo "  kubectl describe secret net-kingdom-pg-privacyidea-app -n databases"
+umask 077
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+exec python3 "$SCRIPT_DIR/../../bootstrap/apply-service-secrets.py" postgresql "$@"

sso-mfa/k8s/privacyidea/README.md

@@ -57,7 +57,7 @@ kubectl logs -n mfa -l app.kubernetes.io/name=privacyidea --previous
 Common causes:
 - `privacyidea-config` Secret missing → run `create-secrets.sh`
 - PostgreSQL not reachable → verify T03, check NetworkPolicies
-- Wrong DB password → re-run `create-secrets.sh` with corrected secrets
+- Wrong DB password → reconcile the database, CNPG application Secret and privacyIDEA URI through the reviewed named `PI_DB_PASSWORD` rotation; ordinary reapplication refuses differing values. See [service Secret custody](../../bootstrap/README.md#service-secret-creation-and-explicit-rotation).
 
 ---
 

sso-mfa/k8s/privacyidea/create-secrets.sh

@@ -1,65 +1,7 @@
 #!/usr/bin/env bash
-# create-secrets.sh — create the privacyidea-config K8s Secret
-#
-# Usage:
-#   ./create-secrets.sh [secrets-dir]
-#
-# <secrets-dir> is the output directory from sso-mfa/bootstrap/gen-secrets.sh
-# (default: ../../bootstrap/secrets).
-#
-# Creates ONE Secret in the mfa namespace:
-#   privacyidea-config   — PI_SECRET_KEY, PI_PEPPER, PI_SQLALCHEMY_DATABASE_URI
-#
-# This secret must exist before applying deployment.yaml.
-#
-# The enckey and auditkey Secrets (privacyidea-enckey, privacyidea-auditkeys)
-# are created separately by enckey-bootstrap.sh AFTER the first pod start,
-# because those keys are auto-generated by the container on first run.
-#
-# Re-run this script if you rotate PI_SECRET_KEY or PI_PEPPER in KeePassXC.
-
+# Private creation or exact-match verification; rotations require --rotate-key.
+set +x
 set -euo pipefail
-
-SECRETS_DIR="${1:-../../bootstrap/secrets}"
-PI_ENV="$SECRETS_DIR/privacyidea/secrets.env"
-
-if [[ ! -d "$SECRETS_DIR" ]]; then
-    echo "ERROR: secrets directory not found: $SECRETS_DIR" >&2
-    echo "Run sso-mfa/bootstrap/gen-secrets.sh first." >&2
-    exit 1
-fi
-
-if [[ ! -f "$PI_ENV" ]]; then
-    echo "ERROR: $PI_ENV not found" >&2
-    exit 1
-fi
-
-# Read values from the generated env file in a subshell to avoid polluting env.
-PI_SECRET_KEY=$(bash -c "source '$PI_ENV' 2>/dev/null; echo \$PI_SECRET_KEY")
-PI_PEPPER=$(bash -c "source '$PI_ENV' 2>/dev/null; echo \$PI_PEPPER")
-PI_DB_PASSWORD=$(bash -c "source '$PI_ENV' 2>/dev/null; echo \$PI_DB_PASSWORD")
-
-if [[ -z "$PI_SECRET_KEY" || -z "$PI_PEPPER" || -z "$PI_DB_PASSWORD" ]]; then
-    echo "ERROR: could not read PI_SECRET_KEY, PI_PEPPER, or PI_DB_PASSWORD from $PI_ENV" >&2
-    echo "Check that gen-secrets.sh ran successfully." >&2
-    exit 1
-fi
-
-# Construct the SQLAlchemy database URI.
-# CloudNativePG read-write service: net-kingdom-pg-rw.databases.svc.cluster.local
-PI_DB_URI="postgresql://privacyidea:${PI_DB_PASSWORD}@net-kingdom-pg-rw.databases.svc.cluster.local:5432/privacyidea_db"
-
-echo "Creating K8s Secret: privacyidea-config (namespace: mfa)"
-kubectl create secret generic privacyidea-config \
-    --namespace=mfa \
-    --from-literal=PI_SECRET_KEY="$PI_SECRET_KEY" \
-    --from-literal=PI_PEPPER="$PI_PEPPER" \
-    --from-literal=PI_SQLALCHEMY_DATABASE_URI="$PI_DB_URI" \
-    --dry-run=client -o yaml | kubectl apply -f -
-
-echo ""
-echo "Done. Secret privacyidea-config created in namespace: mfa"
-echo ""
-echo "Next:"
-echo "  Apply manifests (see README.md apply order)."
-echo "  After the pod is Running, run: ./enckey-bootstrap.sh"
+umask 077
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+exec python3 "$SCRIPT_DIR/../../bootstrap/apply-service-secrets.py" privacyidea "$@"

tests/test_rotation_output.py

@@ -26,7 +26,7 @@ def rotation(tmp_path):
     shutil.copyfile(ROOT / 'Makefile', repo / 'Makefile')
     (boot / 'creds-state.yaml').write_text('version: 2\n')
     secrets = tmp_path / 'private'; secrets.mkdir(mode=0o700)
-    for component in ('privacyidea', 'lldap', 'authelia', 'keycape', 'breakglass'):
+    for component in ('postgresql', 'privacyidea', 'lldap', 'authelia', 'keycape', 'breakglass'):
         directory = secrets / component; directory.mkdir(mode=0o700)
         entries = [f'{key}=old-value' for key, value in CASES.items() if value == component]
         if component == 'lldap': entries.append('LLDAP_LDAP_USER_PASS=old-value')
@@ -34,7 +34,10 @@ def rotation(tmp_path):
         env_file.write_text('\n'.join(entries) + '\n'); env_file.chmod(0o600)
         helper = repo / f'sso-mfa/k8s/{component}'; helper.mkdir(parents=True)
         (helper / 'create-secrets.sh').write_text(
-            '#!/bin/bash\nprintf "%s\\n" "$SYNTHETIC" >&2\nexit "${HELPER_FAIL:-0}"\n')
+            '#!/bin/bash\npython3 - "$@" <<\'PYTRACE\'\n'
+            'import json,os,sys\nfrom pathlib import Path\n'
+            'with open(os.environ["HELPERS"],"a") as f:f.write(json.dumps({"component":Path.cwd().name,"args":sys.argv[1:]})+"\\n")\n'
+            'PYTRACE\nprintf "%s\\n" "$SYNTHETIC" >&2\nexit "${HELPER_FAIL:-0}"\n')
     binary = tmp_path / 'bin'; binary.mkdir()
     (binary / 'openssl').write_text('#!/bin/sh\nprintf "%s\\n" "$SYNTHETIC"\n')
     (binary / 'kubectl').write_text('''#!/usr/bin/python3
@@ -52,7 +55,7 @@ else:
     for command in binary.iterdir(): command.chmod(0o700)
     env = {'PATH': str(binary) + ':' + os.environ['PATH'], 'HOME': str(tmp_path),
            'KUBECONFIG': str(tmp_path / 'nonexistent'), 'CALLS': str(tmp_path / 'calls'),
-           'SYNTHETIC': SYNTHETIC}
+           'SYNTHETIC': SYNTHETIC, 'HELPERS': str(tmp_path / 'helpers')}
     return repo, boot, secrets, env
 
 
@@ -185,7 +188,7 @@ def test_declining_rotation_preserves_custody_file_and_cluster(rotation):
     assert bool(SYNTHETIC[:40] in result.stdout + result.stderr) is False
 
 
-@pytest.mark.parametrize('key',['AUTHELIA_KEYCAPE_CLIENT_SECRET','KEYCAPE_RSA_KEY'])
+@pytest.mark.parametrize('key',['AUTHELIA_KEYCAPE_CLIENT_SECRET','KEYCAPE_RSA_KEY','LLDAP_LDAP_USER_PASS'])
 @pytest.mark.parametrize('attended',[False,True])
 def test_retired_keycape_replacement_refuses_before_any_credential_change(rotation,key,attended):
     repo,boot,secrets,env=rotation
@@ -199,3 +202,25 @@ def test_retired_keycape_replacement_refuses_before_any_credential_change(rotati
     assert {p:p.read_bytes() for p in secrets.rglob('*') if p.is_file()}==before
     assert 'last_rotated_' not in (boot/'creds-state.yaml').read_text()
     assert 'NK-WP-0049-T02' in result.stderr
+
+
+@pytest.mark.parametrize('key,components',[
+    ('PI_SECRET_KEY',['privacyidea']),
+    ('PI_DB_PASSWORD',['postgresql','privacyidea']),
+    ('LLDAP_JWT_SECRET',['lldap']),
+    ('AUTHELIA_SESSION_SECRET',['authelia']),
+])
+def test_rotation_forwards_exact_selector_to_every_secret_consumer(rotation,key,components):
+    result=run(rotation,key)
+    assert result.returncode==0
+    observed=[json.loads(line) for line in Path(rotation[3]['HELPERS']).read_text().splitlines()]
+    assert observed==[{'component':component,'args':[str(rotation[2]),'--rotate-key',key]} for component in components]
+
+
+def test_database_secret_failure_stops_before_other_consumer_or_restart(rotation):
+    result=run(rotation,'PI_DB_PASSWORD',HELPER_FAIL='1')
+    assert result.returncode!=0
+    assert [json.loads(line)['component'] for line in Path(rotation[3]['HELPERS']).read_text().splitlines()]==['postgresql']
+    assert [call['args'][0] for call in calls(rotation)]==['get','exec']
+    assert 'last_rotated_' not in (rotation[1]/'creds-state.yaml').read_text()
+    assert 'PI_DB_PASSWORD='+SYNTHETIC[:40] in (rotation[2]/'privacyidea/secrets.env').read_text()

tests/test_service_apply_caller.py

@@ -0,0 +1,41 @@
+"""Batch transport tests use synthetic noisy helpers and no real cluster."""
+import json
+import os
+from pathlib import Path
+import shutil
+import subprocess
+
+import pytest
+
+ROOT=Path(__file__).resolve().parents[1]
+MARKER='SyntheticChildOutputNeverInParentTranscript'
+
+
+@pytest.mark.parametrize('failed',[None,'authelia'])
+def test_batch_captures_children_and_only_records_successful_components(tmp_path,failed):
+    repo=tmp_path/'repo';boot=repo/'sso-mfa/bootstrap';boot.mkdir(parents=True)
+    shutil.copyfile(ROOT/'sso-mfa/bootstrap/creds-apply.sh',boot/'creds-apply.sh')
+    state=boot/'creds-state.yaml';state.write_text('secrets_applied:\n  postgres: false\n  lldap: false\n  authelia: false\n  privacyidea: false\n')
+    private=tmp_path/'private with spaces';private.mkdir(mode=0o700)
+    for name in ('lldap','privacyidea'):
+        p=private/name;p.mkdir(mode=0o700);(p/'secrets.env').write_text('# no actual value needed for transport fixture\n')
+    binary=tmp_path/'bin';binary.mkdir();(binary/'kubectl').write_text('#!/bin/sh\nprintf "%s\\n" "$MARKER"\nexit 0\n');(binary/'kubectl').chmod(0o700)
+    for component in ('postgresql','lldap','authelia','privacyidea'):
+        p=repo/f'sso-mfa/k8s/{component}';p.mkdir(parents=True)
+        (p/'create-secrets.sh').write_text('''#!/bin/bash
+printf '%s\\n' "$MARKER"
+printf '%s\\n' "$MARKER" >&2
+python3 - "$1" <<'PYTHON'
+import json,os,sys
+with open(os.environ['ARGS'],'a') as file:file.write(json.dumps(sys.argv[1:])+'\\n')
+PYTHON
+'''+ ('exit 1\n' if component==failed else 'exit 0\n'))
+    env={'PATH':str(binary)+':'+os.environ['PATH'],'HOME':str(tmp_path),'KUBECONFIG':str(tmp_path/'unused'),'ARGS':str(tmp_path/'args'),'MARKER':MARKER}
+    result=subprocess.run(['bash','-x',str(boot/'creds-apply.sh'),private.name],cwd=tmp_path,env=env,capture_output=True,text=True,timeout=10)
+    assert (result.returncode==0)==(failed is None)
+    assert MARKER not in result.stdout+result.stderr
+    args=[json.loads(line) for line in Path(env['ARGS']).read_text().splitlines()]
+    assert args==[[str(private)]]*(4 if failed is None else 3)
+    recorded=state.read_text();assert 'postgres: true' in recorded and 'lldap: true' in recorded
+    assert ('authelia: true' in recorded)==(failed is None)
+    assert ('privacyidea: true' in recorded)==(failed is None)

tests/test_service_secret_custody.py

@@ -0,0 +1,276 @@
+"""Service helpers against synthetic files and a recording Kubernetes API stub."""
+import base64
+import json
+import os
+from pathlib import Path
+import subprocess
+
+import bcrypt
+import pytest
+
+ROOT = Path(__file__).resolve().parents[1]
+PASSWORD = 'SyntheticServiceCredential12345678901234567890'
+CHANGED = 'ReplacementSyntheticCredential12345678901234567890'
+VALUES = {
+    'privacyidea': ['PI_SECRET_KEY', 'PI_PEPPER', 'PI_DB_PASSWORD'],
+    'lldap': ['LLDAP_JWT_SECRET', 'LLDAP_LDAP_USER_PASS'],
+    'authelia': ['AUTHELIA_JWT_SECRET', 'AUTHELIA_SESSION_SECRET', 'AUTHELIA_STORAGE_ENCRYPTION_KEY',
+                 'AUTHELIA_OIDC_HMAC_SECRET', 'AUTHELIA_KEYCAPE_CLIENT_SECRET'],
+}
+COMPONENTS = ['postgresql', 'privacyidea', 'lldap', 'authelia']
+ROTATIONS = [('postgresql','PI_DB_PASSWORD','password'), ('privacyidea','PI_SECRET_KEY','PI_SECRET_KEY'),
+             ('privacyidea','PI_DB_PASSWORD','PI_SQLALCHEMY_DATABASE_URI'),
+             ('lldap','LLDAP_JWT_SECRET','LLDAP_JWT_SECRET'), ('authelia','AUTHELIA_SESSION_SECRET','session_secret')]
+
+
+@pytest.fixture
+def case(tmp_path):
+    private=tmp_path/'private';private.mkdir(mode=0o700)
+    for component,names in VALUES.items():
+        directory=private/component;directory.mkdir(mode=0o700)
+        env=directory/'secrets.env';env.write_text(''.join(f'{name}={PASSWORD}\n' for name in names));env.chmod(0o600)
+    binary=tmp_path/'bin';binary.mkdir()
+    (binary/'kubectl').write_text('''#!/usr/bin/python3
+import json,os,sys
+from pathlib import Path
+args=sys.argv[1:]
+raw=sys.stdin.read() if '-f' in args or '--patch-file=/dev/stdin' in args else ''
+with open(os.environ['CALLS'],'a') as f:f.write(json.dumps({'args':args,'stdin':raw})+'\\n')
+if os.environ.get('FAIL')==args[0]:
+ print(os.environ['MARKER']);print(os.environ['MARKER'],file=sys.stderr);sys.exit(1)
+p=Path(os.environ['STATE'])
+if args[0]=='get':
+ if os.environ.get('BAD_GET'):print(os.environ['BAD_GET'])
+ elif p.exists():print(p.read_text())
+ sys.exit(0)
+if args[0]=='create' and raw:
+ if p.exists() or os.environ.get('RACE'):sys.exit(1)
+ state=json.loads(raw);state['metadata'].update(uid='fixture-uid',resourceVersion='1')
+elif args[0]=='patch':
+ if os.environ.get('RACE'):sys.exit(1)
+ state=json.loads(p.read_text())
+ for op in json.loads(raw):
+  parts=[s.replace('~1','/').replace('~0','~') for s in op['path'].split('/')[1:]]
+  parent=state
+  for part in parts[:-1]:parent=parent[part]
+  if op['op']=='test':
+   if parent.get(parts[-1])!=op['value']:sys.exit(1)
+  elif op['op']=='remove':del parent[parts[-1]]
+  elif op['op']=='replace':parent[parts[-1]]=op['value']
+  else:sys.exit(1)
+ state['metadata']['resourceVersion']=str(int(state['metadata']['resourceVersion'])+1)
+else:sys.exit(2)
+p.write_text(json.dumps(state))
+if os.environ.get('BAD_RESPONSE'):state['data']['unexpected']='wrong'
+print(json.dumps(state))
+''')
+    (binary/'kubectl').chmod(0o700)
+    env={'PATH':str(binary)+':'+os.environ['PATH'],'HOME':str(tmp_path),
+         'CALLS':str(tmp_path/'calls'),'STATE':str(tmp_path/'state'),'MARKER':PASSWORD}
+    return private,env
+
+
+def invoke(case,component,rotate=None,**extra):
+    args=['bash','-x',str(ROOT/f'sso-mfa/k8s/{component}/create-secrets.sh'),str(case[0])]
+    if rotate:args+=['--rotate-key',rotate]
+    return subprocess.run(args,env={**case[1],**extra},capture_output=True,text=True,timeout=20)
+
+
+def calls(case):
+    p=Path(case[1]['CALLS']);return [json.loads(line) for line in p.read_text().splitlines()] if p.exists() else []
+
+
+def state(case):return json.loads(Path(case[1]['STATE']).read_text())
+
+def put_state(case,obj):Path(case[1]['STATE']).write_text(json.dumps(obj))
+
+def check_private_result(result):
+    assert PASSWORD not in result.stdout+result.stderr
+    assert CHANGED not in result.stdout+result.stderr
+    assert 'BEGIN PRIVATE KEY' not in result.stdout+result.stderr
+
+
+def source(component):return 'privacyidea' if component=='postgresql' else component
+
+
+@pytest.mark.parametrize('component',COMPONENTS)
+def test_create_uses_literal_inputs_stdin_and_no_apply_annotation(case,component):
+    result=invoke(case,component);assert result.returncode==0,result.stderr;check_private_result(result)
+    assert PASSWORD not in json.dumps([c['args'] for c in calls(case)])
+    writes=[c for c in calls(case) if c['args'][0]!='get']
+    assert len(writes)==1 and writes[0]['args']==['create','-o','json','-f','-']
+    obj=state(case);assert 'annotations' not in obj['metadata']
+    values={k:base64.b64decode(v).decode() for k,v in obj['data'].items()}
+    if component=='postgresql':assert values=={'username':'privacyidea','password':PASSWORD}
+    elif component=='privacyidea':
+        assert values['PI_PEPPER']==PASSWORD and values['PI_SECRET_KEY']==PASSWORD
+        assert values['PI_SQLALCHEMY_DATABASE_URI']==f'postgresql://privacyidea:{PASSWORD}@net-kingdom-pg-rw.databases.svc.cluster.local:5432/privacyidea_db'
+    elif component=='lldap':assert values=={k:PASSWORD for k in VALUES['lldap']}
+    else:
+        key=case[0]/'authelia/oidc_private_key.pem'
+        assert key.stat().st_mode & 0o077==0 and values['oidc_issuer_private_key']==key.read_text()
+        assert bcrypt.checkpw(PASSWORD.encode(),values['keycape_client_secret_hash'].encode())
+        assert values['ldap_password']==PASSWORD
+
+
+@pytest.mark.parametrize('component',COMPONENTS)
+def test_repeated_apply_is_noop_and_preserves_other_data(case,component):
+    assert invoke(case,component).returncode==0
+    old=state(case);old['data']['owner-added']='c3ludGhldGlj';old['metadata']['annotations']={'owner':'retained'};put_state(case,old)
+    count=len(calls(case));result=invoke(case,component)
+    assert result.returncode==0;check_private_result(result)
+    assert state(case)==old
+    assert all(c['args'][0]=='get' for c in calls(case)[count:])
+
+
+@pytest.mark.parametrize('component,key,field',ROTATIONS)
+def test_explicit_rotation_is_conditional_and_changes_only_named_field(case,component,key,field):
+    assert invoke(case,component).returncode==0
+    old=state(case);old['data']['owner-added']='c3ludGhldGlj';old['metadata']['annotations']={'owner':'retained','kubectl.kubernetes.io/last-applied-configuration':'synthetic-old-copy'};put_state(case,old)
+    file=case[0]/source(component)/'secrets.env';file.write_text(file.read_text().replace(f'{key}={PASSWORD}',f'{key}={CHANGED}'))
+    result=invoke(case,component,rotate=key);assert result.returncode==0,result.stderr;check_private_result(result)
+    updated=state(case)
+    assert updated['metadata']['annotations']=={'owner':'retained'}
+    assert updated['data'][field]!=old['data'][field]
+    assert {k:v for k,v in updated['data'].items() if k!=field}=={k:v for k,v in old['data'].items() if k!=field}
+    patch=next(c for c in reversed(calls(case)) if c['args'][0]=='patch')
+    tests={op['path']:op['value'] for op in json.loads(patch['stdin']) if op['op']=='test'}
+    assert tests['/metadata/uid']==old['metadata']['uid'] and tests['/metadata/resourceVersion']==old['metadata']['resourceVersion']
+    assert PASSWORD not in json.dumps(patch['args']) and CHANGED not in json.dumps(patch['args'])
+
+
+@pytest.mark.parametrize('component',COMPONENTS)
+def test_differing_values_require_explicit_rotation(case,component):
+    assert invoke(case,component).returncode==0;old=state(case)
+    file=case[0]/source(component)/'secrets.env';file.write_text(file.read_text().replace(PASSWORD,CHANGED))
+    count=len(calls(case));result=invoke(case,component)
+    assert result.returncode!=0;check_private_result(result);assert state(case)==old
+    assert all(c['args'][0]=='get' for c in calls(case)[count:])
+
+
+@pytest.mark.parametrize('component',COMPONENTS)
+@pytest.mark.parametrize('defect',['shell','duplicate','public','symlink','hardlink','public-parent'])
+def test_bad_private_input_never_reaches_cluster(case,component,defect):
+    file=case[0]/source(component)/'secrets.env';marker=case[0]/'executed'
+    if defect=='shell':file.write_text(file.read_text()+f'$(touch {marker})\n')
+    elif defect=='duplicate':file.write_text(file.read_text()*2)
+    elif defect=='public':file.chmod(0o644)
+    elif defect=='public-parent':file.parent.chmod(0o755)
+    elif defect=='hardlink':os.link(file,file.with_name('alias'))
+    else:
+        target=file.with_name('other');file.rename(target);file.symlink_to(target)
+    result=invoke(case,component);assert result.returncode!=0;check_private_result(result)
+    assert not marker.exists() and not calls(case)
+
+
+@pytest.mark.parametrize('component',COMPONENTS)
+@pytest.mark.parametrize('failure',['get','create'])
+def test_failed_child_diagnostics_never_escape(case,component,failure):
+    result=invoke(case,component,FAIL=failure);assert result.returncode!=0;check_private_result(result)
+    assert not Path(case[1]['STATE']).exists()
+
+
+@pytest.mark.parametrize('defect',['owner','eso-label','identity','type','basic-auth','data','immutable','missing-uid'])
+def test_wrong_or_controller_owned_existing_secret_is_refused(case,defect):
+    assert invoke(case,'lldap').returncode==0;old=state(case)
+    if defect=='owner':old['metadata']['ownerReferences']=[{'kind':'ExternalSecret','name':'fixture'}]
+    elif defect=='eso-label':old['metadata']['labels']={'reconcile.external-secrets.io/managed':'true'}
+    elif defect=='missing-uid':old['metadata']['uid']=None
+    elif defect=='identity':old['metadata']['namespace']='other'
+    elif defect=='type':old['type']='kubernetes.io/tls'
+    elif defect=='basic-auth':old['type']='kubernetes.io/basic-auth'
+    elif defect=='data':old['data']['LLDAP_JWT_SECRET']='invalid-base64'
+    else:old['immutable']=True
+    put_state(case,old);count=len(calls(case));result=invoke(case,'lldap')
+    assert result.returncode!=0 and state(case)==old
+    assert all(c['args'][0]=='get' for c in calls(case)[count:])
+
+
+@pytest.mark.parametrize('component,key,field',ROTATIONS)
+def test_concurrent_change_stops_rotation_without_retry(case,component,key,field):
+    assert invoke(case,component).returncode==0;old=state(case)
+    file=case[0]/source(component)/'secrets.env';file.write_text(file.read_text().replace(f'{key}={PASSWORD}',f'{key}={CHANGED}'))
+    count=len(calls(case));result=invoke(case,component,rotate=key,RACE='1')
+    assert result.returncode!=0 and state(case)==old;check_private_result(result)
+    assert sum(c['args'][0]=='patch' for c in calls(case)[count:])==1
+
+
+def test_create_race_never_switches_to_overwrite(case):
+    result=invoke(case,'lldap',RACE='1');assert result.returncode!=0
+    assert [c['args'][0] for c in calls(case)]==['get','create']
+
+
+def test_authelia_failure_retains_same_key_on_retry(case):
+    result=invoke(case,'authelia',FAIL='create');assert result.returncode!=0
+    key=case[0]/'authelia/oidc_private_key.pem';original=key.read_bytes()
+    assert invoke(case,'authelia').returncode==0 and key.read_bytes()==original
+
+
+def test_rotating_session_never_generates_or_replaces_existing_signing_key(case):
+    assert invoke(case,'authelia').returncode==0;old=state(case)
+    (case[0]/'authelia/oidc_private_key.pem').unlink()
+    file=case[0]/'authelia/secrets.env';file.write_text(file.read_text().replace(f'AUTHELIA_SESSION_SECRET={PASSWORD}',f'AUTHELIA_SESSION_SECRET={CHANGED}'))
+    assert invoke(case,'authelia',rotate='AUTHELIA_SESSION_SECRET').returncode==0
+    assert state(case)['data']['oidc_issuer_private_key']==old['data']['oidc_issuer_private_key']
+    assert not (case[0]/'authelia/oidc_private_key.pem').exists()
+
+
+def test_wrong_response_is_not_reported_as_success(case):
+    result=invoke(case,'lldap',BAD_RESPONSE='1');assert result.returncode!=0;check_private_result(result)
+
+
+@pytest.mark.parametrize('annotation',['', 'synthetic-old-copy'])
+def test_annotation_cleanup_preserves_secret_values_and_other_annotations(case,annotation):
+    assert invoke(case,'lldap').returncode==0;old=state(case)
+    old['metadata']['annotations']={'owner':'retained','kubectl.kubernetes.io/last-applied-configuration':annotation};put_state(case,old)
+    result=invoke(case,'lldap');assert result.returncode==0
+    assert state(case)['data']==old['data']
+    assert state(case)['metadata']['annotations']=={'owner':'retained'}
+
+
+@pytest.mark.parametrize('component,key,field',ROTATIONS)
+def test_rotation_refuses_missing_secret_before_create(case,component,key,field):
+    result=invoke(case,component,rotate=key);assert result.returncode!=0
+    assert all(c['args'][0]=='get' for c in calls(case))
+    assert not (case[0]/'authelia/oidc_private_key.pem').exists()
+
+
+def test_rotation_cannot_change_an_unrelated_field(case):
+    assert invoke(case,'privacyidea').returncode==0;old=state(case)
+    file=case[0]/'privacyidea/secrets.env';file.write_text(file.read_text().replace(PASSWORD,CHANGED))
+    result=invoke(case,'privacyidea',rotate='PI_SECRET_KEY');assert result.returncode!=0
+    assert state(case)==old
+    assert not any(c['args'][0]=='patch' for c in calls(case))
+
+
+@pytest.mark.parametrize('defect',['symlink','hardlink','public','public-parent','invalid'])
+def test_explicit_authelia_key_custody_is_validated_without_overwrite(case,defect):
+    key=case[0]/'authelia/external.pem';key.write_text('synthetic-invalid-key');key.chmod(0o600)
+    if defect=='symlink':
+        target=key.with_name('other');key.rename(target);key.symlink_to(target)
+    elif defect=='hardlink':os.link(key,key.with_name('alias'))
+    elif defect=='public':key.chmod(0o644)
+    elif defect=='public-parent':key.parent.chmod(0o755)
+    file=case[0]/'authelia/secrets.env';file.write_text(file.read_text()+f'AUTHELIA_OIDC_PRIVATE_KEY_FILE={key}\n')
+    result=invoke(case,'authelia');assert result.returncode!=0
+    assert key.read_text()=='synthetic-invalid-key'
+    assert not any(c['args'][0]!='get' for c in calls(case))
+
+
+def test_native_kubectl_applies_stdin_patch_and_refuses_stale_version(case,tmp_path):
+    import shutil
+    binary=shutil.which('kubectl')
+    if binary is None:pytest.skip('native kubectl not available')
+    assert invoke(case,'lldap').returncode==0;old=state(case)
+    file=case[0]/'lldap/secrets.env';file.write_text(file.read_text().replace(f'LLDAP_JWT_SECRET={PASSWORD}',f'LLDAP_JWT_SECRET={CHANGED}'))
+    assert invoke(case,'lldap',rotate='LLDAP_JWT_SECRET').returncode==0
+    patch=next(c['stdin'] for c in reversed(calls(case)) if c['args'][0]=='patch')
+    original=tmp_path/'original.json';original.write_text(json.dumps(old));original.chmod(0o600)
+    env={**os.environ,'KUBECONFIG':str(tmp_path/'no-cluster-config'),'HOME':str(tmp_path)}
+    command=[binary,'patch','--local','--type=json','--patch-file=/dev/stdin','-f',str(original),'-o','json']
+    result=subprocess.run(command,input=patch,env=env,capture_output=True,text=True,timeout=20)
+    assert result.returncode==0
+    assert json.loads(result.stdout)['data']==state(case)['data']
+    old['metadata']['resourceVersion']='999';original.write_text(json.dumps(old))
+    refused=subprocess.run(command,input=patch,env=env,capture_output=True,text=True,timeout=20)
+    assert refused.returncode!=0

workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md

@@ -83,8 +83,9 @@ Concrete uncovered callers remain (T02 stays progress):
 - RSA and Authelia-to-KeyCape client-secret rotation need configuration-preserving
   updates, retained rollback custody and native multi-consumer acceptance. The
   legacy replacement paths now refuse before mutation; this is not rotation closure.
-- Delegated create/apply helpers still need full tracing/error/Secret-annotation
-  output review; transport capture and native effect evidence remain mandatory.
+- PostgreSQL/privacyIDEA/LLDAP/Authelia generators and the batch apply caller
+  are repaired below. Other diagnostic/bootstrap/key-export callers and direct
+  SOPS edit/encrypt/rotate remain; native effect/recovery evidence is mandatory.
 
 2026-10-12 Make caller repair: sops-setup now validates only the public recipient;
 creds-init no longer requires or installs a permanent private key. sops-decrypt
@@ -151,6 +152,36 @@ current deployed consumer acceptance and other create/apply/SOPS callers remain
 open. Source verification and its limits: Custodian
 `docs/evidence/2026-10-13-netkingdom-security/keycape-caller-reconciliation.json`.
 
+2026-10-13 delegated service caller repair: the PostgreSQL, privacyIDEA,
+LLDAP and Authelia generators now share private literal-input parsing, captured
+child diagnostics and stdin-only Secret transfer. Ordinary invocation creates
+an absent object or verifies matching fields; it cannot silently rotate an
+existing object. Named rotation updates only its allowed field. UID/resource
+version/data tests guard patches; unknown data and unrelated metadata survive,
+and last-applied copies are removed under the same guard. Owner references,
+ESO management labels, immutable objects and wrong target identity are refused.
+The returned API object is checked before success; write failure remains an
+unknown remote outcome requiring reconciliation, never blind retry.
+
+Authelia first creation retains a private signing key atomically; API failure
+keeps it for recovery. Existing issuer keys and matching bcrypt hashes survive
+repeat apply and session rotation. Explicit unsafe/differing keys are refused.
+The batch caller suppresses child output and records only successful components.
+Rotation callers pass exact field selectors; database rotation now updates the
+CNPG application Secret as well as the privacyIDEA URI. The attended LDAP path
+also depended on retired full KeyCape replacement, so now refuses before any
+change. SQL, Secrets and restarts remain non-atomic; coordinated recovery and
+native database/consumer acceptance are still required.
+
+All 63 initial regression cases failed on the old helpers; the replacement and
+expanded caller tests pass, including real OpenSSL/bcrypt and actual kubectl
+local stdin patch/stale-version refusal with no cluster. Final source-bound
+suite results, dependency skips and test limitations are in Custodian
+`docs/evidence/2026-10-13-netkingdom-security/service-secret-callers.json`.
+No live Secret, signing key, credential or deployment was changed. T02 remains
+progress for the explicit callers/lifecycle work above; T03 remains waiting for
+its owner-scoped native recovery ceremony.
+
 ## Verify owner-approved recovery and current native callers
 
 ```task