feat(sso-mfa): T07/T08 user mgmt, backups, DR & break-glass (NK-WP-0001-T07/T08)
T07 — User management & self-service:
- k8s/lldap/bootstrap-users.sh: creates net-kingdom-users and net-kingdom-admins
groups in LLDAP via GraphQL API; idempotent.
- k8s/lldap/break-glass.sh: creates break-glass bypass account in LLDAP,
sets BREAKGLASS_PASSWORD, assigns to net-kingdom-admins.
- k8s/verify-t07.sh: 6 checks — groups, break-glass, self-service portal,
KeyCape OIDC client registrations.
T08 — Backups, DR, break-glass:
- k8s/backup/cronjob-sqlite-backups.yaml: daily CronJobs for LLDAP SQLite,
Authelia SQLite (with scale-down/up RBAC), and privacyIDEA enckey backup.
7-day retention, 03:00/03:15/03:30 UTC staggered schedule.
- k8s/backup/DR-RUNBOOK.md: full restore runbook — scenarios, restore order,
LLDAP/Authelia/PI SQLite restore procedure, full node rebuild sequence,
offsite age-encrypted export.
- k8s/verify-t08.sh: 9 checks — CronJobs, RBAC, run history, backup files
on PVCs, DR runbook presence, offsite backup (manual confirmation).
- WORKPLAN.md: T07/T08 sections with done-criteria added.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-19 09:17:03 +00:00
#!/usr/bin/env bash
# verify-t07.sh — verify NK-WP-0001-T07 done-criteria
#
# Checks user management and self-service readiness.
#
# Sections:
# 1. LLDAP group: net-kingdom-users exists
# 2. LLDAP group: net-kingdom-admins exists
# 3. At least one non-admin user exists in LLDAP
# 4. Break-glass user exists and is in net-kingdom-admins
# 5. privacyIDEA self-service portal reachable
2026-05-26 02:13:55 +02:00
# 6. KeyCape config has OIDC clients, including openbao-admin
feat(sso-mfa): T07/T08 user mgmt, backups, DR & break-glass (NK-WP-0001-T07/T08)
T07 — User management & self-service:
- k8s/lldap/bootstrap-users.sh: creates net-kingdom-users and net-kingdom-admins
groups in LLDAP via GraphQL API; idempotent.
- k8s/lldap/break-glass.sh: creates break-glass bypass account in LLDAP,
sets BREAKGLASS_PASSWORD, assigns to net-kingdom-admins.
- k8s/verify-t07.sh: 6 checks — groups, break-glass, self-service portal,
KeyCape OIDC client registrations.
T08 — Backups, DR, break-glass:
- k8s/backup/cronjob-sqlite-backups.yaml: daily CronJobs for LLDAP SQLite,
Authelia SQLite (with scale-down/up RBAC), and privacyIDEA enckey backup.
7-day retention, 03:00/03:15/03:30 UTC staggered schedule.
- k8s/backup/DR-RUNBOOK.md: full restore runbook — scenarios, restore order,
LLDAP/Authelia/PI SQLite restore procedure, full node rebuild sequence,
offsite age-encrypted export.
- k8s/verify-t08.sh: 9 checks — CronJobs, RBAC, run history, backup files
on PVCs, DR runbook presence, offsite backup (manual confirmation).
- WORKPLAN.md: T07/T08 sections with done-criteria added.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-19 09:17:03 +00:00
#
# Usage:
# chmod +x verify-t07.sh
# ./verify-t07.sh [lldap-url] [secrets-dir]
#
# <lldap-url> default: https://lldap.coulomb.social
# <secrets-dir> default: ../bootstrap/secrets
set -euo pipefail
LLDAP_URL = " ${ 1 :- https : //lldap.coulomb.social } "
SECRETS_DIR = " ${ 2 :- ../bootstrap/secrets } "
LLDAP_ENV = " $SECRETS_DIR /lldap/secrets.env "
SSO_NAMESPACE = "sso"
PASS = 0
FAIL = 0
WARN = 0
pass( ) { echo " [PASS] $1 " ; ( ( PASS++) ) ; }
fail( ) { echo " [FAIL] $1 " ; ( ( FAIL++) ) ; }
warn( ) { echo " [WARN] $1 " ; ( ( WARN++) ) ; }
section( ) { echo "" ; echo " ── $1 ────────────────────────────────────── " ; }
# ── Authenticate to LLDAP ─────────────────────────────────────────────────────
LLDAP_TOKEN = ""
if [ [ -f " $LLDAP_ENV " ] ] ; then
read_env( ) { bash -c " source ' $1 ' 2>/dev/null; echo \${ $2 } " ; }
LLDAP_ADMIN_PASS = $( read_env " $LLDAP_ENV " LLDAP_LDAP_USER_PASS)
if [ [ -n " $LLDAP_ADMIN_PASS " ] ] ; then
AUTH_RESP = $( curl -sf -X POST " $LLDAP_URL /auth/simple/login " \
-H "Content-Type: application/json" \
-d " {\"username\":\"admin\",\"password\":\" $LLDAP_ADMIN_PASS \"} " \
2>/dev/null || echo "CURL_FAILED" )
if [ [ " $AUTH_RESP " != "CURL_FAILED" ] ] ; then
LLDAP_TOKEN = $( echo " $AUTH_RESP " | python3 -c \
"import sys,json; print(json.load(sys.stdin).get('token',''))" 2>/dev/null || echo "" )
fi
fi
fi
gql( ) {
if [ [ -z " $LLDAP_TOKEN " ] ] ; then echo "NO_TOKEN" ; return ; fi
local query = " $1 " ; local vars = " ${ 2 :- { } } "
local body
body = $( python3 -c "
import json, sys
print( json.dumps( { 'query' : sys.argv[ 1] , 'variables' : json.loads( sys.argv[ 2] ) } ) )
" " $query " " $vars " )
curl -sf -X POST " $LLDAP_URL /api/graphql " \
-H " Authorization: Bearer $LLDAP_TOKEN " \
-H "Content-Type: application/json" \
-d " $body " 2>/dev/null || echo "CURL_FAILED"
}
GROUPS_RESP = $( gql 'query { groups { id displayName members { id } } }' )
# ── 1. net-kingdom-users group ───────────────────────────────────────────────
section "1. LLDAP group: net-kingdom-users"
if [ [ " $GROUPS_RESP " = = "NO_TOKEN" ] ] ; then
warn " Skipping — could not authenticate to LLDAP at $LLDAP_URL "
elif [ [ " $GROUPS_RESP " = = "CURL_FAILED" ] ] ; then
fail "Could not query LLDAP groups — is LLDAP up?"
else
EXISTS = $( echo " $GROUPS_RESP " | python3 -c \
"import sys,json; d=json.load(sys.stdin); print('yes' if any(g['displayName']=='net-kingdom-users' for g in d.get('data',{}).get('groups',[])) else 'no')" \
2>/dev/null || echo "no" )
if [ [ " $EXISTS " = = "yes" ] ] ; then
pass "Group 'net-kingdom-users' exists"
else
fail "Group 'net-kingdom-users' not found — run lldap/bootstrap-users.sh"
fi
fi
# ── 2. net-kingdom-admins group ──────────────────────────────────────────────
section "2. LLDAP group: net-kingdom-admins"
if [ [ " $GROUPS_RESP " = = "NO_TOKEN" || " $GROUPS_RESP " = = "CURL_FAILED" ] ] ; then
warn "Skipping — see section 1"
else
EXISTS = $( echo " $GROUPS_RESP " | python3 -c \
"import sys,json; d=json.load(sys.stdin); print('yes' if any(g['displayName']=='net-kingdom-admins' for g in d.get('data',{}).get('groups',[])) else 'no')" \
2>/dev/null || echo "no" )
if [ [ " $EXISTS " = = "yes" ] ] ; then
pass "Group 'net-kingdom-admins' exists"
else
fail "Group 'net-kingdom-admins' not found — run lldap/bootstrap-users.sh"
fi
fi
# ── 3. At least one non-admin user ───────────────────────────────────────────
section "3. At least one regular user in LLDAP"
USERS_RESP = $( gql 'query { users { id displayName email } }' )
if [ [ " $USERS_RESP " = = "NO_TOKEN" || " $USERS_RESP " = = "CURL_FAILED" ] ] ; then
warn "Skipping — could not query users"
else
USERS = $( echo " $USERS_RESP " | python3 -c \
"import sys,json; d=json.load(sys.stdin); us=[u['id'] for u in d.get('data',{}).get('users',[]) if u['id'] not in ('admin','break-glass')]; print(len(us))" \
2>/dev/null || echo "0" )
if [ [ " $USERS " -gt 0 ] ] ; then
pass " $USERS regular user(s) exist in LLDAP "
else
warn " No regular users found — add users via the LLDAP WebUI ( $LLDAP_URL ) or provisioning script "
fi
fi
# ── 4. Break-glass user ───────────────────────────────────────────────────────
section "4. Break-glass account"
if [ [ " $USERS_RESP " = = "NO_TOKEN" || " $USERS_RESP " = = "CURL_FAILED" ] ] ; then
warn "Skipping — could not query users"
else
BG_EXISTS = $( echo " $USERS_RESP " | python3 -c \
"import sys,json; d=json.load(sys.stdin); print('yes' if any(u['id']=='break-glass' for u in d.get('data',{}).get('users',[])) else 'no')" \
2>/dev/null || echo "no" )
if [ [ " $BG_EXISTS " = = "yes" ] ] ; then
pass "User 'break-glass' exists in LLDAP"
# Check group membership
BG_IN_ADMINS = $( echo " $GROUPS_RESP " | python3 -c \
"import sys,json; d=json.load(sys.stdin); gs=[g for g in d.get('data',{}).get('groups',[]) if g['displayName']=='net-kingdom-admins']; print('yes' if gs and any(m['id']=='break-glass' for m in gs[0].get('members',[])) else 'no')" \
2>/dev/null || echo "no" )
if [ [ " $BG_IN_ADMINS " = = "yes" ] ] ; then
pass "'break-glass' is in net-kingdom-admins group"
else
warn "'break-glass' is not in net-kingdom-admins — run lldap/break-glass.sh"
fi
else
fail "User 'break-glass' not found — run lldap/break-glass.sh"
fi
fi
# ── 5. Self-service portal ────────────────────────────────────────────────────
section "5. privacyIDEA self-service portal (pink-account.coulomb.social)"
PORTAL_STATUS = $( curl -sf -o /dev/null -w "%{http_code}" \
"https://pink-account.coulomb.social" 2>/dev/null || echo "000" )
if [ [ " $PORTAL_STATUS " = = "200" || " $PORTAL_STATUS " = = "302" ] ] ; then
pass " Self-service portal reachable (HTTP $PORTAL_STATUS ) "
elif [ [ " $PORTAL_STATUS " = = "000" ] ] ; then
warn "Self-service portal not reachable — check DNS and ingress"
else
warn " Self-service portal returned HTTP $PORTAL_STATUS "
fi
# ── 6. KeyCape OIDC client registrations ─────────────────────────────────────
section "6. KeyCape OIDC client registrations"
KC_POD = $( kubectl get pod -n " $SSO_NAMESPACE " \
-l app.kubernetes.io/name= keycape \
--field-selector= status.phase= Running \
-o jsonpath = '{.items[0].metadata.name}' 2>/dev/null || echo "" )
if [ [ -n " $KC_POD " ] ] ; then
DISCOVERY = $( kubectl exec -n " $SSO_NAMESPACE " " $KC_POD " -- \
wget -qO- "http://localhost:8080/.well-known/openid-configuration" 2>/dev/null || echo "" )
if [ [ -n " $DISCOVERY " ] ] ; then
pass "KeyCape OIDC discovery endpoint accessible"
# Check config for registered clients (KeyCape config in keycape-config Secret)
CONFIG = $( kubectl get secret keycape-config -n " $SSO_NAMESPACE " \
-o jsonpath = '{.data.config\.yaml}' 2>/dev/null | base64 -d 2>/dev/null || echo "" )
CLIENT_COUNT = $( echo " $CONFIG " | python3 -c \
"import sys; import yaml; cfg=yaml.safe_load(sys.stdin.read()); print(len(cfg.get('clients',[])))" \
2>/dev/null || echo "0" )
if [ [ " $CLIENT_COUNT " -gt 0 ] ] ; then
pass " $CLIENT_COUNT OIDC client(s) registered in KeyCape "
else
warn "No OIDC clients registered — add clients to keycape/create-secrets.sh and re-run it"
fi
2026-05-26 02:13:55 +02:00
OPENBAO_CLIENT_CHECK = $( echo " $CONFIG " | python3 -c '
import sys
import yaml
cfg = yaml.safe_load( sys.stdin.read( ) ) or { }
clients = cfg.get( "clients" ) or [ ]
target = next( ( client for client in clients if client.get( "clientId" ) = = "openbao-admin" ) , None)
if not target:
print( "missing openbao-admin client" )
raise SystemExit( 2)
required_redirects = {
"http://localhost:8250/oidc/callback" ,
"http://127.0.0.1:8250/oidc/callback" ,
2026-08-23 13:58:18 +02:00
"http://127.0.0.1:18200/ui/vault/auth/netkingdom/oidc/callback" ,
2026-06-18 01:23:02 +02:00
"https://bao.coulomb.social/ui/vault/auth/netkingdom/oidc/callback" ,
"https://bao.coulomb.social/ui/vault/auth/keycape/oidc/callback" ,
2026-05-26 02:13:55 +02:00
}
required_scopes = { "openid" , "profile" , "email" , "groups" }
missing_redirects = sorted( required_redirects - set( target.get( "redirectUris" ) or [ ] ) )
missing_scopes = sorted( required_scopes - set( target.get( "allowedScopes" ) or [ ] ) )
if target.get( "clientType" ) != "public" :
print( "openbao-admin clientType must be public for the current PKCE-only KeyCape profile" )
raise SystemExit( 3)
if missing_redirects:
print( "openbao-admin missing redirect URI(s): " + ", " .join( missing_redirects) )
raise SystemExit( 4)
if missing_scopes:
print( "openbao-admin missing scope(s): " + ", " .join( missing_scopes) )
raise SystemExit( 5)
2026-06-18 01:23:02 +02:00
print( "openbao-admin client has CLI/browser redirects and required scopes" )
2026-05-26 02:13:55 +02:00
' 2>/dev/null || echo "missing or invalid openbao-admin client" )
if [ [ " $OPENBAO_CLIENT_CHECK " = = openbao-admin* ] ] ; then
pass " $OPENBAO_CLIENT_CHECK "
else
fail " $OPENBAO_CLIENT_CHECK "
fi
feat(sso-mfa): T07/T08 user mgmt, backups, DR & break-glass (NK-WP-0001-T07/T08)
T07 — User management & self-service:
- k8s/lldap/bootstrap-users.sh: creates net-kingdom-users and net-kingdom-admins
groups in LLDAP via GraphQL API; idempotent.
- k8s/lldap/break-glass.sh: creates break-glass bypass account in LLDAP,
sets BREAKGLASS_PASSWORD, assigns to net-kingdom-admins.
- k8s/verify-t07.sh: 6 checks — groups, break-glass, self-service portal,
KeyCape OIDC client registrations.
T08 — Backups, DR, break-glass:
- k8s/backup/cronjob-sqlite-backups.yaml: daily CronJobs for LLDAP SQLite,
Authelia SQLite (with scale-down/up RBAC), and privacyIDEA enckey backup.
7-day retention, 03:00/03:15/03:30 UTC staggered schedule.
- k8s/backup/DR-RUNBOOK.md: full restore runbook — scenarios, restore order,
LLDAP/Authelia/PI SQLite restore procedure, full node rebuild sequence,
offsite age-encrypted export.
- k8s/verify-t08.sh: 9 checks — CronJobs, RBAC, run history, backup files
on PVCs, DR runbook presence, offsite backup (manual confirmation).
- WORKPLAN.md: T07/T08 sections with done-criteria added.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-19 09:17:03 +00:00
else
warn "Skipping client check — KeyCape not reachable in-cluster"
fi
else
warn "Skipping OIDC client check — no running KeyCape pod"
fi
# ── Summary ───────────────────────────────────────────────────────────────────
echo ""
echo "════════════════════════════════════════════════════════════"
echo " T07 verification: PASS= $PASS WARN= $WARN FAIL= $FAIL "
echo "════════════════════════════════════════════════════════════"
if [ [ " $FAIL " -gt 0 ] ] ; then
echo " Result: INCOMPLETE — resolve FAIL items before marking T07 done"
exit 1
elif [ [ " $WARN " -gt 0 ] ] ; then
echo " Result: PARTIAL — required structure is in place; review WARN items"
exit 0
else
echo " Result: COMPLETE — T07 done-criteria met; proceed to T08 (Backups, DR, break-glass)"
exit 0
fi