railiance-cluster/docs/keycape-tls-le-staging-rehearsal.md

kc.coulomb.social TLS reissue — Let's Encrypt STAGING rehearsal

Work record: RCLUSTER-WP-0018-T06, for NK-WP-0049-T33 (request message 6d5e748b). Founder decision 9f9d0d88 (A, 2026-10-21): recovery TLS for kc.coulomb.social is a reissue through letsencrypt-prod ACME HTTP-01 via Traefik, chained to ISRG Root X1. No self-signed substitute. Rehearse on staging first so a failed attempt cannot use up the production limit of 5 duplicate certificates per week.

This page is source preparation. It authorizes no cluster change. An operator runs the commands below, attended, from a shell with cluster-admin kubeconfig on railiance01 or through the usual tunnel.

What the rehearsal proves, and what it does not

It proves: public DNS for kc.coulomb.social reaches Traefik on port 80, Traefik routes /.well-known/acme-challenge/* for that host to a cert-manager solver while the live KeyCape ingress keeps serving, cert-manager can register an ACME account and finish an order, and the result is a chain for exactly kc.coulomb.social.

It does not prove: anything about the production directory, the production account key letsencrypt-prod-account-key, the sso/keycape ingress-shim path (cert-manager.io/cluster-issuer annotation → Certificate sso/kc-tls), ISRG Root X1 chaining (staging chains to the (STAGING) roots), or original-key recovery. Those belong to the real attempt (RCLUSTER-WP-0018-T04 / NK49-T33).

Isolation

Manifest: install/cert-manager/rehearsal/kc-le-staging.yaml, checked by tests/test_kc_le_staging_rehearsal.py.

Object	Name	Why it is safe
Namespace	tls-rehearsal-kc	throwaway; cleanup is deleting it
Issuer (namespaced)	letsencrypt-staging-rehearsal	staging directory only; not a ClusterIssuer, so no collision with any existing letsencrypt-* ClusterIssuer and nothing cluster-scoped is left behind
Certificate	kc-le-staging-rehearsal	secretName: kc-le-staging-rehearsal-tls in tls-rehearsal-kc; it cannot write sso/kc-tls
Generated Secrets	account key + staging leaf key	created by cert-manager in tls-rehearsal-kc only; destroyed with the namespace

No object references namespace sso. The manifest declares no email contact, because Let's Encrypt no longer sends expiry mail and staging needs none.

HTTP-01 coexistence with the live ingress (risk)

cert-manager creates a temporary solver Ingress in tls-rehearsal-kc for host kc.coulomb.social, path /.well-known/acme-challenge/<token>, and a solver pod/Service there. Traefik merges Ingress rules across namespaces and ranks routers by rule length. The solver's exact-path rule is longer than the live Host(kc.coulomb.social) && PathPrefix(/), so only the challenge path is diverted. The live router, its middlewares and TLS are not touched.

Residual risks, all short-lived:

• The challenge path for that host is served by the solver for 1–2 minutes. It is not a KeyCape path, so no real traffic depends on it.
• If a live sso/kc-tls renewal is pending at the same moment, both solvers coexist. Each has its own token path, which is the normal cert-manager case.
• A default-deny NetworkPolicy, or a policy that blocks Traefik → new namespaces, would leave the challenge pending. This shows as a timeout, not as an outage. None is declared in railiance-cluster source, but live state has not been checked.
• If Traefik ever adds a global HTTP→HTTPS redirect, Let's Encrypt follows it to 443 and still validates. The current HelmChartConfig in source has no redirect.
• Staging limits are loose (for example 30,000 certificates per week per domain), but do not loop on failures. Allow at most two attempts per session.

Operator commands

Run them from a railiance-cluster checkout at the reviewed commit.

cd ~/railiance-cluster
M=install/cert-manager/rehearsal/kc-le-staging.yaml
NS=tls-rehearsal-kc
WORK=$(mktemp -d) && chmod 700 "$WORK"

# 0. Preflight (read-only)
python3 -m unittest tests/test_kc_le_staging_rehearsal.py      # static guards pass
kubectl get crd certificates.cert-manager.io issuers.cert-manager.io
kubectl get namespace "$NS" 2>/dev/null && echo "ABORT: $NS already exists"
dig +short kc.coulomb.social A                                  # expect 92.205.62.239
curl -sS -o /dev/null -w '%{http_code}\n' http://kc.coulomb.social/.well-known/acme-challenge/probe  # any HTTP status; a timeout means abort
LIVE_BEFORE=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256); echo "$LIVE_BEFORE"
kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}'   # metadata only, no data
# A server dry run does not persist the Namespace, so the namespaced Issuer and
# Certificate would fail with NotFound. Create the empty namespace first.
kubectl create namespace "$NS"
kubectl apply --dry-run=server -f "$M"                          # expect the namespace configured + 2 created, all in $NS

# 1. Apply
kubectl apply -f "$M"

# 2. Wait (bounded) and observe
kubectl -n "$NS" wait certificate/kc-le-staging-rehearsal --for=condition=Ready --timeout=10m
kubectl -n "$NS" get certificate,certificaterequest,order,challenge,ingress -o wide

# 3. Verify the staging chain (public cert only: the tls.crt key, never tls.key)
kubectl -n "$NS" get secret kc-le-staging-rehearsal-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > "$WORK/chain.pem"
openssl x509 -in "$WORK/chain.pem" -noout -subject -issuer -ext subjectAltName -dates
#   expect: issuer CN begins "(STAGING)", SAN = DNS:kc.coulomb.social
curl -fsSL https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem -o "$WORK/stg-root-x1.pem"
openssl verify -CAfile "$WORK/stg-root-x1.pem" -untrusted "$WORK/chain.pem" "$WORK/chain.pem"   # expect: OK
python3 tools/public-tls-metadata.py --public-cert "$WORK/chain.pem" --public-trust "$WORK/stg-root-x1.pem"
#   expect: leaf_dns_sans == ["kc.coulomb.social"], server_chain_certificates >= 2

# 4. Confirm live TLS untouched
LIVE_AFTER=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256)
[ "$LIVE_BEFORE" = "$LIVE_AFTER" ] && echo "live cert unchanged" || echo "ABORT: live cert changed"
kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}'   # same as preflight

# 5. Cleanup (always, including after any abort)
kubectl delete namespace tls-rehearsal-kc --wait=true
kubectl get ingress -A | grep -i acme || echo "no solver ingress left"
rm -rf "$WORK"

Keep the JSON from step 3, the openssl subject/issuer/SAN/dates lines, and the step 4 comparison as the public rehearsal receipt for NK-WP-0049-T33. Do not keep tls.key or the account key.

Abort conditions

Stop and go to step 5 cleanup if any of these is true:

• The static guard test fails, or the server dry-run shows an object outside tls-rehearsal-kc or a name other than the three expected.
• tls-rehearsal-kc already exists before the preflight creates it.
• DNS for kc.coulomb.social does not resolve to the railiance01 ingress address, or port 80 does not answer.
• The live fingerprint or the sso/kc-tls resourceVersion changes at any point.
• The Certificate is not Ready within 10 minutes, or the Challenge/Order shows invalid. Capture kubectl -n tls-rehearsal-kc describe challenge first. At most one retry, which means delete the namespace and apply again.
• The issued issuer CN does not begin with (STAGING). That would mean the wrong directory was used: stop and report.

Moving to the production attempt

A green rehearsal (Ready, (STAGING) chain verifies, SAN correct, live unchanged, cleanup done) is the entry condition for the production reissue in NK-WP-0049-T33. That attempt runs through NetKingdom's sso/keycape ingress and ClusterIssuer/letsencrypt-prod under RCLUSTER-WP-0018-T04's owner-attended gate. It is not part of this runbook.

Rehearsal receipt (2026-10-23, attended by the founder)

Run from railiance-cluster d905a6f with the operator system:admin kubeconfig. Certificate Ready and Order valid on the first order (no retry).

Check	Result
Issuer	C = US, O = Let's Encrypt, CN = (STAGING) Ersatz Emmer YR2
Subject / SAN	CN = kc.coulomb.social / DNS:kc.coulomb.social
Validity	2026-10-23T05:56:44Z to 2026-12-26T05:56:43Z
openssl verify against staging Root X1	OK
tls_cert_sha256 (chain, public)	11ad7ad35885d16577f46842d886cf50fa1bd2bd37966612db8b4fa86532d40e
leaf_public_spki_sha256	sha256:f6504168524fdce4b3453449baac330f05739fec740a1f894bfa5bfc8035ff0d
trust_sha256 (staging root)	3a5e1171e5f5c2d41522d438f225602e4236a68fb29c32399a95921a4ae80e73
Chain certificates	3
Live fingerprint before = after	D5:DB:2B:55:5E:74:F7:FB:CD:56:DD:A3:28:5D:FF:FF:B8:6A:C1:64:BF:6C:28:88:1B:3E:FE:58:0F:C4:CE:5C
sso/kc-tls resourceVersion before = after	66444692
Cleanup	namespace deleted; no ACME solver ingress left

No private key or account key was retained. Deviation: the server dry run needed the namespace created first (runbook corrected above).