railiance-cluster/docs/keycape-tls-le-prod-nk49-attempt.md

kc.coulomb.social TLS reissue — production, NK49 attempt

Work record: RCLUSTER-WP-0018-T04, for NK-WP-0049-T18 ceremony step 3 (founder decision 9f9d0d88 = A). Entry condition: the green staging rehearsal of 2026-10-23 (docs/keycape-tls-le-staging-rehearsal.md). This is the same manifest shape against the production directory.

This page is source preparation and authorizes no cluster change. The founder runs the commands below, attended, with the admin kubeconfig, inside the signed attempt window (ends 2026-10-24T05:00:00Z).

Isolation

Manifest: install/cert-manager/attempt/kc-le-prod-nk49.yaml, checked by tests/test_kc_le_prod_nk49.py.

Object	Name	Why it is safe
Namespace	tls-attempt-kc	throwaway; cleanup is deleting it
Issuer (namespaced)	letsencrypt-nk49-attempt	production directory, own fresh account key; the live ClusterIssuer/letsencrypt-prod and its account key are not used
Certificate	kc-nk49-attempt	secretName: kc-nk49-attempt-tls in tls-attempt-kc; it cannot write sso/kc-tls

HTTP-01 coexistence with the live ingress is the same as in the staging runbook: only /.well-known/acme-challenge/<token> for the host is diverted, for 1–2 minutes. One attempt only. Each production issuance for the identical name set uses one of 5 per week; on failure, clean up and stop, do not retry.

The issued key leaves the cluster only into the founder's private attempt directory (~/nk49-attempt-20261023, mode 0700), as launcher input. It is never committed, pasted or logged. The namespace (and the in-cluster copy) is deleted right after export.

Operator commands

cd ~/railiance-cluster
export KUBECONFIG=~/.kube/config-railiance01
M=install/cert-manager/attempt/kc-le-prod-nk49.yaml
NS=tls-attempt-kc
A=~/nk49-attempt-20261023

# 0. Preflight (read-only)
python3 -m unittest tests/test_kc_le_prod_nk49.py
kubectl get namespace "$NS" 2>/dev/null && echo "ABORT: $NS already exists"
dig +short kc.coulomb.social A                                  # expect 92.205.62.239
LIVE_BEFORE=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256); echo "$LIVE_BEFORE"
kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}'   # metadata only
kubectl create namespace "$NS"
kubectl apply --dry-run=server -f "$M"                          # namespace configured + 2 created, all in $NS

# 1. Apply and wait (bounded)
kubectl apply -f "$M"
kubectl -n "$NS" wait certificate/kc-nk49-attempt --for=condition=Ready --timeout=10m
kubectl -n "$NS" get certificate,certificaterequest,order,challenge -o wide

# 2. Export into the private attempt directory (key never printed)
( umask 077
  kubectl -n "$NS" get secret kc-nk49-attempt-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > "$A/kc-tls.crt"
  kubectl -n "$NS" get secret kc-nk49-attempt-tls -o jsonpath='{.data.tls\.key}' | base64 -d > "$A/kc-tls.key" )
ls -l "$A"

# 3. Verify the public chain against ISRG Root X1
sha256sum "$A/isrgrootx1.pem"   # expect 22b557a27055b33606b6559f37703928d3e4ad79f110b407d04986e1843543d1
openssl x509 -in "$A/kc-tls.crt" -noout -subject -issuer -ext subjectAltName -dates
openssl verify -CAfile "$A/isrgrootx1.pem" -untrusted "$A/kc-tls.crt" "$A/kc-tls.crt"   # expect: OK
python3 tools/public-tls-metadata.py --public-cert "$A/kc-tls.crt" --public-trust "$A/isrgrootx1.pem"
#   expect: leaf_dns_sans == ["kc.coulomb.social"]; issuer not "(STAGING)"
#   check the key matches the certificate (prints two equal hashes, no key material)
openssl x509 -in "$A/kc-tls.crt" -noout -pubkey | openssl sha256
openssl pkey -in "$A/kc-tls.key" -pubout | openssl sha256

# 4. Confirm live TLS untouched
LIVE_AFTER=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256)
[ "$LIVE_BEFORE" = "$LIVE_AFTER" ] && echo "live cert unchanged" || echo "ABORT: live cert changed"
kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}'   # same as preflight

# 5. Cleanup (always, including after any abort)
kubectl delete namespace tls-attempt-kc --wait=true
kubectl get ingress -A | grep -i acme || echo "no solver ingress left"

The public receipt is the step 3 JSON, the openssl subject/issuer/SAN/dates lines, the matching public-key hashes and the step 4 comparison. kc-tls.key stays in the attempt directory until the launcher has run, then is removed with the directory under T18 cleanup.

Abort conditions

Go to step 5 and stop if: the guard test fails; the dry run shows anything outside tls-attempt-kc; DNS or port 80 fails; the Certificate is not Ready in 10 minutes or the Order/Challenge is invalid (capture kubectl -n tls-attempt-kc describe challenge first); the issuer CN contains (STAGING); openssl verify is not OK; the public-key hashes differ; or the live fingerprint or sso/kc-tls resourceVersion changes.