RCLUSTER-WP-0018-T04: production kc.coulomb.social reissue for the NK49 attempt (isolated namespace, guarded)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 4384@bnt-lap001
Assistant-Session: 80d1f3da-7a91-4519-9737-9ba128b4da64

docs/keycape-tls-le-prod-nk49-attempt.md

@@ -0,0 +1,96 @@
+# kc.coulomb.social TLS reissue — production, NK49 attempt
+
+Work record: RCLUSTER-WP-0018-T04, for NK-WP-0049-T18 ceremony step 3 (founder
+decision 9f9d0d88 = A). Entry condition: the green staging rehearsal of
+2026-10-23 (`docs/keycape-tls-le-staging-rehearsal.md`). This is the same
+manifest shape against the production directory.
+
+This page is source preparation and authorizes no cluster change. The founder
+runs the commands below, attended, with the admin kubeconfig, inside the signed
+attempt window (ends 2026-10-24T05:00:00Z).
+
+## Isolation
+
+Manifest: `install/cert-manager/attempt/kc-le-prod-nk49.yaml`, checked by
+`tests/test_kc_le_prod_nk49.py`.
+
+| Object | Name | Why it is safe |
+| --- | --- | --- |
+| Namespace | `tls-attempt-kc` | throwaway; cleanup is deleting it |
+| Issuer (namespaced) | `letsencrypt-nk49-attempt` | production directory, own fresh account key; the live `ClusterIssuer/letsencrypt-prod` and its account key are not used |
+| Certificate | `kc-nk49-attempt` | `secretName: kc-nk49-attempt-tls` in `tls-attempt-kc`; it cannot write `sso/kc-tls` |
+
+HTTP-01 coexistence with the live ingress is the same as in the staging runbook:
+only `/.well-known/acme-challenge/<token>` for the host is diverted, for 1–2
+minutes. **One attempt only.** Each production issuance for the identical name
+set uses one of 5 per week; on failure, clean up and stop, do not retry.
+
+The issued key leaves the cluster only into the founder's private attempt
+directory (`~/nk49-attempt-20261023`, mode 0700), as launcher input. It is never
+committed, pasted or logged. The namespace (and the in-cluster copy) is deleted
+right after export.
+
+## Operator commands
+
+```sh
+cd ~/railiance-cluster
+export KUBECONFIG=~/.kube/config-railiance01
+M=install/cert-manager/attempt/kc-le-prod-nk49.yaml
+NS=tls-attempt-kc
+A=~/nk49-attempt-20261023
+
+# 0. Preflight (read-only)
+python3 -m unittest tests/test_kc_le_prod_nk49.py
+kubectl get namespace "$NS" 2>/dev/null && echo "ABORT: $NS already exists"
+dig +short kc.coulomb.social A                                  # expect 92.205.62.239
+LIVE_BEFORE=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social </dev/null 2>/dev/null \
+  | openssl x509 -noout -fingerprint -sha256); echo "$LIVE_BEFORE"
+kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}'   # metadata only
+kubectl create namespace "$NS"
+kubectl apply --dry-run=server -f "$M"                          # namespace configured + 2 created, all in $NS
+
+# 1. Apply and wait (bounded)
+kubectl apply -f "$M"
+kubectl -n "$NS" wait certificate/kc-nk49-attempt --for=condition=Ready --timeout=10m
+kubectl -n "$NS" get certificate,certificaterequest,order,challenge -o wide
+
+# 2. Export into the private attempt directory (key never printed)
+( umask 077
+  kubectl -n "$NS" get secret kc-nk49-attempt-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > "$A/kc-tls.crt"
+  kubectl -n "$NS" get secret kc-nk49-attempt-tls -o jsonpath='{.data.tls\.key}' | base64 -d > "$A/kc-tls.key" )
+ls -l "$A"
+
+# 3. Verify the public chain against ISRG Root X1
+sha256sum "$A/isrgrootx1.pem"   # expect 22b557a27055b33606b6559f37703928d3e4ad79f110b407d04986e1843543d1
+openssl x509 -in "$A/kc-tls.crt" -noout -subject -issuer -ext subjectAltName -dates
+openssl verify -CAfile "$A/isrgrootx1.pem" -untrusted "$A/kc-tls.crt" "$A/kc-tls.crt"   # expect: OK
+python3 tools/public-tls-metadata.py --public-cert "$A/kc-tls.crt" --public-trust "$A/isrgrootx1.pem"
+#   expect: leaf_dns_sans == ["kc.coulomb.social"]; issuer not "(STAGING)"
+#   check the key matches the certificate (prints two equal hashes, no key material)
+openssl x509 -in "$A/kc-tls.crt" -noout -pubkey | openssl sha256
+openssl pkey -in "$A/kc-tls.key" -pubout | openssl sha256
+
+# 4. Confirm live TLS untouched
+LIVE_AFTER=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social </dev/null 2>/dev/null \
+  | openssl x509 -noout -fingerprint -sha256)
+[ "$LIVE_BEFORE" = "$LIVE_AFTER" ] && echo "live cert unchanged" || echo "ABORT: live cert changed"
+kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}'   # same as preflight
+
+# 5. Cleanup (always, including after any abort)
+kubectl delete namespace tls-attempt-kc --wait=true
+kubectl get ingress -A | grep -i acme || echo "no solver ingress left"
+```
+
+The public receipt is the step 3 JSON, the `openssl` subject/issuer/SAN/dates
+lines, the matching public-key hashes and the step 4 comparison. `kc-tls.key`
+stays in the attempt directory until the launcher has run, then is removed with
+the directory under T18 cleanup.
+
+## Abort conditions
+
+Go to step 5 and stop if: the guard test fails; the dry run shows anything
+outside `tls-attempt-kc`; DNS or port 80 fails; the Certificate is not Ready in
+10 minutes or the Order/Challenge is `invalid` (capture `kubectl -n
+tls-attempt-kc describe challenge` first); the issuer CN contains `(STAGING)`;
+`openssl verify` is not `OK`; the public-key hashes differ; or the live
+fingerprint or `sso/kc-tls` resourceVersion changes.

install/cert-manager/attempt/kc-le-prod-nk49.yaml

@@ -0,0 +1,58 @@
+# Let's Encrypt PRODUCTION reissue of kc.coulomb.social for the NK49 attempt.
+#
+# Purpose: RCLUSTER-WP-0018-T04 / NK-WP-0049-T18 ceremony step 3. Founder
+# decision 9f9d0d88 (A, 2026-10-21): recovery TLS is a reissue via ACME HTTP-01
+# through Traefik, chained to ISRG Root X1. The staging rehearsal (T06,
+# kc-le-staging.yaml) proved the same path green on 2026-10-23.
+#
+# Isolation by construction (asserted by tests/test_kc_le_prod_nk49.py):
+#   - every object lives in the throwaway namespace tls-attempt-kc;
+#   - a namespaced Issuer with its own fresh account key: the live
+#     ClusterIssuer letsencrypt-prod and letsencrypt-prod-account-key are not
+#     referenced or reused;
+#   - secretName kc-nk49-attempt-tls in tls-attempt-kc, so it can never
+#     overwrite sso/kc-tls; nothing here references namespace sso;
+#   - no Secret is declared.
+# One issuance uses one of the 5-per-week duplicate-certificate allowance.
+#
+# Runbook: docs/keycape-tls-le-prod-nk49-attempt.md
+# Cleanup: kubectl delete namespace tls-attempt-kc
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: tls-attempt-kc
+  labels:
+    app.kubernetes.io/part-of: railiance-cluster
+    railiance.io/purpose: nk49-attempt-tls-reissue
+    railiance.io/throwaway: "true"
+---
+apiVersion: cert-manager.io/v1
+kind: Issuer
+metadata:
+  name: letsencrypt-nk49-attempt
+  namespace: tls-attempt-kc
+spec:
+  acme:
+    server: https://acme-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: letsencrypt-nk49-attempt-account-key
+    solvers:
+      - http01:
+          ingress:
+            ingressClassName: traefik
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: kc-nk49-attempt
+  namespace: tls-attempt-kc
+spec:
+  secretName: kc-nk49-attempt-tls
+  dnsNames:
+    - kc.coulomb.social
+  issuerRef:
+    group: cert-manager.io
+    kind: Issuer
+    name: letsencrypt-nk49-attempt
+  privateKey:
+    rotationPolicy: Always

tests/test_kc_le_prod_nk49.py

@@ -0,0 +1,64 @@
+"""Static guards for the kc.coulomb.social production reissue for the NK49 attempt.
+
+The reissue must never touch the live sso/kc-tls Secret or reuse the live
+letsencrypt-prod ClusterIssuer/account key. These checks are offline.
+"""
+from pathlib import Path
+import unittest
+
+import yaml
+
+ROOT = Path(__file__).parents[1]
+MANIFEST = ROOT / "install/cert-manager/attempt/kc-le-prod-nk49.yaml"
+RUNBOOK = ROOT / "docs/keycape-tls-le-prod-nk49-attempt.md"
+NAMESPACE = "tls-attempt-kc"
+PRODUCTION = "https://acme-v02.api.letsencrypt.org/directory"
+
+
+def load():
+    return [d for d in yaml.safe_load_all(MANIFEST.read_text()) if d]
+
+
+def by_kind(docs, kind):
+    found = [d for d in docs if d["kind"] == kind]
+    assert len(found) == 1, kind
+    return found[0]
+
+
+class ProductionAttemptManifest(unittest.TestCase):
+    def setUp(self):
+        self.docs = load()
+
+    def test_only_expected_kinds_no_cluster_scope_no_secrets(self):
+        self.assertEqual(sorted(d["kind"] for d in self.docs), ["Certificate", "Issuer", "Namespace"])
+
+    def test_everything_is_in_the_throwaway_namespace(self):
+        self.assertEqual(by_kind(self.docs, "Namespace")["metadata"]["name"], NAMESPACE)
+        for doc in self.docs:
+            if doc["kind"] != "Namespace":
+                self.assertEqual(doc["metadata"]["namespace"], NAMESPACE)
+        self.assertNotIn("namespace: sso", MANIFEST.read_text())
+
+    def test_issuer_is_production_with_its_own_account_key(self):
+        acme = by_kind(self.docs, "Issuer")["spec"]["acme"]
+        self.assertEqual(acme["server"], PRODUCTION)
+        self.assertNotIn("staging", yaml.safe_dump(self.docs))
+        self.assertNotEqual(acme["privateKeySecretRef"]["name"], "letsencrypt-prod-account-key")
+        self.assertEqual(acme["solvers"], [{"http01": {"ingress": {"ingressClassName": "traefik"}}}])
+
+    def test_certificate_cannot_overwrite_live_secret(self):
+        spec = by_kind(self.docs, "Certificate")["spec"]
+        self.assertNotEqual(spec["secretName"], "kc-tls")
+        self.assertEqual(spec["dnsNames"], ["kc.coulomb.social"])
+        issuer = by_kind(self.docs, "Issuer")["metadata"]["name"]
+        self.assertEqual(spec["issuerRef"], {"group": "cert-manager.io", "kind": "Issuer", "name": issuer})
+
+    def test_runbook_names_manifest_cleanup_and_root(self):
+        text = RUNBOOK.read_text()
+        self.assertIn("install/cert-manager/attempt/kc-le-prod-nk49.yaml", text)
+        self.assertIn(f"kubectl delete namespace {NAMESPACE}", text)
+        self.assertIn("22b557a27055b33606b6559f37703928d3e4ad79f110b407d04986e1843543d1", text)
+
+
+if __name__ == "__main__":
+    unittest.main()