RCLUSTER-WP-0018-T06: Let's Encrypt staging rehearsal for kc.coulomb.social reissue

Throwaway namespace, namespaced staging Issuer and a Certificate with a distinct target name (cannot touch the live kc-tls object), operator runbook with verify/cleanup/abort, and static guard tests. Source only; for NK-WP-0049-T33 (decision 9f9d0d88 = A). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 696317@bnt-lap001 Assistant-Session: bab9e511-0915-45b3-926a-2fab552edff6
2026-10-23 06:57:56 +02:00 · 2026-10-23 06:57:56 +02:00 · a7c0e5b666
commit a7c0e5b666
parent 0e8df471c7
5 changed files with 298 additions and 1 deletions
--- a/WORK-RECORDS.md
+++ b/WORK-RECORDS.md
@ -72,6 +72,7 @@
 | task | RCLUSTER-WP-0018-T03 | wait | — | workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md |
 | task | RCLUSTER-WP-0018-T04 | wait | — | workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md |
 | task | RCLUSTER-WP-0018-T05 | todo | — | workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md |
+| task | RCLUSTER-WP-0018-T06 | done | — | workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md |
 | task | RCLUSTER-WP-0019-T01 | done | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md |
 | task | RCLUSTER-WP-0019-T02 | done | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md |
 | task | RCLUSTER-WP-0019-T03 | done | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md |
--- a/docs/keycape-tls-le-staging-rehearsal.md
+++ b/docs/keycape-tls-le-staging-rehearsal.md
@ -0,0 +1,142 @@
+# kc.coulomb.social TLS reissue — Let's Encrypt STAGING rehearsal
+
+Work record: RCLUSTER-WP-0018-T06, for NK-WP-0049-T33 (request message
+6d5e748b). Founder decision 9f9d0d88 (A, 2026-10-21): recovery TLS for
+`kc.coulomb.social` is a **reissue** through `letsencrypt-prod` ACME HTTP-01 via
+Traefik, chained to ISRG Root X1. No self-signed substitute. Rehearse on staging
+first so a failed attempt cannot use up the production limit of 5 duplicate
+certificates per week.
+
+This page is source preparation. It authorizes no cluster change. An operator
+runs the commands below, attended, from a shell with cluster-admin kubeconfig
+on railiance01 or through the usual tunnel.
+
+## What the rehearsal proves, and what it does not
+
+It proves: public DNS for `kc.coulomb.social` reaches Traefik on port 80, Traefik
+routes `/.well-known/acme-challenge/*` for that host to a cert-manager solver
+while the live KeyCape ingress keeps serving, cert-manager can register an ACME
+account and finish an order, and the result is a chain for exactly
+`kc.coulomb.social`.
+
+It does not prove: anything about the production directory, the production
+account key `letsencrypt-prod-account-key`, the `sso/keycape` ingress-shim path
+(`cert-manager.io/cluster-issuer` annotation → Certificate `sso/kc-tls`), ISRG
+Root X1 chaining (staging chains to the `(STAGING)` roots), or original-key
+recovery. Those belong to the real attempt (RCLUSTER-WP-0018-T04 / NK49-T33).
+
+## Isolation
+
+Manifest: `install/cert-manager/rehearsal/kc-le-staging.yaml`, checked by
+`tests/test_kc_le_staging_rehearsal.py`.
+
+| Object | Name | Why it is safe |
+| --- | --- | --- |
+| Namespace | `tls-rehearsal-kc` | throwaway; cleanup is deleting it |
+| Issuer (namespaced) | `letsencrypt-staging-rehearsal` | staging directory only; not a ClusterIssuer, so no collision with any existing `letsencrypt-*` ClusterIssuer and nothing cluster-scoped is left behind |
+| Certificate | `kc-le-staging-rehearsal` | `secretName: kc-le-staging-rehearsal-tls` in `tls-rehearsal-kc`; it cannot write `sso/kc-tls` |
+| Generated Secrets | account key + staging leaf key | created by cert-manager in `tls-rehearsal-kc` only; destroyed with the namespace |
+
+No object references namespace `sso`. The manifest declares no email contact,
+because Let's Encrypt no longer sends expiry mail and staging needs none.
+
+### HTTP-01 coexistence with the live ingress (risk)
+
+cert-manager creates a temporary solver Ingress in `tls-rehearsal-kc` for host
+`kc.coulomb.social`, path `/.well-known/acme-challenge/<token>`, and a solver
+pod/Service there. Traefik merges Ingress rules across namespaces and ranks
+routers by rule length. The solver's exact-path rule is longer than the live
+`Host(kc.coulomb.social) && PathPrefix(/)`, so only the challenge path is
+diverted. The live router, its middlewares and TLS are not touched.
+
+Residual risks, all short-lived:
+
+- The challenge path for that host is served by the solver for 1–2 minutes. It
+  is not a KeyCape path, so no real traffic depends on it.
+- If a live `sso/kc-tls` renewal is pending at the same moment, both solvers
+  coexist. Each has its own token path, which is the normal cert-manager case.
+- A default-deny NetworkPolicy, or a policy that blocks Traefik → new
+  namespaces, would leave the challenge pending. This shows as a timeout, not as
+  an outage. None is declared in railiance-cluster source, but live state has not
+  been checked.
+- If Traefik ever adds a global HTTP→HTTPS redirect, Let's Encrypt follows it to
+  443 and still validates. The current HelmChartConfig in source has no redirect.
+- Staging limits are loose (for example 30,000 certificates per week per domain),
+  but do not loop on failures. Allow at most two attempts per session.
+
+## Operator commands
+
+Run them from a railiance-cluster checkout at the reviewed commit.
+
+```sh
+cd ~/railiance-cluster
+M=install/cert-manager/rehearsal/kc-le-staging.yaml
+NS=tls-rehearsal-kc
+WORK=$(mktemp -d) && chmod 700 "$WORK"
+
+# 0. Preflight (read-only)
+python3 -m unittest tests/test_kc_le_staging_rehearsal.py      # static guards pass
+kubectl get crd certificates.cert-manager.io issuers.cert-manager.io
+kubectl get namespace "$NS" 2>/dev/null && echo "ABORT: $NS already exists"
+dig +short kc.coulomb.social A                                  # expect 92.205.62.239
+curl -sS -o /dev/null -w '%{http_code}\n' http://kc.coulomb.social/.well-known/acme-challenge/probe  # any HTTP status; a timeout means abort
+LIVE_BEFORE=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social </dev/null 2>/dev/null \
+  | openssl x509 -noout -fingerprint -sha256); echo "$LIVE_BEFORE"
+kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}'   # metadata only, no data
+kubectl apply --dry-run=server -f "$M"                          # expect exactly 3 objects, all in $NS
+
+# 1. Apply
+kubectl apply -f "$M"
+
+# 2. Wait (bounded) and observe
+kubectl -n "$NS" wait certificate/kc-le-staging-rehearsal --for=condition=Ready --timeout=10m
+kubectl -n "$NS" get certificate,certificaterequest,order,challenge,ingress -o wide
+
+# 3. Verify the staging chain (public cert only: the tls.crt key, never tls.key)
+kubectl -n "$NS" get secret kc-le-staging-rehearsal-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > "$WORK/chain.pem"
+openssl x509 -in "$WORK/chain.pem" -noout -subject -issuer -ext subjectAltName -dates
+#   expect: issuer CN begins "(STAGING)", SAN = DNS:kc.coulomb.social
+curl -fsSL https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem -o "$WORK/stg-root-x1.pem"
+openssl verify -CAfile "$WORK/stg-root-x1.pem" -untrusted "$WORK/chain.pem" "$WORK/chain.pem"   # expect: OK
+python3 tools/public-tls-metadata.py --public-cert "$WORK/chain.pem" --public-trust "$WORK/stg-root-x1.pem"
+#   expect: leaf_dns_sans == ["kc.coulomb.social"], server_chain_certificates >= 2
+
+# 4. Confirm live TLS untouched
+LIVE_AFTER=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social </dev/null 2>/dev/null \
+  | openssl x509 -noout -fingerprint -sha256)
+[ "$LIVE_BEFORE" = "$LIVE_AFTER" ] && echo "live cert unchanged" || echo "ABORT: live cert changed"
+kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}'   # same as preflight
+
+# 5. Cleanup (always, including after any abort)
+kubectl delete namespace tls-rehearsal-kc --wait=true
+kubectl get ingress -A | grep -i acme || echo "no solver ingress left"
+rm -rf "$WORK"
+```
+
+Keep the JSON from step 3, the `openssl` subject/issuer/SAN/dates lines, and the
+step 4 comparison as the public rehearsal receipt for NK-WP-0049-T33. Do not keep
+`tls.key` or the account key.
+
+## Abort conditions
+
+Stop and go to step 5 cleanup if any of these is true:
+
+- The static guard test fails, or the server dry-run shows an object outside
+  `tls-rehearsal-kc` or a name other than the three expected.
+- `tls-rehearsal-kc` already exists before apply.
+- DNS for `kc.coulomb.social` does not resolve to the railiance01 ingress
+  address, or port 80 does not answer.
+- The live fingerprint or the `sso/kc-tls` resourceVersion changes at any point.
+- The Certificate is not Ready within 10 minutes, or the Challenge/Order shows
+  `invalid`. Capture `kubectl -n tls-rehearsal-kc describe challenge` first. At
+  most one retry, which means delete the namespace and apply again.
+- The issued issuer CN does not begin with `(STAGING)`. That would mean the wrong
+  directory was used: stop and report.
+
+## Moving to the production attempt
+
+A green rehearsal (Ready, `(STAGING)` chain verifies, SAN correct, live unchanged,
+cleanup done) is the entry condition for the production reissue in NK-WP-0049-T33.
+That attempt runs through NetKingdom's `sso/keycape` ingress and
+`ClusterIssuer/letsencrypt-prod` under RCLUSTER-WP-0018-T04's owner-attended gate.
+It is not part of this runbook.
--- a/install/cert-manager/rehearsal/kc-le-staging.yaml
+++ b/install/cert-manager/rehearsal/kc-le-staging.yaml
@ -0,0 +1,59 @@
+# Let's Encrypt STAGING rehearsal for the kc.coulomb.social reissue path.
+#
+# Purpose: NK-WP-0049-T33 / RCLUSTER-WP-0018-T06. Founder decision 9f9d0d88 (A,
+# 2026-10-21): recovery TLS for kc.coulomb.social is a reissue via
+# letsencrypt-prod ACME HTTP-01 through Traefik, chained to ISRG Root X1. This
+# file rehearses the same HTTP-01/Traefik path against the STAGING directory
+# first, so a failed attempt never consumes the production duplicate-certificate
+# limit (5 per week for the identical name set).
+#
+# Isolation by construction (asserted by tests/test_kc_le_staging_rehearsal.py):
+#   - every object lives in the throwaway namespace tls-rehearsal-kc;
+#   - a namespaced Issuer, not a ClusterIssuer: no cluster-scoped object, no
+#     name collision with any existing letsencrypt-* ClusterIssuer;
+#   - secretName kc-le-staging-rehearsal-tls in tls-rehearsal-kc, so it can
+#     never overwrite sso/kc-tls; nothing here references namespace sso;
+#   - no Secret is declared; cert-manager generates the ACME account key and the
+#     throwaway staging key inside tls-rehearsal-kc only.
+#
+# Runbook: docs/keycape-tls-le-staging-rehearsal.md
+# Cleanup: kubectl delete namespace tls-rehearsal-kc
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: tls-rehearsal-kc
+  labels:
+    app.kubernetes.io/part-of: railiance-cluster
+    railiance.io/purpose: le-staging-rehearsal
+    railiance.io/throwaway: "true"
+---
+apiVersion: cert-manager.io/v1
+kind: Issuer
+metadata:
+  name: letsencrypt-staging-rehearsal
+  namespace: tls-rehearsal-kc
+spec:
+  acme:
+    server: https://acme-staging-v02.api.letsencrypt.org/directory
+    privateKeySecretRef:
+      name: letsencrypt-staging-rehearsal-account-key
+    solvers:
+      - http01:
+          ingress:
+            ingressClassName: traefik
+---
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: kc-le-staging-rehearsal
+  namespace: tls-rehearsal-kc
+spec:
+  secretName: kc-le-staging-rehearsal-tls
+  dnsNames:
+    - kc.coulomb.social
+  issuerRef:
+    group: cert-manager.io
+    kind: Issuer
+    name: letsencrypt-staging-rehearsal
+  privateKey:
+    rotationPolicy: Always
--- a/tests/test_kc_le_staging_rehearsal.py
+++ b/tests/test_kc_le_staging_rehearsal.py
@ -0,0 +1,68 @@
+"""Static guards for the kc.coulomb.social Let's Encrypt STAGING rehearsal.
+
+The rehearsal must never be able to touch the live sso/kc-tls Secret or use the
+production ACME directory. These checks are offline; no cluster access.
+"""
+from pathlib import Path
+import unittest
+
+import yaml
+
+ROOT = Path(__file__).parents[1]
+MANIFEST = ROOT / "install/cert-manager/rehearsal/kc-le-staging.yaml"
+RUNBOOK = ROOT / "docs/keycape-tls-le-staging-rehearsal.md"
+NAMESPACE = "tls-rehearsal-kc"
+STAGING = "https://acme-staging-v02.api.letsencrypt.org/directory"
+
+
+def load():
+    return [d for d in yaml.safe_load_all(MANIFEST.read_text()) if d]
+
+
+def by_kind(docs, kind):
+    found = [d for d in docs if d["kind"] == kind]
+    assert len(found) == 1, kind
+    return found[0]
+
+
+class StagingRehearsalManifest(unittest.TestCase):
+    def setUp(self):
+        self.docs = load()
+
+    def test_only_expected_kinds_no_cluster_scope_no_secrets(self):
+        kinds = sorted(d["kind"] for d in self.docs)
+        self.assertEqual(kinds, ["Certificate", "Issuer", "Namespace"])
+        self.assertNotIn("ClusterIssuer", kinds)
+        self.assertNotIn("Secret", kinds)
+
+    def test_everything_is_in_the_throwaway_namespace(self):
+        self.assertEqual(by_kind(self.docs, "Namespace")["metadata"]["name"], NAMESPACE)
+        for doc in self.docs:
+            if doc["kind"] != "Namespace":
+                self.assertEqual(doc["metadata"]["namespace"], NAMESPACE)
+        self.assertNotIn("namespace: sso", MANIFEST.read_text())
+
+    def test_issuer_is_staging_only(self):
+        acme = by_kind(self.docs, "Issuer")["spec"]["acme"]
+        self.assertEqual(acme["server"], STAGING)
+        self.assertNotIn("acme-v02.api.letsencrypt.org", MANIFEST.read_text())
+        self.assertNotIn("prod", acme["privateKeySecretRef"]["name"])
+        self.assertEqual(acme["solvers"], [{"http01": {"ingress": {"ingressClassName": "traefik"}}}])
+
+    def test_certificate_cannot_overwrite_live_secret(self):
+        spec = by_kind(self.docs, "Certificate")["spec"]
+        self.assertNotEqual(spec["secretName"], "kc-tls")
+        self.assertIn("staging", spec["secretName"])
+        self.assertEqual(spec["dnsNames"], ["kc.coulomb.social"])
+        issuer = by_kind(self.docs, "Issuer")["metadata"]["name"]
+        self.assertEqual(spec["issuerRef"], {"group": "cert-manager.io", "kind": "Issuer", "name": issuer})
+
+    def test_runbook_names_manifest_and_cleanup(self):
+        text = RUNBOOK.read_text()
+        self.assertIn("install/cert-manager/rehearsal/kc-le-staging.yaml", text)
+        self.assertIn(f"kubectl delete namespace {NAMESPACE}", text)
+        self.assertIn("(STAGING)", text)
+
+
+if __name__ == "__main__":
+    unittest.main()
--- a/workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md
+++ b/workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md
@ -8,7 +8,7 @@ status: active
 owner: codex
 topic_slug: railiance
 created: "2026-10-17"
-updated: "2026-10-17"
+updated: "2026-10-23"
 flavor: implementation
 depends_on: []
 related: [CUST-WP-0080, NK-WP-0049, RCLUSTER-WP-0007, RCLUSTER-WP-0017]
@ -104,6 +104,12 @@ reissue are distinct claims; no self-signed trust substitute, new broker or
 implicit production action. Done when the exact source proposal, public metadata
 preflight, negatives, timeout, abort and custody obligations are reviewed.

+Checkpoint 2026-10-23: founder decision 9f9d0d88 (A, 2026-10-21) selected the
+reissue alternative for recovery TLS: letsencrypt-prod ACME HTTP-01 through
+Traefik, chained to ISRG Root X1, no self-signed substitute, staging rehearsal
+first. The staging rehearsal source is prepared under T06. This task's status is
+left for owner review against T02.
+
 ## Verify admitted original TLS recovery through the actual owner boundary

 ```task
@ -143,6 +149,27 @@ or rewrite a historical archive without those owner reviews/authority. Done when
 the future contract requirement has an exact tracked disposition/handoff; it is
 not proof of current archive coverage or actual future capture.

+## Prepare the Let's Encrypt staging rehearsal for the kc.coulomb.social reissue
+
+```task
+id: RCLUSTER-WP-0018-T06
+status: done
+priority: high
+assignee: claude
+```
+
+Source preparation for NK-WP-0049-T33 (request 6d5e748b, decision 9f9d0d88 = A).
+Delivered install/cert-manager/rehearsal/kc-le-staging.yaml (throwaway namespace
+tls-rehearsal-kc, a namespaced staging Issuer, and a Certificate whose target
+name differs from the live one, so it cannot write sso/kc-tls), the runbook
+docs/keycape-tls-le-staging-rehearsal.md (preflight, apply, bounded wait,
+"(STAGING)" chain verification with openssl and tools/public-tls-metadata.py, a
+check that the live certificate is unchanged, cleanup, abort conditions), and
+static guards in tests/test_kc_le_staging_rehearsal.py. No kubectl or cluster
+write happened, and no credential was read. Running the rehearsal is an
+operator-attended step. Its public receipt is the entry condition for the
+production reissue under T04.
+
 ## Coordination and completion

 Beneficiary NK-WP-0049-T35/CUST-WP-0080 remains linked to this obligor record by