diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 37e4612..5335e22 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -72,6 +72,7 @@ | task | RCLUSTER-WP-0018-T03 | wait | — | workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md | | task | RCLUSTER-WP-0018-T04 | wait | — | workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md | | task | RCLUSTER-WP-0018-T05 | todo | — | workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md | +| task | RCLUSTER-WP-0018-T06 | done | — | workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md | | task | RCLUSTER-WP-0019-T01 | done | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md | | task | RCLUSTER-WP-0019-T02 | done | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md | | task | RCLUSTER-WP-0019-T03 | done | — | workplans/RCLUSTER-WP-0019-k3s-admin-kubeconfig-mode.md | diff --git a/docs/keycape-tls-le-staging-rehearsal.md b/docs/keycape-tls-le-staging-rehearsal.md new file mode 100644 index 0000000..b1ebfcd --- /dev/null +++ b/docs/keycape-tls-le-staging-rehearsal.md @@ -0,0 +1,142 @@ +# kc.coulomb.social TLS reissue — Let's Encrypt STAGING rehearsal + +Work record: RCLUSTER-WP-0018-T06, for NK-WP-0049-T33 (request message +6d5e748b). Founder decision 9f9d0d88 (A, 2026-10-21): recovery TLS for +`kc.coulomb.social` is a **reissue** through `letsencrypt-prod` ACME HTTP-01 via +Traefik, chained to ISRG Root X1. No self-signed substitute. Rehearse on staging +first so a failed attempt cannot use up the production limit of 5 duplicate +certificates per week. + +This page is source preparation. It authorizes no cluster change. An operator +runs the commands below, attended, from a shell with cluster-admin kubeconfig +on railiance01 or through the usual tunnel. + +## What the rehearsal proves, and what it does not + +It proves: public DNS for `kc.coulomb.social` reaches Traefik on port 80, Traefik +routes `/.well-known/acme-challenge/*` for that host to a cert-manager solver +while the live KeyCape ingress keeps serving, cert-manager can register an ACME +account and finish an order, and the result is a chain for exactly +`kc.coulomb.social`. + +It does not prove: anything about the production directory, the production +account key `letsencrypt-prod-account-key`, the `sso/keycape` ingress-shim path +(`cert-manager.io/cluster-issuer` annotation → Certificate `sso/kc-tls`), ISRG +Root X1 chaining (staging chains to the `(STAGING)` roots), or original-key +recovery. Those belong to the real attempt (RCLUSTER-WP-0018-T04 / NK49-T33). + +## Isolation + +Manifest: `install/cert-manager/rehearsal/kc-le-staging.yaml`, checked by +`tests/test_kc_le_staging_rehearsal.py`. + +| Object | Name | Why it is safe | +| --- | --- | --- | +| Namespace | `tls-rehearsal-kc` | throwaway; cleanup is deleting it | +| Issuer (namespaced) | `letsencrypt-staging-rehearsal` | staging directory only; not a ClusterIssuer, so no collision with any existing `letsencrypt-*` ClusterIssuer and nothing cluster-scoped is left behind | +| Certificate | `kc-le-staging-rehearsal` | `secretName: kc-le-staging-rehearsal-tls` in `tls-rehearsal-kc`; it cannot write `sso/kc-tls` | +| Generated Secrets | account key + staging leaf key | created by cert-manager in `tls-rehearsal-kc` only; destroyed with the namespace | + +No object references namespace `sso`. The manifest declares no email contact, +because Let's Encrypt no longer sends expiry mail and staging needs none. + +### HTTP-01 coexistence with the live ingress (risk) + +cert-manager creates a temporary solver Ingress in `tls-rehearsal-kc` for host +`kc.coulomb.social`, path `/.well-known/acme-challenge/`, and a solver +pod/Service there. Traefik merges Ingress rules across namespaces and ranks +routers by rule length. The solver's exact-path rule is longer than the live +`Host(kc.coulomb.social) && PathPrefix(/)`, so only the challenge path is +diverted. The live router, its middlewares and TLS are not touched. + +Residual risks, all short-lived: + +- The challenge path for that host is served by the solver for 1–2 minutes. It + is not a KeyCape path, so no real traffic depends on it. +- If a live `sso/kc-tls` renewal is pending at the same moment, both solvers + coexist. Each has its own token path, which is the normal cert-manager case. +- A default-deny NetworkPolicy, or a policy that blocks Traefik → new + namespaces, would leave the challenge pending. This shows as a timeout, not as + an outage. None is declared in railiance-cluster source, but live state has not + been checked. +- If Traefik ever adds a global HTTP→HTTPS redirect, Let's Encrypt follows it to + 443 and still validates. The current HelmChartConfig in source has no redirect. +- Staging limits are loose (for example 30,000 certificates per week per domain), + but do not loop on failures. Allow at most two attempts per session. + +## Operator commands + +Run them from a railiance-cluster checkout at the reviewed commit. + +```sh +cd ~/railiance-cluster +M=install/cert-manager/rehearsal/kc-le-staging.yaml +NS=tls-rehearsal-kc +WORK=$(mktemp -d) && chmod 700 "$WORK" + +# 0. Preflight (read-only) +python3 -m unittest tests/test_kc_le_staging_rehearsal.py # static guards pass +kubectl get crd certificates.cert-manager.io issuers.cert-manager.io +kubectl get namespace "$NS" 2>/dev/null && echo "ABORT: $NS already exists" +dig +short kc.coulomb.social A # expect 92.205.62.239 +curl -sS -o /dev/null -w '%{http_code}\n' http://kc.coulomb.social/.well-known/acme-challenge/probe # any HTTP status; a timeout means abort +LIVE_BEFORE=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social /dev/null \ + | openssl x509 -noout -fingerprint -sha256); echo "$LIVE_BEFORE" +kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}' # metadata only, no data +kubectl apply --dry-run=server -f "$M" # expect exactly 3 objects, all in $NS + +# 1. Apply +kubectl apply -f "$M" + +# 2. Wait (bounded) and observe +kubectl -n "$NS" wait certificate/kc-le-staging-rehearsal --for=condition=Ready --timeout=10m +kubectl -n "$NS" get certificate,certificaterequest,order,challenge,ingress -o wide + +# 3. Verify the staging chain (public cert only: the tls.crt key, never tls.key) +kubectl -n "$NS" get secret kc-le-staging-rehearsal-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > "$WORK/chain.pem" +openssl x509 -in "$WORK/chain.pem" -noout -subject -issuer -ext subjectAltName -dates +# expect: issuer CN begins "(STAGING)", SAN = DNS:kc.coulomb.social +curl -fsSL https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem -o "$WORK/stg-root-x1.pem" +openssl verify -CAfile "$WORK/stg-root-x1.pem" -untrusted "$WORK/chain.pem" "$WORK/chain.pem" # expect: OK +python3 tools/public-tls-metadata.py --public-cert "$WORK/chain.pem" --public-trust "$WORK/stg-root-x1.pem" +# expect: leaf_dns_sans == ["kc.coulomb.social"], server_chain_certificates >= 2 + +# 4. Confirm live TLS untouched +LIVE_AFTER=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social /dev/null \ + | openssl x509 -noout -fingerprint -sha256) +[ "$LIVE_BEFORE" = "$LIVE_AFTER" ] && echo "live cert unchanged" || echo "ABORT: live cert changed" +kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}' # same as preflight + +# 5. Cleanup (always, including after any abort) +kubectl delete namespace tls-rehearsal-kc --wait=true +kubectl get ingress -A | grep -i acme || echo "no solver ingress left" +rm -rf "$WORK" +``` + +Keep the JSON from step 3, the `openssl` subject/issuer/SAN/dates lines, and the +step 4 comparison as the public rehearsal receipt for NK-WP-0049-T33. Do not keep +`tls.key` or the account key. + +## Abort conditions + +Stop and go to step 5 cleanup if any of these is true: + +- The static guard test fails, or the server dry-run shows an object outside + `tls-rehearsal-kc` or a name other than the three expected. +- `tls-rehearsal-kc` already exists before apply. +- DNS for `kc.coulomb.social` does not resolve to the railiance01 ingress + address, or port 80 does not answer. +- The live fingerprint or the `sso/kc-tls` resourceVersion changes at any point. +- The Certificate is not Ready within 10 minutes, or the Challenge/Order shows + `invalid`. Capture `kubectl -n tls-rehearsal-kc describe challenge` first. At + most one retry, which means delete the namespace and apply again. +- The issued issuer CN does not begin with `(STAGING)`. That would mean the wrong + directory was used: stop and report. + +## Moving to the production attempt + +A green rehearsal (Ready, `(STAGING)` chain verifies, SAN correct, live unchanged, +cleanup done) is the entry condition for the production reissue in NK-WP-0049-T33. +That attempt runs through NetKingdom's `sso/keycape` ingress and +`ClusterIssuer/letsencrypt-prod` under RCLUSTER-WP-0018-T04's owner-attended gate. +It is not part of this runbook. diff --git a/install/cert-manager/rehearsal/kc-le-staging.yaml b/install/cert-manager/rehearsal/kc-le-staging.yaml new file mode 100644 index 0000000..3046270 --- /dev/null +++ b/install/cert-manager/rehearsal/kc-le-staging.yaml @@ -0,0 +1,59 @@ +# Let's Encrypt STAGING rehearsal for the kc.coulomb.social reissue path. +# +# Purpose: NK-WP-0049-T33 / RCLUSTER-WP-0018-T06. Founder decision 9f9d0d88 (A, +# 2026-10-21): recovery TLS for kc.coulomb.social is a reissue via +# letsencrypt-prod ACME HTTP-01 through Traefik, chained to ISRG Root X1. This +# file rehearses the same HTTP-01/Traefik path against the STAGING directory +# first, so a failed attempt never consumes the production duplicate-certificate +# limit (5 per week for the identical name set). +# +# Isolation by construction (asserted by tests/test_kc_le_staging_rehearsal.py): +# - every object lives in the throwaway namespace tls-rehearsal-kc; +# - a namespaced Issuer, not a ClusterIssuer: no cluster-scoped object, no +# name collision with any existing letsencrypt-* ClusterIssuer; +# - secretName kc-le-staging-rehearsal-tls in tls-rehearsal-kc, so it can +# never overwrite sso/kc-tls; nothing here references namespace sso; +# - no Secret is declared; cert-manager generates the ACME account key and the +# throwaway staging key inside tls-rehearsal-kc only. +# +# Runbook: docs/keycape-tls-le-staging-rehearsal.md +# Cleanup: kubectl delete namespace tls-rehearsal-kc +apiVersion: v1 +kind: Namespace +metadata: + name: tls-rehearsal-kc + labels: + app.kubernetes.io/part-of: railiance-cluster + railiance.io/purpose: le-staging-rehearsal + railiance.io/throwaway: "true" +--- +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: letsencrypt-staging-rehearsal + namespace: tls-rehearsal-kc +spec: + acme: + server: https://acme-staging-v02.api.letsencrypt.org/directory + privateKeySecretRef: + name: letsencrypt-staging-rehearsal-account-key + solvers: + - http01: + ingress: + ingressClassName: traefik +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: kc-le-staging-rehearsal + namespace: tls-rehearsal-kc +spec: + secretName: kc-le-staging-rehearsal-tls + dnsNames: + - kc.coulomb.social + issuerRef: + group: cert-manager.io + kind: Issuer + name: letsencrypt-staging-rehearsal + privateKey: + rotationPolicy: Always diff --git a/tests/test_kc_le_staging_rehearsal.py b/tests/test_kc_le_staging_rehearsal.py new file mode 100644 index 0000000..e45965b --- /dev/null +++ b/tests/test_kc_le_staging_rehearsal.py @@ -0,0 +1,68 @@ +"""Static guards for the kc.coulomb.social Let's Encrypt STAGING rehearsal. + +The rehearsal must never be able to touch the live sso/kc-tls Secret or use the +production ACME directory. These checks are offline; no cluster access. +""" +from pathlib import Path +import unittest + +import yaml + +ROOT = Path(__file__).parents[1] +MANIFEST = ROOT / "install/cert-manager/rehearsal/kc-le-staging.yaml" +RUNBOOK = ROOT / "docs/keycape-tls-le-staging-rehearsal.md" +NAMESPACE = "tls-rehearsal-kc" +STAGING = "https://acme-staging-v02.api.letsencrypt.org/directory" + + +def load(): + return [d for d in yaml.safe_load_all(MANIFEST.read_text()) if d] + + +def by_kind(docs, kind): + found = [d for d in docs if d["kind"] == kind] + assert len(found) == 1, kind + return found[0] + + +class StagingRehearsalManifest(unittest.TestCase): + def setUp(self): + self.docs = load() + + def test_only_expected_kinds_no_cluster_scope_no_secrets(self): + kinds = sorted(d["kind"] for d in self.docs) + self.assertEqual(kinds, ["Certificate", "Issuer", "Namespace"]) + self.assertNotIn("ClusterIssuer", kinds) + self.assertNotIn("Secret", kinds) + + def test_everything_is_in_the_throwaway_namespace(self): + self.assertEqual(by_kind(self.docs, "Namespace")["metadata"]["name"], NAMESPACE) + for doc in self.docs: + if doc["kind"] != "Namespace": + self.assertEqual(doc["metadata"]["namespace"], NAMESPACE) + self.assertNotIn("namespace: sso", MANIFEST.read_text()) + + def test_issuer_is_staging_only(self): + acme = by_kind(self.docs, "Issuer")["spec"]["acme"] + self.assertEqual(acme["server"], STAGING) + self.assertNotIn("acme-v02.api.letsencrypt.org", MANIFEST.read_text()) + self.assertNotIn("prod", acme["privateKeySecretRef"]["name"]) + self.assertEqual(acme["solvers"], [{"http01": {"ingress": {"ingressClassName": "traefik"}}}]) + + def test_certificate_cannot_overwrite_live_secret(self): + spec = by_kind(self.docs, "Certificate")["spec"] + self.assertNotEqual(spec["secretName"], "kc-tls") + self.assertIn("staging", spec["secretName"]) + self.assertEqual(spec["dnsNames"], ["kc.coulomb.social"]) + issuer = by_kind(self.docs, "Issuer")["metadata"]["name"] + self.assertEqual(spec["issuerRef"], {"group": "cert-manager.io", "kind": "Issuer", "name": issuer}) + + def test_runbook_names_manifest_and_cleanup(self): + text = RUNBOOK.read_text() + self.assertIn("install/cert-manager/rehearsal/kc-le-staging.yaml", text) + self.assertIn(f"kubectl delete namespace {NAMESPACE}", text) + self.assertIn("(STAGING)", text) + + +if __name__ == "__main__": + unittest.main() diff --git a/workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md b/workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md index f8ff4eb..2dd60f2 100644 --- a/workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md +++ b/workplans/RCLUSTER-WP-0018-keycape-original-tls-recovery.md @@ -8,7 +8,7 @@ status: active owner: codex topic_slug: railiance created: "2026-10-17" -updated: "2026-10-17" +updated: "2026-10-23" flavor: implementation depends_on: [] related: [CUST-WP-0080, NK-WP-0049, RCLUSTER-WP-0007, RCLUSTER-WP-0017] @@ -104,6 +104,12 @@ reissue are distinct claims; no self-signed trust substitute, new broker or implicit production action. Done when the exact source proposal, public metadata preflight, negatives, timeout, abort and custody obligations are reviewed. +Checkpoint 2026-10-23: founder decision 9f9d0d88 (A, 2026-10-21) selected the +reissue alternative for recovery TLS: letsencrypt-prod ACME HTTP-01 through +Traefik, chained to ISRG Root X1, no self-signed substitute, staging rehearsal +first. The staging rehearsal source is prepared under T06. This task's status is +left for owner review against T02. + ## Verify admitted original TLS recovery through the actual owner boundary ```task @@ -143,6 +149,27 @@ or rewrite a historical archive without those owner reviews/authority. Done when the future contract requirement has an exact tracked disposition/handoff; it is not proof of current archive coverage or actual future capture. +## Prepare the Let's Encrypt staging rehearsal for the kc.coulomb.social reissue + +```task +id: RCLUSTER-WP-0018-T06 +status: done +priority: high +assignee: claude +``` + +Source preparation for NK-WP-0049-T33 (request 6d5e748b, decision 9f9d0d88 = A). +Delivered install/cert-manager/rehearsal/kc-le-staging.yaml (throwaway namespace +tls-rehearsal-kc, a namespaced staging Issuer, and a Certificate whose target +name differs from the live one, so it cannot write sso/kc-tls), the runbook +docs/keycape-tls-le-staging-rehearsal.md (preflight, apply, bounded wait, +"(STAGING)" chain verification with openssl and tools/public-tls-metadata.py, a +check that the live certificate is unchanged, cleanup, abort conditions), and +static guards in tests/test_kc_le_staging_rehearsal.py. No kubectl or cluster +write happened, and no credential was read. Running the rehearsal is an +operator-attended step. Its public receipt is the entry condition for the +production reissue under T04. + ## Coordination and completion Beneficiary NK-WP-0049-T35/CUST-WP-0080 remains linked to this obligor record by