RCLUSTER-WP-0018-T06: Let's Encrypt staging rehearsal for kc.coulomb.social reissue
Throwaway namespace, namespaced staging Issuer and a Certificate with a distinct target name (cannot touch the live kc-tls object), operator runbook with verify/cleanup/abort, and static guard tests. Source only; for NK-WP-0049-T33 (decision 9f9d0d88 = A). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 696317@bnt-lap001 Assistant-Session: bab9e511-0915-45b3-926a-2fab552edff6
This commit is contained in:
codex
parent
0e8df471c7
commit
a7c0e5b666
5 changed files with 298 additions and 1 deletions
142
docs/keycape-tls-le-staging-rehearsal.md
Normal file
142
docs/keycape-tls-le-staging-rehearsal.md
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
# kc.coulomb.social TLS reissue — Let's Encrypt STAGING rehearsal
|
||||
|
||||
Work record: RCLUSTER-WP-0018-T06, for NK-WP-0049-T33 (request message
|
||||
6d5e748b). Founder decision 9f9d0d88 (A, 2026-10-21): recovery TLS for
|
||||
`kc.coulomb.social` is a **reissue** through `letsencrypt-prod` ACME HTTP-01 via
|
||||
Traefik, chained to ISRG Root X1. No self-signed substitute. Rehearse on staging
|
||||
first so a failed attempt cannot use up the production limit of 5 duplicate
|
||||
certificates per week.
|
||||
|
||||
This page is source preparation. It authorizes no cluster change. An operator
|
||||
runs the commands below, attended, from a shell with cluster-admin kubeconfig
|
||||
on railiance01 or through the usual tunnel.
|
||||
|
||||
## What the rehearsal proves, and what it does not
|
||||
|
||||
It proves: public DNS for `kc.coulomb.social` reaches Traefik on port 80, Traefik
|
||||
routes `/.well-known/acme-challenge/*` for that host to a cert-manager solver
|
||||
while the live KeyCape ingress keeps serving, cert-manager can register an ACME
|
||||
account and finish an order, and the result is a chain for exactly
|
||||
`kc.coulomb.social`.
|
||||
|
||||
It does not prove: anything about the production directory, the production
|
||||
account key `letsencrypt-prod-account-key`, the `sso/keycape` ingress-shim path
|
||||
(`cert-manager.io/cluster-issuer` annotation → Certificate `sso/kc-tls`), ISRG
|
||||
Root X1 chaining (staging chains to the `(STAGING)` roots), or original-key
|
||||
recovery. Those belong to the real attempt (RCLUSTER-WP-0018-T04 / NK49-T33).
|
||||
|
||||
## Isolation
|
||||
|
||||
Manifest: `install/cert-manager/rehearsal/kc-le-staging.yaml`, checked by
|
||||
`tests/test_kc_le_staging_rehearsal.py`.
|
||||
|
||||
| Object | Name | Why it is safe |
|
||||
| --- | --- | --- |
|
||||
| Namespace | `tls-rehearsal-kc` | throwaway; cleanup is deleting it |
|
||||
| Issuer (namespaced) | `letsencrypt-staging-rehearsal` | staging directory only; not a ClusterIssuer, so no collision with any existing `letsencrypt-*` ClusterIssuer and nothing cluster-scoped is left behind |
|
||||
| Certificate | `kc-le-staging-rehearsal` | `secretName: kc-le-staging-rehearsal-tls` in `tls-rehearsal-kc`; it cannot write `sso/kc-tls` |
|
||||
| Generated Secrets | account key + staging leaf key | created by cert-manager in `tls-rehearsal-kc` only; destroyed with the namespace |
|
||||
|
||||
No object references namespace `sso`. The manifest declares no email contact,
|
||||
because Let's Encrypt no longer sends expiry mail and staging needs none.
|
||||
|
||||
### HTTP-01 coexistence with the live ingress (risk)
|
||||
|
||||
cert-manager creates a temporary solver Ingress in `tls-rehearsal-kc` for host
|
||||
`kc.coulomb.social`, path `/.well-known/acme-challenge/<token>`, and a solver
|
||||
pod/Service there. Traefik merges Ingress rules across namespaces and ranks
|
||||
routers by rule length. The solver's exact-path rule is longer than the live
|
||||
`Host(kc.coulomb.social) && PathPrefix(/)`, so only the challenge path is
|
||||
diverted. The live router, its middlewares and TLS are not touched.
|
||||
|
||||
Residual risks, all short-lived:
|
||||
|
||||
- The challenge path for that host is served by the solver for 1–2 minutes. It
|
||||
is not a KeyCape path, so no real traffic depends on it.
|
||||
- If a live `sso/kc-tls` renewal is pending at the same moment, both solvers
|
||||
coexist. Each has its own token path, which is the normal cert-manager case.
|
||||
- A default-deny NetworkPolicy, or a policy that blocks Traefik → new
|
||||
namespaces, would leave the challenge pending. This shows as a timeout, not as
|
||||
an outage. None is declared in railiance-cluster source, but live state has not
|
||||
been checked.
|
||||
- If Traefik ever adds a global HTTP→HTTPS redirect, Let's Encrypt follows it to
|
||||
443 and still validates. The current HelmChartConfig in source has no redirect.
|
||||
- Staging limits are loose (for example 30,000 certificates per week per domain),
|
||||
but do not loop on failures. Allow at most two attempts per session.
|
||||
|
||||
## Operator commands
|
||||
|
||||
Run them from a railiance-cluster checkout at the reviewed commit.
|
||||
|
||||
```sh
|
||||
cd ~/railiance-cluster
|
||||
M=install/cert-manager/rehearsal/kc-le-staging.yaml
|
||||
NS=tls-rehearsal-kc
|
||||
WORK=$(mktemp -d) && chmod 700 "$WORK"
|
||||
|
||||
# 0. Preflight (read-only)
|
||||
python3 -m unittest tests/test_kc_le_staging_rehearsal.py # static guards pass
|
||||
kubectl get crd certificates.cert-manager.io issuers.cert-manager.io
|
||||
kubectl get namespace "$NS" 2>/dev/null && echo "ABORT: $NS already exists"
|
||||
dig +short kc.coulomb.social A # expect 92.205.62.239
|
||||
curl -sS -o /dev/null -w '%{http_code}\n' http://kc.coulomb.social/.well-known/acme-challenge/probe # any HTTP status; a timeout means abort
|
||||
LIVE_BEFORE=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social </dev/null 2>/dev/null \
|
||||
| openssl x509 -noout -fingerprint -sha256); echo "$LIVE_BEFORE"
|
||||
kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}' # metadata only, no data
|
||||
kubectl apply --dry-run=server -f "$M" # expect exactly 3 objects, all in $NS
|
||||
|
||||
# 1. Apply
|
||||
kubectl apply -f "$M"
|
||||
|
||||
# 2. Wait (bounded) and observe
|
||||
kubectl -n "$NS" wait certificate/kc-le-staging-rehearsal --for=condition=Ready --timeout=10m
|
||||
kubectl -n "$NS" get certificate,certificaterequest,order,challenge,ingress -o wide
|
||||
|
||||
# 3. Verify the staging chain (public cert only: the tls.crt key, never tls.key)
|
||||
kubectl -n "$NS" get secret kc-le-staging-rehearsal-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > "$WORK/chain.pem"
|
||||
openssl x509 -in "$WORK/chain.pem" -noout -subject -issuer -ext subjectAltName -dates
|
||||
# expect: issuer CN begins "(STAGING)", SAN = DNS:kc.coulomb.social
|
||||
curl -fsSL https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem -o "$WORK/stg-root-x1.pem"
|
||||
openssl verify -CAfile "$WORK/stg-root-x1.pem" -untrusted "$WORK/chain.pem" "$WORK/chain.pem" # expect: OK
|
||||
python3 tools/public-tls-metadata.py --public-cert "$WORK/chain.pem" --public-trust "$WORK/stg-root-x1.pem"
|
||||
# expect: leaf_dns_sans == ["kc.coulomb.social"], server_chain_certificates >= 2
|
||||
|
||||
# 4. Confirm live TLS untouched
|
||||
LIVE_AFTER=$(openssl s_client -connect kc.coulomb.social:443 -servername kc.coulomb.social </dev/null 2>/dev/null \
|
||||
| openssl x509 -noout -fingerprint -sha256)
|
||||
[ "$LIVE_BEFORE" = "$LIVE_AFTER" ] && echo "live cert unchanged" || echo "ABORT: live cert changed"
|
||||
kubectl -n sso get secret kc-tls -o jsonpath='{.metadata.resourceVersion}{"\n"}' # same as preflight
|
||||
|
||||
# 5. Cleanup (always, including after any abort)
|
||||
kubectl delete namespace tls-rehearsal-kc --wait=true
|
||||
kubectl get ingress -A | grep -i acme || echo "no solver ingress left"
|
||||
rm -rf "$WORK"
|
||||
```
|
||||
|
||||
Keep the JSON from step 3, the `openssl` subject/issuer/SAN/dates lines, and the
|
||||
step 4 comparison as the public rehearsal receipt for NK-WP-0049-T33. Do not keep
|
||||
`tls.key` or the account key.
|
||||
|
||||
## Abort conditions
|
||||
|
||||
Stop and go to step 5 cleanup if any of these is true:
|
||||
|
||||
- The static guard test fails, or the server dry-run shows an object outside
|
||||
`tls-rehearsal-kc` or a name other than the three expected.
|
||||
- `tls-rehearsal-kc` already exists before apply.
|
||||
- DNS for `kc.coulomb.social` does not resolve to the railiance01 ingress
|
||||
address, or port 80 does not answer.
|
||||
- The live fingerprint or the `sso/kc-tls` resourceVersion changes at any point.
|
||||
- The Certificate is not Ready within 10 minutes, or the Challenge/Order shows
|
||||
`invalid`. Capture `kubectl -n tls-rehearsal-kc describe challenge` first. At
|
||||
most one retry, which means delete the namespace and apply again.
|
||||
- The issued issuer CN does not begin with `(STAGING)`. That would mean the wrong
|
||||
directory was used: stop and report.
|
||||
|
||||
## Moving to the production attempt
|
||||
|
||||
A green rehearsal (Ready, `(STAGING)` chain verifies, SAN correct, live unchanged,
|
||||
cleanup done) is the entry condition for the production reissue in NK-WP-0049-T33.
|
||||
That attempt runs through NetKingdom's `sso/keycape` ingress and
|
||||
`ClusterIssuer/letsencrypt-prod` under RCLUSTER-WP-0018-T04's owner-attended gate.
|
||||
It is not part of this runbook.
|
||||
Loading…
Add table
Add a link
Reference in a new issue