railiance-platform/docs/runbooks/rpf-wp-0044-t06-eso-adoption.md

RPF-WP-0044-T06 — adopt external-secrets into Argo (founder runbook)

Authority: D2 (89f1058f-33bc-47a8-8a49-0b44d426c67e). No further founder approval is needed. Highest-risk adoption: ESO serves every ExternalSecret on railiance01. Prep receipt: docs/evidence/2026-10-21-eso-adoption-prep/prep-receipt.json.

State at preparation (2026-10-21, agent, offline only)

• Draft argocd/railiance01/drafts/external-secrets.application.yaml: chart 0.16.1, ServerSideApply=true, no automated, no finalizer, no Force/Replace, and CRD values crds.annotations: argocd.argoproj.io/sync-options: Prune=false,Delete=false. The readiness gate passes the source check.
• Offline render (render sha256 55236645…f136f): 39 objects, made up of 20 CRDs (all 20 prune/delete protected), 3 Deployments, 3 ServiceAccounts, 5 ClusterRoles, 2 ClusterRoleBindings, 1 Role, 1 RoleBinding, 1 Service, 2 ValidatingWebhookConfigurations and the empty webhook-cert holder (metadata only). No ClusterSecretStore or ExternalSecret is rendered, so adoption cannot delete or replace a store. The webhooks declare no caBundle, so server-side apply does not take it from the cert-controller. ignoreDifferences is added only if the diff below shows drift.
• Not done by the agent: the fresh live diff, the store baseline and the fresh predecessor proofs. Since the CUST-WP-0073 cutover (2026-10-20) the agent's ssh railiance01 login (tegwick) cannot read the k3s kubeconfig, and the custodian-agent observer role has no CRD, RBAC, webhook or ESO read. Run every ssh railiance01 "kubectl …" below through a login that still has cluster-admin read.

Attended script (preferred, 2026-10-21)

The founder runs one script in their own terminal. It covers sections 1–4 below, with go/no-go gates and receipts under docs/evidence/2026-10-21-eso-adoption-prep/attended-<UTC stamp>/:

cd ~/railiance-platform && git pull --ff-only && (cd ~/rapp-issue-core && git pull --ff-only)
scripts/rpf_wp_0044_t06_attended.sh --precheck-only   # optional: pre-checks only, changes nothing
scripts/rpf_wp_0044_t06_attended.sh                    # pre-checks, create Application, type ADOPT, sync, post-checks

• Route. KUBECTL_ROUTE=founder-sudo is the script's default and runs ssh railiance01 sudo -n kubectl …. sudo -n never prompts, so no ssh -t is needed and the pipes stay clean. The same variable selects the route in workload_adoption_proof.py, argocd_adoption_proof.py, argocd_public_transport.py, activity_promotion_readiness.py, argocd_child_pin_drift.py, capture_service_observation.py and rapp-issue-core tools/verify_live.py. Their default stays agent, so an agent run never escalates.
• Smoke write. The pre-check runs issue-core verify-live --smoke, which creates and closes one Forgejo issue. The founder allowed this on 2026-10-21.
• Exit codes. 2 = no-go before any change, 4 = post-check failed (rollback steps are printed, nothing is pruned), 0 = adopted. --dry-run prints every command without running it.
• Credential-kind objects. They are split out of the render before any live read, and the evaluator (scripts/eso_attended_checks.py) refuses live input that contains one.

The manual sections below remain the reference for what the script does.

1. Pre-check (founder, or agent once read access exists)

cd ~/railiance-platform && git pull --ff-only
D=docs/evidence/2026-10-21-eso-adoption-prep
# a) fresh predecessor proofs (gate window is 300 s, so run all three back to back)
( cd ~/rapp-issue-core && make -s verify-live VERIFY_ARGS="--expected-revision 186c817b67d3919a64acdddbd3889e539841f946 --expected-source-url ssh://git@forgejo.coulomb.social:30022/coulomb/rapp-issue-core.git --smoke" ) > $D/issue-core-proof.json
python3 scripts/workload_adoption_proof.py --live --source-root ~/target-revenue --observation $D/target-revenue-observation.json --receipt $D/target-revenue-proof.json
python3 scripts/eso_adoption_readiness.py --draft argocd/railiance01/drafts/external-secrets.application.yaml --issue-core-proof $D/issue-core-proof.json --target-revenue-proof $D/target-revenue-proof.json | tee $D/readiness-r2.json
#    must print "ready_for_fresh_native_render_diff": true
# b) fresh render + client-side diff (T05 method); credential data in diff output is masked by kubectl
python3 -c "import yaml;print(yaml.safe_load(open('argocd/railiance01/drafts/external-secrets.application.yaml'))['spec']['source']['helm']['values'])" > /tmp/eso-values.yaml
helm template external-secrets external-secrets --repo https://charts.external-secrets.io --version 0.16.1 -n external-secrets -f /tmp/eso-values.yaml > /tmp/eso-render.yaml
ssh railiance01 "kubectl diff -f -" < /tmp/eso-render.yaml > $D/client-diff.txt; echo rc=$?
# c) store baseline
ssh railiance01 "kubectl get clustersecretstores -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[0].reason --no-headers" | tee $D/store-baseline-before.txt

Go only if all of these hold:

• the readiness gate is true;
• the diff touches CRDs in metadata only (the protective annotation) and changes no CRD spec;
• nothing is deleted or replaced;
• the baseline shows every store Valid except openbao-activity-core and openbao-email-connect (ValidationFailed).

If the diff shows webhook caBundle drift, an agent adds ignoreDifferences for ValidatingWebhookConfiguration /webhooks/0/clientConfig/caBundle and /webhooks/1/clientConfig/caBundle and re-runs b.

2. Create the Application (an agent, after the gate is green)

The agent moves the draft to argocd/railiance01/applications/external-secrets.application.yaml, commits and pushes, runs make argocd-pin-drift, then creates the child only with ssh railiance01 "kubectl apply --dry-run=server -f -" < <file> followed by the same command without --dry-run. The root also carries other OutOfSync children. Creating the Application syncs nothing.

3. The founder's sync command (one command)

ssh railiance01 "sudo -n kubectl -n argocd patch application external-secrets --type merge -p '{\"operation\":{\"initiatedBy\":{\"username\":\"rpf-wp-0044-t06\"},\"sync\":{\"revision\":\"0.16.1\",\"prune\":false,\"syncOptions\":[\"ServerSideApply=true\"],\"syncStrategy\":{\"apply\":{\"force\":false}}}}}'"

(Local double quotes; the remote shell receives the JSON inside single quotes. Prune off, force off, SSA.)

4. Post-sync checks

ssh railiance01 "kubectl -n argocd get application external-secrets -o custom-columns=SYNC:.status.sync.status,HEALTH:.status.health.status,PHASE:.status.operationState.phase,REV:.status.sync.revision"
#   expect Synced  Healthy  Succeeded  0.16.1
ssh railiance01 "kubectl -n external-secrets get deploy"   # 3/3 available, no restarts beyond rollout
ssh railiance01 "kubectl get crd -o name | grep -c external-secrets.io"   # still 20
ssh railiance01 "kubectl get clustersecretstores -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[0].reason --no-headers" > $D/store-baseline-after.txt
diff $D/store-baseline-before.txt $D/store-baseline-after.txt && echo baseline-unchanged
# ExternalSecret refresh sample (issue-core): force a refresh and confirm Ready
ssh railiance01 "kubectl -n issue-core annotate externalsecret issue-core-runtime force-sync=\$(date +%s) --overwrite && sleep 20 && kubectl -n issue-core get externalsecret issue-core-runtime -o custom-columns=READY:.status.conditions[0].status,REASON:.status.conditions[0].reason"
#   expect True SecretSynced

The agent records these as docs/evidence/<date>-eso-argocd-adoption.json, re-runs issue-core verify-live, and marks T06 done after the proving period.

Rollback (abort owner: railiance-platform)

Never prune and never delete the Application with cascade. Revert the Application to the previously reviewed chart 0.16.1 values in Git (git revert <adoption commit>), re-apply the child, then run the same sync command as above. To stop Argo management without touching workloads, delete only the Application object. It has no finalizer, so no cascade happens, and the CRDs are protected anyway.

Separate later step (not part of adoption)

Retire the stale pending-upgrade Helm release record (sh.helm.release.v1.external-secrets.v*) under its own reviewed cleanup. ArgoCD does not track it, and it must not be replaced silently.