railiance-platform/workplans/RPF-WP-0044-argocd-phase-b-adopt-existing-applications.md

424 lines
33 KiB
Markdown
Raw Permalink Normal View History

---
id: RPF-WP-0044
type: workplan
title: "ArgoCD phase B: adopt the four existing Applications on railiance01"
domain: financials
repo: railiance-platform
status: blocked
flavor: planning
owner: railiance-platform
topic_slug: railiance
created: "2026-09-21"
updated: "2026-10-11"
related: [RPF-WP-0043, RPF-WP-0022]
state_hub_workstream_id: "98140775-3b9a-5cf9-9af6-722502d487dc"
---
The founder, Bernd Worsch, decided ArchitectureBlueprint §5.6 on 2026-09-21,
exercising `GOVERN @ estate`: option 1, **adopt properly**. Phase A installed
Argo CD Core v3.5.3 on railiance01 (record:
`the-custodian/docs/kubernetes-change-gate-decision.md`). This plan is phase B:
bring the four Applications in `argocd/applications/` under that ArgoCD, **one
at a time, each with its own founder go-ahead**. Terms follow SecurityCanon
Mode of Authority v0.2.0 (draft).
**This plan changes nothing live.** Every task that would change railiance01
is `wait` on the founder. Each live step is `ADMINISTER @ realm:kubernetes/railiance01`
(bootstrap) or `CONSTRUCT @ railiance-platform` with the merge as the gate,
both `activation=APPROVED`.
Related, kept separate: `RPF-WP-0043` onboards `rapp-policy-nexus`, which is
not one of these four. It needs T02 of this plan (a working root app on
railiance01) before its own T04. Phase C, retiring coulombcore's ArgoCD, is
not planned here (T08).
## Read-only evidence, 2026-09-21
Full record: `docs/evidence/argocd-phase-b-adoption-diff-2026-09-21.json`.
Rendered locally (`helm template`, `kubectl kustomize`), then `kubectl diff`
client-side and `--server-side` dry run over `ssh railiance01`. Nothing was
applied.
| App | Live spec diff | Would prune | Repo credential | Blocker |
| --- | --- | --- | --- | --- |
| openbao-secretstore | none (1 ClusterSecretStore) | nothing | none, public | none |
| issue-core | none **after repointing** | nothing | **new**: `rapp-issue-core` is private | declared path `issue-core.git k8s/railiance` no longer exists |
| target-revenue | none on Deployment/Service/Ingress/ExternalSecret/Cluster | nothing | none, public | CNPG `Cluster` not in `railiance-tenants` whitelist; two Sync-hook Jobs |
| external-secrets | none (39 objects, 20 CRDs) | nothing now | none, public Helm repo | cluster-wide blast radius; CRDs must never be pruned |
Nothing is pruned at adoption because no live object carries an ArgoCD
tracking annotation; ArgoCD only prunes what it tracks. The one change every
adoption makes is that tracking annotation, on metadata only, so no rollout.
Sync settings found: `railiance-apps-root` and all four children declare
`automated: {prune: true, selfHeal: true}`; none has a finalizer. Applying
`argocd/bootstrap/` as committed would therefore adopt all four at once. T01
removes that before anything is applied.
Also found: the `external-secrets` Helm release is stuck at revision 7,
`pending-upgrade`, since 2026-07-07 (values identical to the Application);
`issue-core` is now server-side applied by `rapp-issue-core`;
`target-revenue` runs a mutable image tag (`0.1.4`). ArgoCD on railiance01
has 0 AppProjects, 0 Applications, 0 repository Secrets, and there is no
`argocd` CLI on the node, so the commands below use `kubectl` against the
Application resource. Not verified: repo-server egress to Forgejo and
`charts.external-secrets.io`; T02 proves it.
## Neutralise automated sync and fix the declared state
```task
id: RPF-WP-0044-T01
status: done
priority: high
state_hub_task_id: "2d7230f5-50c8-5eee-a050-1501ce5af62a"
```
**Done 2026-09-21, commit `c3ebd6d`.** The founder chose **Option A**
(`GOVERN @ estate`): a railiance01-only source path. Repository-only; nothing
applied. `argocd/applications/` and `argocd/bootstrap/` are untouched, because
coulombcore's ArgoCD still reads `argocd/applications/` on `main` and its
bootstrap may be re-applied from `argocd/bootstrap/`. coulombcore could not be
read in this session, so the railiance01 bootstrap has its own path.
Declared (all under `argocd/railiance01/`):
1. `bootstrap/`: the three AppProjects and `railiance-apps-root` with **no
`automated` block**, path `argocd/railiance01/applications`,
`targetRevision: main` (each sync pins its revision in the operation).
2. `bootstrap/01-railiance-tenants-project.yaml`: `postgresql.cnpg.io/Cluster`
added to `namespaceResourceWhitelist`. The other two AppProjects are copies
of `argocd/bootstrap/`.
3. `applications/` holds only `README.md`. A placeholder **is** needed: git does
not keep an empty directory and a missing path is a `ComparisonError`.
ArgoCD's directory source reads only `*.yaml`/`*.yml`/`*.json`, so it renders
zero children (confirmed live by T02).
4. `drafts/`: the four children, no `automated`, no finalizer, pinned:
openbao-secretstore `d2dbc19` (railiance-platform), issue-core `a78f38c`
(`rapp-issue-core.git`, path `manifests`), target-revenue `f1109d5`,
external-secrets chart `0.16.1`. No root syncs `drafts/`.
5. `argocd/repositories/rapp-issue-core.repository.sops.yaml.template` (no
value); `issue-core.repository.sops.yaml.template` marked obsolete.
Offline proof: `kubectl kustomize argocd/railiance01/bootstrap` renders exactly
3 AppProjects and 1 Application with no `automated` field;
`ssh railiance01 'kubectl apply --dry-run=server -f -'` of that render
returned rc=0 (all four "created (server dry run)").
Hazard kept as is: `make argocd-bootstrap-deploy` still defaults to
`argocd/bootstrap` (automated root). Do not run it on railiance01.
## Apply the AppProjects and the root app, automated sync off (live)
```task
id: RPF-WP-0044-T02
status: done
priority: high
state_hub_task_id: "7d97ee9c-b2fb-520f-8468-3ecf7eea6c01"
```
**Done 2026-09-21** (founder go-ahead; custodian session). 3 AppProjects + `railiance-apps-root` applied from the clean export of `c3ebd6d`; root Synced/Healthy, op Succeeded at `c3ebd6d`, zero children, 0 ArgoCD-managed namespaces. Repo-server reaches Forgejo. One `OrphanedResourceWarning` (project-level, informational).
Applies the render of
`argocd/railiance01/bootstrap` at commit `c3ebd6dddc830373c75144da178f829b47c3e5de`, from a clean export (not
the working tree), then syncs the root by hand at that commit.
```bash
cd ~/railiance-platform && git fetch origin && git merge-base --is-ancestor c3ebd6dddc830373c75144da178f829b47c3e5de origin/main && echo pinned-commit-on-main
R=$(mktemp -d) && git -C ~/railiance-platform archive c3ebd6dddc830373c75144da178f829b47c3e5de argocd/railiance01/bootstrap | tar -x -C "$R"
kubectl kustomize "$R/argocd/railiance01/bootstrap" > "$R/bootstrap.yaml" && grep -c '^kind: AppProject' "$R/bootstrap.yaml" && grep -c automated "$R/bootstrap.yaml" # expect 3 and 0
ssh railiance01 'kubectl apply --dry-run=server -f -' < "$R/bootstrap.yaml"
ssh railiance01 'kubectl apply -f -' < "$R/bootstrap.yaml"
ssh railiance01 'kubectl -n argocd patch application railiance-apps-root --type merge -p "{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"c3ebd6dddc830373c75144da178f829b47c3e5de\",\"prune\":false}}}"'
ssh railiance01 'kubectl -n argocd get appprojects,applications'
ssh railiance01 'kubectl -n argocd get application railiance-apps-root -o jsonpath="{.status.sync.status} {.status.operationState.phase} {.status.operationState.syncResult.revision} resources={.status.resources}{\"\n\"}"'
```
Expect three AppProjects and `railiance-apps-root` `Synced`, operation
`Succeeded` at `c3ebd6d`, and no resources (zero children). A
`ComparisonError` means the repo-server cannot reach Forgejo: stop and fix
egress before any app task. No repository Secret is needed;
`coulomb/railiance-platform` is public.
Rollback: `kubectl -n argocd delete application railiance-apps-root` (no
finalizer, no children), then delete the three AppProjects. Nothing else is
touched.
## Per-app procedure (T03–T06)
Every adoption task follows the same steps. `<app>` and `<sha>` vary.
1. **Diff.** Re-run the evidence render and `kubectl diff --server-side` at
the commit or chart version being pinned. Any spec change, any missing
whitelist kind, or a different image: stop.
2. **Merge** the child Application (no `automated`) from
`argocd/railiance01/drafts/` into `argocd/railiance01/applications/`, then sync the root by hand so it creates the child object only:
```bash
ssh railiance01 'kubectl -n argocd patch application railiance-apps-root --type merge -p "{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"<sha>\",\"prune\":false}}}"'
```
3. **ArgoCD's own diff:** read
`kubectl -n argocd get application <app> -o jsonpath='{.status.sync.status} {.status.health.status}'`
and `.status.resources`. Only metadata differences are acceptable.
4. **Manual sync, prune off, hooks skipped** (`apply` strategy):
```bash
ssh railiance01 'kubectl -n argocd patch application <app> --type merge -p "{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"<sha>\",\"prune\":false,\"syncStrategy\":{\"apply\":{}}}}}"'
```
Then verify Synced/Healthy and run the owner's live check.
5. **Prove it:** Synced and Healthy for at least 24 hours (7 days for
external-secrets). Only then, with a **second** go-ahead, a merge adds
`automated: {selfHeal: true}`; `prune: true` is a third step once the
app's tracked set is confirmed complete.
6. Record `docs/evidence/<date>-<app>-argocd-adoption.json`.
Rollback:
- **R1, before automated sync:** revert the merge, then
`kubectl -n argocd delete application <app>`. No finalizer, so nothing
cascades; the workload keeps running. Tracking annotations stay and are
harmless.
- **R2, after automated sync:** revert the automated-sync commit first and
sync the root by hand, then R1. Never hand-patch a child while the root
self-heals; that is `BREAK_GLASS`, recorded and reconciled into git.
- **Workload restore:** only once ArgoCD no longer manages the app, with the
previous deploy method named in the task.
## Adopt openbao-secretstore (lowest risk)
```task
id: RPF-WP-0044-T03
status: done
decision_id: 89f1058f-33bc-47a8-8a49-0b44d426c67e
priority: high
state_hub_task_id: "6d5fc801-e361-579b-bcd8-6fe719a82e94"
```
2026-10-11: requalified after founder decision D2 (ArgoCD owns production) resolved — see the-custodian docs/decisions/2026-10-11-security-unblock-decisions.md.
**Adopted 2026-09-21** (founder go-ahead; custodian session): diff at `d2dbc19` rc=0; child merged in `182e788`; root synced at `182e788`; child manual sync (apply strategy, prune off) Succeeded at `d2dbc19`, Synced/Healthy, automated off. `ClusterSecretStore/openbao` Valid with tracking annotation; `issue-core-runtime` SecretSynced. Status `progress` = in the 24h proving period; `selfHeal` needs a second go-ahead. Evidence: `docs/evidence/2026-09-21-openbao-secretstore-argocd-adoption.json`.
One object,
`ClusterSecretStore/openbao`, zero diff, public source. It is the store
issue-core's `ExternalSecret` reads, so adopt it before issue-core. The 24
other ClusterSecretStores on railiance01 are not in the kustomization and stay
untracked (declared gap, RPF-WP-0043-T05). Live check:
`kubectl get clustersecretstore openbao` stays `Valid`, and
`externalsecret/issue-core-runtime` stays `SecretSynced`. Workload restore:
`kubectl apply -f argocd/platform-addons/openbao-secretstore/openbao.clustersecretstore.yaml`.
Exact commands (the root sync pins the merge commit `$SHA`; the child is
pinned to `d2dbc19`, the last commit touching its source path):
```bash
# 1. Diff at the pinned source commit (expect no output, rc=0)
cd ~/railiance-platform && git fetch origin && git checkout main && git pull --ff-only
R=$(mktemp -d) && git archive d2dbc19c254247652c49fda8721c80d53bca206a argocd/platform-addons/openbao-secretstore | tar -x -C "$R"
ssh railiance01 'kubectl diff --server-side -f -' < <(kubectl kustomize "$R/argocd/platform-addons/openbao-secretstore"); echo rc=$?
# 2. Merge the child (drop the 3-line DRAFT header), push, sync the root at that commit
sed '1,3d' argocd/railiance01/drafts/openbao-secretstore.application.yaml > argocd/railiance01/applications/openbao-secretstore.application.yaml
git rm -q argocd/railiance01/drafts/openbao-secretstore.application.yaml
git add argocd/railiance01/applications/openbao-secretstore.application.yaml
git commit -m "Adopt openbao-secretstore on railiance01 (RPF-WP-0044-T03)" && git push origin main
SHA=$(git rev-parse HEAD) && echo "$SHA"
ssh railiance01 "kubectl -n argocd patch application railiance-apps-root --type merge -p '{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"$SHA\",\"prune\":false}}}'"
# 3. ArgoCD's own diff (expect OutOfSync or Synced, one resource ClusterSecretStore/openbao, metadata-only)
ssh railiance01 'kubectl -n argocd get application openbao-secretstore -o jsonpath="{.status.sync.status} {.status.health.status} {.status.resources}{\"\n\"}"'
# 4. Manual sync, prune off, apply strategy (hooks skipped)
ssh railiance01 'kubectl -n argocd patch application openbao-secretstore --type merge -p "{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"d2dbc19c254247652c49fda8721c80d53bca206a\",\"prune\":false,\"syncStrategy\":{\"apply\":{}}}}}"'
ssh railiance01 'kubectl -n argocd get application openbao-secretstore -o jsonpath="{.status.sync.status} {.status.health.status} {.status.operationState.phase}{\"\n\"}"'
ssh railiance01 'kubectl get clustersecretstore openbao; kubectl -n issue-core get externalsecret issue-core-runtime'
```
Rollback (R1): `git revert` the merge commit and push, then
`ssh railiance01 'kubectl -n argocd delete application openbao-secretstore'`
(no finalizer; the ClusterSecretStore keeps running).
2026-10-03: human gate routed to hub decision 89f1058f-33bc-47a8-8a49-0b44d426c67e (D2: ArgoCD owns production: go-ahead sequence for railiance-apps-root, issue-core, policy-nexus and activity-core automation) by CUST-WP-0075; resolving that decision raises C-40 here.
2026-10-11: **24h adoption proof verified; T03 done.**
`scripts/argocd_adoption_proof.py` freshly verifies estate IP 92.205.62.239,
local-cluster destination, exact source pin `d2dbc19`, one tracked
ClusterSecretStore/openbao, manual idle policy, successful Sep21 apply/prune-off
sync, exact ownership annotation and uninterrupted Ready conditions on the store
and issue-core consumer since Aug19. The completed proving period is 1,735,984
seconds. Current Application is Synced/Healthy; the nonblocking
OrphanedResourceWarning is retained and no broader namespace adoption is claimed.
Both server-side and client-side native pinned-render diffs return **0** after
canonical Argo v3 annotation injection. This is bound to the observed v3.5.3
controller, `argocd-cm` tracking configuration and the Application destination
namespace. The cluster-scoped store's tracking ID uses `external-secrets` as
Argo's destination-namespace fallback; no instance label is injected in annotation
mode. Same-source-API-version server dry-runs show **no changed JSON paths**.
Only source-derived desired fields and Argo's exact tracking annotation are
rendered; no live values are copied into the desired source, and unknown or
substantive diffs fail closed. Upstream implementation:
https://github.com/argoproj/argo-cd/blob/v3.5.3/util/argo/resource_tracking.go
(`UnstructuredToAppInstanceValue`, `SetAppInstance`).
Evidence: `docs/evidence/2026-10-11-openbao-adoption-proof/`:
`observation-argo-normalized.json`, `proof-argo-normalized-final.json`, and
`validation.json`. The proof binds the verifier and observation digests. Thirteen
meaningful fixture tests and compilation pass. Earlier empty-namespace and raw
source diff refusals remain visible: raw source omits Argo-owned tracking and
last-applied metadata; an initial API-version comparison also used the API
default rather than the pinned source version. No native mutation, Secret read
or annotation value/error output occurred. This closes the existing adoption's
proving period without changing sync/automation policy or using a new human gate.
## Adopt issue-core (from rapp-issue-core)
```task
id: RPF-WP-0044-T04
status: done
intervention_note: "Orange attended Platform OpenBao OIDC/MFA login for the already D2-approved exact repository reader; no second source-scope approval."
decision_id: 89f1058f-33bc-47a8-8a49-0b44d426c67e
priority: high
state_hub_task_id: "0e619658-0453-51ed-b9a0-bed28b5544fd"
```
D2 already authorizes this exact private repository read-only reader and the ordered adoption. No renewed founder go-ahead is required. The remaining human act is the existing Platform OpenBao attended OIDC/MFA login; route it through the single attended-window checklist.
Source preparation now disables wrapper direct deployment/rollback and verifies Argo ownership using the exact HTTPS or SSH source URL. The fixed credential executor is `scripts/provision_argocd_issue_core_repository.py`, reviewed in `docs/evidence/2026-10-11-issue-core-credential/owner-review.json`. It creates one repository-only read-only SSH deploy key, stores it at `platform/operators/argocd/repositories/rapp-issue-core`, response-wraps for 60 seconds, unwraps once, proves consumed-token denial, and creates only `argocd/rapp-issue-core-repository`. Tests and installed Bao synthetic checks pass; native issuance/adoption have not run. The contract example is incomplete and cannot authorize execution.
Public host trust and exact project admission are prepared in owner source and the bounded `scripts/argocd_public_transport.py`; its server dry-run passed. Preserve each exact attempt and any partial failure. Finish those prerequisites, seal the owner-reviewed contract, use the attended login, then pin the newly committed wrapper source in the Application, inspect client-side native diff, manually sync with prune/force disabled, and run the bounded owner verifier. Image releases and rollback use Git revisions plus Argo; direct `make rollback` is disabled. Never adopt the stale wrapper revision that predates the ExternalSecret metadata guard.
2026-10-20 (CUST-WP-0079-T06 attended-window prep, agent): prerequisites done. Host trust and exact project admission were executed 2026-10-11 (the-custodian `docs/evidence/2026-10-11-issue-core-public-transport/execution.json`); a fresh dry-run refused with `target_preimage_changed` (already applied, kept as `docs/evidence/2026-10-11-issue-core-transport/2026-10-20-dry-run-r1.json`) and `--action verify` passed (`2026-10-20-verify-r1.json`). ssh-keyscan from railiance01 matches the pinned provider key. Sealed contract `docs/evidence/2026-10-11-issue-core-credential/contract-sealed-2026-10-20.json` (sha256 `1ce9d2fdc39395407a01e358bfd0179f83f35c0a448e6658eb8f5aab351d3d3a`, expires 2026-10-27) passes the executor's `validate_contract`; 17 executor/transport tests pass. Wrapper pin target is rapp-issue-core `186c817b67d3919a64acdddbd3889e539841f946`. The only remaining step is the founder's attended login running the executor; exact commands and the agent follow-on (pin, diff, manual sync, verifier) are in `docs/runbooks/rpf-wp-0044-t04-attended.md`. No secret, credential or OpenBao call was made.
2026-10-20 (adoption, agent, founder explicit go + D2): the journal `contract_sha256` 0fd6d6fa... is `sha256(json.dumps(contract, sort_keys=True))` of the sealed contract (recomputed, exact match; the raw file hash 1ce9d2fd... was enforced at launch). Child placed in `argocd/railiance01/applications/issue-core.application.yaml` (SSH repoURL, 186c817) in `d70422c` and applied directly, because `railiance-apps-root` is also OutOfSync on activity-core, informed-decision-requester and state-hub-memo-dispatch (`make argocd-pin-drift`: all ok). Argo fetched the SSH source at 186c817 without ComparisonError; Healthy, OutOfSync (8 untracked objects). Client-side diff of the pinned render (T05 method): rc=0; the render holds no credential objects. **Manual sync not performed**: the agent harness auto-mode classifier denied the Application sync-operation patch. Nothing was rolled back because no workload changed; the new Application has no automated sync. Remaining: the founder runs (or permits) runbook step 4 sync, then the step 5 verifier. Receipt: `docs/evidence/2026-10-20-issue-core-argocd-adoption.json`.
2026-10-20 (attended window, CUST-WP-0079-T06): founder completed the Platform OpenBao OIDC/MFA login and the fixed executor at 10:37Z. Journal `docs/evidence/2026-10-11-issue-core-credential/execution-journal-r1.json`: state `delivered_not_adopted`, consumed-wrap replay denied (400), no secret values observed by the agent, custody version 1, destination `argocd/rapp-issue-core-repository`. The human gate is cleared. Remaining agent steps per `docs/runbooks/rpf-wp-0044-t04-attended.md`: pin the Application, client-side diff, manual sync (prune/force off), bounded verifier. The journal's `contract_sha256` (`0fd6d6fa…`) differs from the sealed file's sha256 (`1ce9d2fd…`). Reconcile this before the adoption sync.
2026-10-20 adoption complete. The founder ran the manual sync (apply strategy, prune/force off). Argo reports issue-core Synced/Healthy at `186c817`, operation Succeeded. The bounded verifier `make verify-live` passed with all checks true: healthz 200, unauthenticated ingest 401, immutable image, ExternalSecret Ready, no secret values observed. Receipt: `docs/evidence/2026-10-20-issue-core-verify-live.json`. The runbook's ssh quoting was fixed. Automated sync is still off; enabling it is a separate decision. T04 done.
## Adopt target-revenue
```task
id: RPF-WP-0044-T05
status: done
decision_id: 89f1058f-33bc-47a8-8a49-0b44d426c67e
priority: medium
state_hub_task_id: "d418068a-6fb0-5416-aac5-d23c93924d9c"
```
2026-10-11: requalified after founder decision D2 (ArgoCD owns production) resolved — see the-custodian docs/decisions/2026-10-11-security-unblock-decisions.md.
**Adopted 2026-09-21** (founder go-ahead; custodian session). The *server-side* diff at `f1109d5` conflicted on `resources.limits.cpu` (live `1` owned by `kubectl-client-side-apply` vs declared `1000m`: the same quantity); the *client-side* diff, which is what the apply strategy uses, was rc=0 for all five non-hook objects. Child merged in `6eeb826`; manual sync with the `apply` strategy, prune off: Succeeded, Synced/Healthy, **0 Jobs created**, same pod (36d, no restart), CNPG healthy 1/1, ExternalSecret SecretSynced, `https://revenue.coulomb.social/healthz` 200 (the live check path is `/healthz`, not `/health`). Status `progress` = proving period. Automated sync still needs target-revenue's confirmation on the two hook Jobs (asked 2026-09-21). Evidence: `docs/evidence/2026-09-21-target-revenue-argocd-adoption.json`.
Original preconditions: T01 item 2 (CNPG
`Cluster` whitelisted) applied to the AppProject by hand, and `target-revenue`
confirms the two Sync-hook Jobs (`target-revenue-migrate`,
`target-revenue-bootstrap-binky`) are safe to re-run against production. They
are absent live and would run on every sync with the default hook strategy.
The adoption sync uses the `apply` strategy, which skips them. Once automated
sync is enabled they run on each sync, so the second go-ahead depends on that
confirmation. Ask `target-revenue` to pin an image digest instead of `0.1.4`.
Live check: `https://revenue.coulomb.social` health and the CNPG cluster
healthy. Workload restore: `kubectl apply -k k8s/railiance` from the previous
target-revenue commit.
2026-10-03: human gate routed to hub decision 89f1058f-33bc-47a8-8a49-0b44d426c67e (D2: ArgoCD owns production: go-ahead sequence for railiance-apps-root, issue-core, policy-nexus and activity-core automation) by CUST-WP-0075; resolving that decision raises C-40 here.
2026-10-11: **manual adoption proving complete**. Fresh read-only proof at 2026-10-11T19:51:12.812423+00:00
verified exact pin `a3a8a27a8cda32ae3c7b55353ee16a131c50a0a0`, manual apply strategy,
prune/force off and no pending operation. The last successful pinned apply finished
2026-09-28T14:23:27Z; 1142865 seconds of proving exceeds 24 hours.
All five non-hook objects are tracked and Synced; Deployment Available (generation observed),
ExternalSecret Ready and CNPG Ready (1/1). ArgoCD remains Synced/Healthy; normalized native
client-side diff is zero and `https://revenue.coulomb.social/healthz` returns 200. Both declared
Sync-hook Jobs are absent and excluded from the apply-strategy diff. No live mutation or
Secret read was performed. Receipt: `docs/evidence/2026-10-11-target-revenue-adoption-proof/receipt.json`;
sanitized observation alongside it. Reusable check: `scripts/workload_adoption_proof.py`;
35 new failure/normalization tests plus 13 existing OpenBao adoption tests pass.
This closes T05 adoption/proving only. Child automated sync stays off. The existing
target-revenue owner handoff for hook re-execution safety remains a prerequisite to future
automation; it is not silently resolved and no new residual is invented here. The observed
OrphanedResourceWarning is non-blocking; other Application condition types fail verification.
## Adopt external-secrets (highest risk)
```task
id: RPF-WP-0044-T06
status: wait
decision_id: 89f1058f-33bc-47a8-8a49-0b44d426c67e
priority: medium
depends_on: [RPF-WP-0044-T04, RPF-WP-0044-T05]
needs_human: true
blocking_reason: "Attended founder run of scripts/rpf_wp_0044_t06_attended.sh (KUBECTL_ROUTE=founder-sudo): fresh pre-checks, Application create, typed ADOPT, manual SSA sync prune/force off, post-checks. Runbook docs/runbooks/rpf-wp-0044-t06-eso-adoption.md."
state_hub_task_id: "4ca54868-c31f-5e3d-b7d3-a15b84c15f80"
```
2026-10-11: requalified after founder decision D2 (ArgoCD owns production) resolved — see the-custodian docs/decisions/2026-10-11-security-unblock-decisions.md.
**Authorized by D2; waits on predecessor automated proof, not another founder go.**
The draft already carries CRD prune/delete protection (verified preparation
2026-09-27; 39 objects, 20 protected CRDs, metadata-only diff). That historical
receipt is preparation, not current adoption proof. The deterministic
`scripts/eso_adoption_readiness.py` checks the draft and bound predecessor
receipts before a fresh native render/diff is considered. ESO serves every
`ExternalSecret` on the cluster (25 ClusterSecretStores). Before the merge,
add `crds.annotations: {argocd.argoproj.io/sync-options: "Prune=false,Delete=false"}`
to the values and re-run the diff; pruning a CRD would delete every
`ExternalSecret` and store. Adopt with `ServerSideApply=true` as declared.
Watch for drift on `caBundle` fields injected by the cert-controller; if
ArgoCD shows it, add `ignoreDifferences` before enabling self-heal. After
adoption, retire the stale `pending-upgrade` Helm release record (its
`sh.helm.release.v1.external-secrets.v*` Secrets) under a separate reviewed cleanup;
ArgoCD does not track them. Live check: all ClusterSecretStores keep their
current `Valid` state (two, `openbao-activity-core` and
`openbao-email-connect`, are already `ValidationFailed`). Workload restore: revert the Platform Application to the previously reviewed
chart 0.16.1 values in Git, then manually sync through ArgoCD; never prune/delete
CRDs or silently replace the Helm release record as part of adoption.
2026-10-03: human gate routed to hub decision 89f1058f-33bc-47a8-8a49-0b44d426c67e (D2: ArgoCD owns production: go-ahead sequence for railiance-apps-root, issue-core, policy-nexus and activity-core automation) by CUST-WP-0075; resolving that decision raises C-40 here.
2026-10-21 (preparation, agent, no live mutation): `scripts/eso_adoption_readiness.py` passes the source check (chart 0.16.1, 20/20 CRDs Prune=false,Delete=false, ServerSideApply, manual, no finalizer, no Force/Replace) but reports T04 and T05 missing. The 2026-10-20 issue-core verify-live receipt has no authenticated smoke statuses (it ran without `--smoke`, and `--smoke` creates and closes a Forgejo issue, so it is not read-only) and is outside the 300 s window. A fresh read-only target-revenue proof failed because `ssh railiance01` (tegwick) can no longer read the k3s kubeconfig after the CUST-WP-0073 cutover. Offline render of 0.16.1 with the draft values: 39 objects, no ClusterSecretStore or ExternalSecret rendered, webhooks declare no caBundle, the webhook-cert holder is metadata-only; no ignoreDifferences added yet. The fresh live client-side diff and the store baseline were not run (no agent cluster read path). Receipts: `docs/evidence/2026-10-21-eso-adoption-prep/`. Runbook with the exact founder sync command: `docs/runbooks/rpf-wp-0044-t06-eso-adoption.md`.
2026-10-21 (founder decisions, agent preparation): the founder will run ONE attended script from their own terminal through their admin path, and allowed the issue-core `--smoke` controlled write in that pre-check. Agent RBAC is not widened. New `scripts/kube_route.py` adds `KUBECTL_ROUTE` (`agent` default, `founder-sudo` = `ssh railiance01 sudo -n …`; no `ssh -t`, since `sudo -n` keeps pipes clean). It is used by the transport, adoption-proof, workload-proof, activity-readiness, pin-drift and service-observation scripts and by rapp-issue-core `tools/verify_live.py`. `scripts/rpf_wp_0044_t06_attended.sh` runs the fresh proofs, readiness gate, render, client-side diff, structured classification and store baseline, stops on a no-go, then creates the Application, requires typed `ADOPT`, syncs (prune/force off, SSA) and runs the post-checks and refresh sample; on failure it prints the rollback and never prunes. Evaluator `scripts/eso_attended_checks.py`; credential-kind objects are split out before any live read. Offline validation: shellcheck clean, `--dry-run` prints commands only, 118 tests pass in the affected files (new: test_kube_route, test_eso_attended_checks; rapp-issue-core: test_kube_route). Not verified live: `sudo -n` pipe behaviour under the host `use_pty` default; the `--precheck-only` run will show it.
## Turn the root back to the production lane
```task
id: RPF-WP-0044-T07
status: wait
decision_id: 89f1058f-33bc-47a8-8a49-0b44d426c67e
depends_on: [RPF-WP-0044-T03, RPF-WP-0044-T04, RPF-WP-0044-T05, RPF-WP-0044-T06]
blocking_reason: "Restore prune/selfHeal once T03-T06 have automated proof; no further founder go-ahead (D2)."
priority: medium
state_hub_task_id: "844ff115-def2-55ca-ab01-413ba7e6857d"
```
2026-10-11: requalified after founder decision D2 (ArgoCD owns production) resolved — see the-custodian docs/decisions/2026-10-11-security-unblock-decisions.md.
**Waits on the founder's go-ahead.** After T03–T06 are proven, restore
`automated: {prune: true, selfHeal: true}` on `railiance-apps-root`, as the
change-gate decision's production row expects. From then on a merge to the
railiance01 path is a live production change, and `RPF-WP-0043-T04`
(policy-nexus) can use it.
2026-10-03: human gate routed to hub decision 89f1058f-33bc-47a8-8a49-0b44d426c67e (D2: ArgoCD owns production: go-ahead sequence for railiance-apps-root, issue-core, policy-nexus and activity-core automation) by CUST-WP-0075; resolving that decision raises C-40 here.
## Plan phase C: retire coulombcore's ArgoCD
```task
id: RPF-WP-0044-T08
status: done
priority: low
state_hub_task_id: "55d1382f-5862-5321-a1c9-96767764ba43"
```
2026-10-11: done by host-side read-only check (docs/evidence/2026-10-11-coulombcore-argocd-frozen.md). k3s certificates expired 2026-08-31, so the old ArgoCD cannot reconcile or prune. Founder: coulombcore is retired and the machine reused; retirement moves to NK-WP-0022-T08. Do not rotate certs or restart k3s there.
2026-10-11: requalified after founder decision D2 (ArgoCD owns production) resolved — see the-custodian docs/decisions/2026-10-11-security-unblock-decisions.md.
Planning only. Needs a read-only check of coulombcore's ArgoCD, outside this
session's scope. Under Option A, retiring it also removes `argocd/applications/`.
Also hand back to the cluster layer: the phase A install is not declared in
any repository and its pods have no resource requests (BestEffort).
2026-10-03: human gate routed to hub decision 89f1058f-33bc-47a8-8a49-0b44d426c67e (D2: ArgoCD owns production: go-ahead sequence for railiance-apps-root, issue-core, policy-nexus and activity-core automation) by CUST-WP-0075; resolving that decision raises C-40 here.
## Loose-end review — 2026-09-27
T03/T05 passed their elapsed healthy observation gate: current ArgoCD status is Synced/Healthy with last healthy transitions on September 21; openbao is Valid and issue-core-runtime is SecretSynced. They now wait on the remaining automation go-aheads and, for target-revenue, hook-owner acceptance. T06 local preparation is complete: the inactive ESO draft protects all 20 CRDs from prune/delete. The pinned 0.16.1 chart renders 39 objects; server-side diff is exactly the 20 protective metadata annotations, with no spec changes. All 25 ClusterSecretStores are Valid. T04 still needs the repository credential/production-owner contract; T07 depends on the per-app gates. T08 has a concrete retirement procedure at `docs/argocd-coulombcore-retirement.md`, but its required live inventory is blocked: SSH works and Kubernetes rejects both default and explicit local k3s kubeconfigs. No controller or workload was changed. Evidence: `docs/evidence/2026-09-27-argocd-loose-end-status.json` and `docs/evidence/2026-09-27-eso-adoption-preparation.json`.