489cc42af1
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 63291@bnt-lap001 Assistant-Session: 8bd77868-ca68-4f49-bb1e-d539ecc0d703
328 lines
18 KiB
Markdown
328 lines
18 KiB
Markdown
---
|
||
id: RPF-WP-0044
|
||
type: workplan
|
||
title: "ArgoCD phase B: adopt the four existing Applications on railiance01"
|
||
domain: financials
|
||
repo: railiance-platform
|
||
status: active
|
||
flavor: planning
|
||
owner: railiance-platform
|
||
topic_slug: railiance
|
||
created: "2026-09-21"
|
||
updated: "2026-09-21"
|
||
related: [RPF-WP-0043, RPF-WP-0022]
|
||
state_hub_workstream_id: "98140775-3b9a-5cf9-9af6-722502d487dc"
|
||
---
|
||
|
||
The founder, Bernd Worsch, decided ArchitectureBlueprint §5.6 on 2026-09-21,
|
||
exercising `GOVERN @ estate`: option 1, **adopt properly**. Phase A installed
|
||
Argo CD Core v3.5.3 on railiance01 (record:
|
||
`the-custodian/docs/kubernetes-change-gate-decision.md`). This plan is phase B:
|
||
bring the four Applications in `argocd/applications/` under that ArgoCD, **one
|
||
at a time, each with its own founder go-ahead**. Terms follow SecurityCanon
|
||
Mode of Authority v0.2.0 (draft).
|
||
|
||
**This plan changes nothing live.** Every task that would change railiance01
|
||
is `wait` on the founder. Each live step is `ADMINISTER @ realm:kubernetes/railiance01`
|
||
(bootstrap) or `CONSTRUCT @ railiance-platform` with the merge as the gate,
|
||
both `activation=APPROVED`.
|
||
|
||
Related, kept separate: `RPF-WP-0043` onboards `rapp-policy-nexus`, which is
|
||
not one of these four. It needs T02 of this plan (a working root app on
|
||
railiance01) before its own T04. Phase C, retiring coulombcore's ArgoCD, is
|
||
not planned here (T08).
|
||
|
||
## Read-only evidence, 2026-09-21
|
||
|
||
Full record: `docs/evidence/argocd-phase-b-adoption-diff-2026-09-21.json`.
|
||
Rendered locally (`helm template`, `kubectl kustomize`), then `kubectl diff`
|
||
client-side and `--server-side` dry run over `ssh railiance01`. Nothing was
|
||
applied.
|
||
|
||
| App | Live spec diff | Would prune | Repo credential | Blocker |
|
||
| --- | --- | --- | --- | --- |
|
||
| openbao-secretstore | none (1 ClusterSecretStore) | nothing | none, public | none |
|
||
| issue-core | none **after repointing** | nothing | **new**: `rapp-issue-core` is private | declared path `issue-core.git k8s/railiance` no longer exists |
|
||
| target-revenue | none on Deployment/Service/Ingress/ExternalSecret/Cluster | nothing | none, public | CNPG `Cluster` not in `railiance-tenants` whitelist; two Sync-hook Jobs |
|
||
| external-secrets | none (39 objects, 20 CRDs) | nothing now | none, public Helm repo | cluster-wide blast radius; CRDs must never be pruned |
|
||
|
||
Nothing is pruned at adoption because no live object carries an ArgoCD
|
||
tracking annotation; ArgoCD only prunes what it tracks. The one change every
|
||
adoption makes is that tracking annotation, on metadata only, so no rollout.
|
||
|
||
Sync settings found: `railiance-apps-root` and all four children declare
|
||
`automated: {prune: true, selfHeal: true}`; none has a finalizer. Applying
|
||
`argocd/bootstrap/` as committed would therefore adopt all four at once. T01
|
||
removes that before anything is applied.
|
||
|
||
Also found: the `external-secrets` Helm release is stuck at revision 7,
|
||
`pending-upgrade`, since 2026-07-07 (values identical to the Application);
|
||
`issue-core` is now server-side applied by `rapp-issue-core`;
|
||
`target-revenue` runs a mutable image tag (`0.1.4`). ArgoCD on railiance01
|
||
has 0 AppProjects, 0 Applications, 0 repository Secrets, and there is no
|
||
`argocd` CLI on the node, so the commands below use `kubectl` against the
|
||
Application resource. Not verified: repo-server egress to Forgejo and
|
||
`charts.external-secrets.io`; T02 proves it.
|
||
|
||
## Neutralise automated sync and fix the declared state
|
||
|
||
```task
|
||
id: RPF-WP-0044-T01
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "2d7230f5-50c8-5eee-a050-1501ce5af62a"
|
||
```
|
||
|
||
**Done 2026-09-21, commit `c3ebd6d`.** The founder chose **Option A**
|
||
(`GOVERN @ estate`): a railiance01-only source path. Repository-only; nothing
|
||
applied. `argocd/applications/` and `argocd/bootstrap/` are untouched, because
|
||
coulombcore's ArgoCD still reads `argocd/applications/` on `main` and its
|
||
bootstrap may be re-applied from `argocd/bootstrap/`. coulombcore could not be
|
||
read in this session, so the railiance01 bootstrap has its own path.
|
||
|
||
Declared (all under `argocd/railiance01/`):
|
||
|
||
1. `bootstrap/`: the three AppProjects and `railiance-apps-root` with **no
|
||
`automated` block**, path `argocd/railiance01/applications`,
|
||
`targetRevision: main` (each sync pins its revision in the operation).
|
||
2. `bootstrap/01-railiance-tenants-project.yaml`: `postgresql.cnpg.io/Cluster`
|
||
added to `namespaceResourceWhitelist`. The other two AppProjects are copies
|
||
of `argocd/bootstrap/`.
|
||
3. `applications/` holds only `README.md`. A placeholder **is** needed: git does
|
||
not keep an empty directory and a missing path is a `ComparisonError`.
|
||
ArgoCD's directory source reads only `*.yaml`/`*.yml`/`*.json`, so it renders
|
||
zero children (confirmed live by T02).
|
||
4. `drafts/`: the four children, no `automated`, no finalizer, pinned:
|
||
openbao-secretstore `d2dbc19` (railiance-platform), issue-core `a78f38c`
|
||
(`rapp-issue-core.git`, path `manifests`), target-revenue `f1109d5`,
|
||
external-secrets chart `0.16.1`. No root syncs `drafts/`.
|
||
5. `argocd/repositories/rapp-issue-core.repository.sops.yaml.template` (no
|
||
value); `issue-core.repository.sops.yaml.template` marked obsolete.
|
||
|
||
Offline proof: `kubectl kustomize argocd/railiance01/bootstrap` renders exactly
|
||
3 AppProjects and 1 Application with no `automated` field;
|
||
`ssh railiance01 'kubectl apply --dry-run=server -f -'` of that render
|
||
returned rc=0 (all four "created (server dry run)").
|
||
|
||
Hazard kept as is: `make argocd-bootstrap-deploy` still defaults to
|
||
`argocd/bootstrap` (automated root). Do not run it on railiance01.
|
||
|
||
## Apply the AppProjects and the root app, automated sync off (live)
|
||
|
||
```task
|
||
id: RPF-WP-0044-T02
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "7d97ee9c-b2fb-520f-8468-3ecf7eea6c01"
|
||
```
|
||
|
||
**Done 2026-09-21** (founder go-ahead; custodian session). 3 AppProjects + `railiance-apps-root` applied from the clean export of `c3ebd6d`; root Synced/Healthy, op Succeeded at `c3ebd6d`, zero children, 0 ArgoCD-managed namespaces. Repo-server reaches Forgejo. One `OrphanedResourceWarning` (project-level, informational).
|
||
|
||
Applies the render of
|
||
`argocd/railiance01/bootstrap` at commit `c3ebd6dddc830373c75144da178f829b47c3e5de`, from a clean export (not
|
||
the working tree), then syncs the root by hand at that commit.
|
||
|
||
```bash
|
||
cd ~/railiance-platform && git fetch origin && git merge-base --is-ancestor c3ebd6dddc830373c75144da178f829b47c3e5de origin/main && echo pinned-commit-on-main
|
||
R=$(mktemp -d) && git -C ~/railiance-platform archive c3ebd6dddc830373c75144da178f829b47c3e5de argocd/railiance01/bootstrap | tar -x -C "$R"
|
||
kubectl kustomize "$R/argocd/railiance01/bootstrap" > "$R/bootstrap.yaml" && grep -c '^kind: AppProject' "$R/bootstrap.yaml" && grep -c automated "$R/bootstrap.yaml" # expect 3 and 0
|
||
ssh railiance01 'kubectl apply --dry-run=server -f -' < "$R/bootstrap.yaml"
|
||
ssh railiance01 'kubectl apply -f -' < "$R/bootstrap.yaml"
|
||
ssh railiance01 'kubectl -n argocd patch application railiance-apps-root --type merge -p "{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"c3ebd6dddc830373c75144da178f829b47c3e5de\",\"prune\":false}}}"'
|
||
ssh railiance01 'kubectl -n argocd get appprojects,applications'
|
||
ssh railiance01 'kubectl -n argocd get application railiance-apps-root -o jsonpath="{.status.sync.status} {.status.operationState.phase} {.status.operationState.syncResult.revision} resources={.status.resources}{\"\n\"}"'
|
||
```
|
||
|
||
Expect three AppProjects and `railiance-apps-root` `Synced`, operation
|
||
`Succeeded` at `c3ebd6d`, and no resources (zero children). A
|
||
`ComparisonError` means the repo-server cannot reach Forgejo: stop and fix
|
||
egress before any app task. No repository Secret is needed;
|
||
`coulomb/railiance-platform` is public.
|
||
|
||
Rollback: `kubectl -n argocd delete application railiance-apps-root` (no
|
||
finalizer, no children), then delete the three AppProjects. Nothing else is
|
||
touched.
|
||
|
||
## Per-app procedure (T03–T06)
|
||
|
||
Every adoption task follows the same steps. `<app>` and `<sha>` vary.
|
||
|
||
1. **Diff.** Re-run the evidence render and `kubectl diff --server-side` at
|
||
the commit or chart version being pinned. Any spec change, any missing
|
||
whitelist kind, or a different image: stop.
|
||
2. **Merge** the child Application (no `automated`) from
|
||
`argocd/railiance01/drafts/` into `argocd/railiance01/applications/`, then sync the root by hand so it creates the child object only:
|
||
```bash
|
||
ssh railiance01 'kubectl -n argocd patch application railiance-apps-root --type merge -p "{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"<sha>\",\"prune\":false}}}"'
|
||
```
|
||
3. **ArgoCD's own diff:** read
|
||
`kubectl -n argocd get application <app> -o jsonpath='{.status.sync.status} {.status.health.status}'`
|
||
and `.status.resources`. Only metadata differences are acceptable.
|
||
4. **Manual sync, prune off, hooks skipped** (`apply` strategy):
|
||
```bash
|
||
ssh railiance01 'kubectl -n argocd patch application <app> --type merge -p "{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"<sha>\",\"prune\":false,\"syncStrategy\":{\"apply\":{}}}}}"'
|
||
```
|
||
Then verify Synced/Healthy and run the owner's live check.
|
||
5. **Prove it:** Synced and Healthy for at least 24 hours (7 days for
|
||
external-secrets). Only then, with a **second** go-ahead, a merge adds
|
||
`automated: {selfHeal: true}`; `prune: true` is a third step once the
|
||
app's tracked set is confirmed complete.
|
||
6. Record `docs/evidence/<date>-<app>-argocd-adoption.json`.
|
||
|
||
Rollback:
|
||
|
||
- **R1, before automated sync:** revert the merge, then
|
||
`kubectl -n argocd delete application <app>`. No finalizer, so nothing
|
||
cascades; the workload keeps running. Tracking annotations stay and are
|
||
harmless.
|
||
- **R2, after automated sync:** revert the automated-sync commit first and
|
||
sync the root by hand, then R1. Never hand-patch a child while the root
|
||
self-heals; that is `BREAK_GLASS`, recorded and reconciled into git.
|
||
- **Workload restore:** only once ArgoCD no longer manages the app, with the
|
||
previous deploy method named in the task.
|
||
|
||
## Adopt openbao-secretstore (lowest risk)
|
||
|
||
```task
|
||
id: RPF-WP-0044-T03
|
||
status: progress
|
||
priority: high
|
||
state_hub_task_id: "6d5fc801-e361-579b-bcd8-6fe719a82e94"
|
||
```
|
||
|
||
**Adopted 2026-09-21** (founder go-ahead; custodian session): diff at `d2dbc19` rc=0; child merged in `182e788`; root synced at `182e788`; child manual sync (apply strategy, prune off) Succeeded at `d2dbc19`, Synced/Healthy, automated off. `ClusterSecretStore/openbao` Valid with tracking annotation; `issue-core-runtime` SecretSynced. Status `progress` = in the 24h proving period; `selfHeal` needs a second go-ahead. Evidence: `docs/evidence/2026-09-21-openbao-secretstore-argocd-adoption.json`.
|
||
|
||
One object,
|
||
`ClusterSecretStore/openbao`, zero diff, public source. It is the store
|
||
issue-core's `ExternalSecret` reads, so adopt it before issue-core. The 24
|
||
other ClusterSecretStores on railiance01 are not in the kustomization and stay
|
||
untracked (declared gap, RPF-WP-0043-T05). Live check:
|
||
`kubectl get clustersecretstore openbao` stays `Valid`, and
|
||
`externalsecret/issue-core-runtime` stays `SecretSynced`. Workload restore:
|
||
`kubectl apply -f argocd/platform-addons/openbao-secretstore/openbao.clustersecretstore.yaml`.
|
||
|
||
Exact commands (the root sync pins the merge commit `$SHA`; the child is
|
||
pinned to `d2dbc19`, the last commit touching its source path):
|
||
|
||
```bash
|
||
# 1. Diff at the pinned source commit (expect no output, rc=0)
|
||
cd ~/railiance-platform && git fetch origin && git checkout main && git pull --ff-only
|
||
R=$(mktemp -d) && git archive d2dbc19c254247652c49fda8721c80d53bca206a argocd/platform-addons/openbao-secretstore | tar -x -C "$R"
|
||
ssh railiance01 'kubectl diff --server-side -f -' < <(kubectl kustomize "$R/argocd/platform-addons/openbao-secretstore"); echo rc=$?
|
||
# 2. Merge the child (drop the 3-line DRAFT header), push, sync the root at that commit
|
||
sed '1,3d' argocd/railiance01/drafts/openbao-secretstore.application.yaml > argocd/railiance01/applications/openbao-secretstore.application.yaml
|
||
git rm -q argocd/railiance01/drafts/openbao-secretstore.application.yaml
|
||
git add argocd/railiance01/applications/openbao-secretstore.application.yaml
|
||
git commit -m "Adopt openbao-secretstore on railiance01 (RPF-WP-0044-T03)" && git push origin main
|
||
SHA=$(git rev-parse HEAD) && echo "$SHA"
|
||
ssh railiance01 "kubectl -n argocd patch application railiance-apps-root --type merge -p '{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"$SHA\",\"prune\":false}}}'"
|
||
# 3. ArgoCD's own diff (expect OutOfSync or Synced, one resource ClusterSecretStore/openbao, metadata-only)
|
||
ssh railiance01 'kubectl -n argocd get application openbao-secretstore -o jsonpath="{.status.sync.status} {.status.health.status} {.status.resources}{\"\n\"}"'
|
||
# 4. Manual sync, prune off, apply strategy (hooks skipped)
|
||
ssh railiance01 'kubectl -n argocd patch application openbao-secretstore --type merge -p "{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"d2dbc19c254247652c49fda8721c80d53bca206a\",\"prune\":false,\"syncStrategy\":{\"apply\":{}}}}}"'
|
||
ssh railiance01 'kubectl -n argocd get application openbao-secretstore -o jsonpath="{.status.sync.status} {.status.health.status} {.status.operationState.phase}{\"\n\"}"'
|
||
ssh railiance01 'kubectl get clustersecretstore openbao; kubectl -n issue-core get externalsecret issue-core-runtime'
|
||
```
|
||
|
||
Rollback (R1): `git revert` the merge commit and push, then
|
||
`ssh railiance01 'kubectl -n argocd delete application openbao-secretstore'`
|
||
(no finalizer; the ClusterSecretStore keeps running).
|
||
|
||
## Adopt issue-core (from rapp-issue-core)
|
||
|
||
```task
|
||
id: RPF-WP-0044-T04
|
||
status: wait
|
||
priority: high
|
||
state_hub_task_id: "0e619658-0453-51ed-b9a0-bed28b5544fd"
|
||
```
|
||
|
||
**Waits on the founder's go-ahead**, and on two preconditions:
|
||
|
||
- `rapp-issue-core` agrees (hub message) that ArgoCD owns production and its
|
||
`make deploy` stops being a production path; it would fight self-heal.
|
||
Image releases become commits to `manifests/30-runtime.yaml`.
|
||
- A CCR for the ArgoCD repository credential, source at
|
||
`platform/operators/argocd/repositories/rapp-issue-core`, read-only deploy
|
||
token. Applied with `make argocd-repo-apply`; no value is handled in this
|
||
plan.
|
||
|
||
Zero diff against live for both the raw `manifests/` and the rendered output
|
||
(digest `sha256:a56c80cc…b92cf`). All kinds are in the `railiance-tenants`
|
||
whitelist. Live check: `rapp-issue-core make verify-live`. Workload restore:
|
||
`rapp-issue-core make rollback`.
|
||
|
||
## Adopt target-revenue
|
||
|
||
```task
|
||
id: RPF-WP-0044-T05
|
||
status: progress
|
||
priority: medium
|
||
state_hub_task_id: "d418068a-6fb0-5416-aac5-d23c93924d9c"
|
||
```
|
||
|
||
**Adopted 2026-09-21** (founder go-ahead; custodian session). The *server-side* diff at `f1109d5` conflicted on `resources.limits.cpu` (live `1` owned by `kubectl-client-side-apply` vs declared `1000m`: the same quantity); the *client-side* diff, which is what the apply strategy uses, was rc=0 for all five non-hook objects. Child merged in `6eeb826`; manual sync with the `apply` strategy, prune off: Succeeded, Synced/Healthy, **0 Jobs created**, same pod (36d, no restart), CNPG healthy 1/1, ExternalSecret SecretSynced, `https://revenue.coulomb.social/healthz` 200 (the live check path is `/healthz`, not `/health`). Status `progress` = proving period. Automated sync still needs target-revenue's confirmation on the two hook Jobs (asked 2026-09-21). Evidence: `docs/evidence/2026-09-21-target-revenue-argocd-adoption.json`.
|
||
|
||
Original preconditions: T01 item 2 (CNPG
|
||
`Cluster` whitelisted) applied to the AppProject by hand, and `target-revenue`
|
||
confirms the two Sync-hook Jobs (`target-revenue-migrate`,
|
||
`target-revenue-bootstrap-binky`) are safe to re-run against production. They
|
||
are absent live and would run on every sync with the default hook strategy.
|
||
The adoption sync uses the `apply` strategy, which skips them. Once automated
|
||
sync is enabled they run on each sync, so the second go-ahead depends on that
|
||
confirmation. Ask `target-revenue` to pin an image digest instead of `0.1.4`.
|
||
Live check: `https://revenue.coulomb.social` health and the CNPG cluster
|
||
healthy. Workload restore: `kubectl apply -k k8s/railiance` from the previous
|
||
target-revenue commit.
|
||
|
||
## Adopt external-secrets (highest risk)
|
||
|
||
```task
|
||
id: RPF-WP-0044-T06
|
||
status: wait
|
||
priority: medium
|
||
state_hub_task_id: "4ca54868-c31f-5e3d-b7d3-a15b84c15f80"
|
||
```
|
||
|
||
**Waits on the founder's go-ahead.** Zero diff, but ESO serves every
|
||
`ExternalSecret` on the cluster (25 ClusterSecretStores). Before the merge,
|
||
add `crds.annotations: {argocd.argoproj.io/sync-options: "Prune=false,Delete=false"}`
|
||
to the values and re-run the diff; pruning a CRD would delete every
|
||
`ExternalSecret` and store. Adopt with `ServerSideApply=true` as declared.
|
||
Watch for drift on `caBundle` fields injected by the cert-controller; if
|
||
ArgoCD shows it, add `ignoreDifferences` before enabling self-heal. After
|
||
adoption, retire the stale `pending-upgrade` Helm release record (its
|
||
`sh.helm.release.v1.external-secrets.v*` Secrets) under a separate go-ahead;
|
||
ArgoCD does not track them. Live check: all ClusterSecretStores keep their
|
||
current `Valid` state (two, `openbao-activity-core` and
|
||
`openbao-email-connect`, are already `ValidationFailed`). Workload restore:
|
||
`helm template` 0.16.1 with the same values, `kubectl apply --server-side`.
|
||
|
||
## Turn the root back to the production lane
|
||
|
||
```task
|
||
id: RPF-WP-0044-T07
|
||
status: wait
|
||
priority: medium
|
||
state_hub_task_id: "844ff115-def2-55ca-ab01-413ba7e6857d"
|
||
```
|
||
|
||
**Waits on the founder's go-ahead.** After T03–T06 are proven, restore
|
||
`automated: {prune: true, selfHeal: true}` on `railiance-apps-root`, as the
|
||
change-gate decision's production row expects. From then on a merge to the
|
||
railiance01 path is a live production change, and `RPF-WP-0043-T04`
|
||
(policy-nexus) can use it.
|
||
|
||
## Plan phase C: retire coulombcore's ArgoCD
|
||
|
||
```task
|
||
id: RPF-WP-0044-T08
|
||
status: todo
|
||
priority: low
|
||
state_hub_task_id: "55d1382f-5862-5321-a1c9-96767764ba43"
|
||
```
|
||
|
||
Planning only. Needs a read-only check of coulombcore's ArgoCD, outside this
|
||
session's scope. Under Option A, retiring it also removes `argocd/applications/`.
|
||
Also hand back to the cluster layer: the phase A install is not declared in
|
||
any repository and its pods have no resource requests (BestEffort).
|