diff --git a/ArchitectureBlueprint.md b/ArchitectureBlueprint.md index 0f0f40a..7ac2898 100644 --- a/ArchitectureBlueprint.md +++ b/ArchitectureBlueprint.md @@ -332,7 +332,27 @@ to `railiance-master` (`04c776c4`, amended `f88f938d`). - delete the local `~/railiance-bootstrap` directory, which is a mis-cloned copy of `railiance-cluster.git` -### 5.6 Does Railiance use ArgoCD? → **REOPENED 2026-08-12 on corrected evidence** +### 5.6 Does Railiance use ArgoCD? → **DECIDED 2026-09-21: option 1, adopt properly** + +**Decision.** Bernd Worsch, the founder, decided on 2026-09-21, exercising +`GOVERN @ estate`, for option 1 below: **adopt properly**. ArgoCD runs on +railiance01, and the live workloads are brought under GitOps one application +at a time. Record: `the-custodian/docs/kubernetes-change-gate-decision.md` +§ "ArgoCD installed on railiance01 (phase A)". + +- **Phase A, done 2026-09-21:** upstream Argo CD Core v3.5.3 (headless) + installed in namespace `argocd` on railiance01 by the custodian session, + `ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`. 0 + Applications, 0 AppProjects, nothing adopted. The install is not yet + declared in any repository. +- **Phase B, planned, not started:** adopt external-secrets, + openbao-secretstore, issue-core and target-revenue, each with its own + founder go-ahead. Plan and read-only diff evidence: `RPF-WP-0044`, + `docs/evidence/argocd-phase-b-adoption-diff-2026-09-21.json`. +- **Phase C, not planned yet:** retire coulombcore's ArgoCD. + +The evidence that reopened the question on 2026-08-12 follows unchanged. + Decided 2026-08-11 as "keep it", on the basis that S3 runs ArgoCD as its live deployment mechanism. **That basis was wrong.** Verified against both clusters diff --git a/docs/argocd-gitops.md b/docs/argocd-gitops.md index 406749a..3082e06 100644 --- a/docs/argocd-gitops.md +++ b/docs/argocd-gitops.md @@ -48,7 +48,9 @@ through git. What this means for platform review in this repository: - A merge to `argocd/applications/` is a live production change. Once the - commit is on `main`, `railiance-apps-root` syncs it with prune and self-heal. + commit is on `main`, `railiance-apps-root` syncs it with prune and self-heal + (today on coulombcore; on railiance01 only after RPF-WP-0044 applies the + bootstrap). Treat the merge as the `APPROVED` gate, not as a draft. - A new Application for a workload that is already running is an **adoption**. Follow the adoption steps in `workplans/RPF-WP-0043-policy-nexus-argocd-onboarding.md`: @@ -63,29 +65,23 @@ Application here. Until 2026-12-21 its changes keep `activation=APPROVED` by founder plan approval, and each one is recorded as production-tier. Onboarding is due by 2026-12-21 (RPF-WP-0043). -**Not verified: that ArgoCD syncs on railiance01.** This repository does not -show it, and some records point the other way: +**ArgoCD on railiance01: installed, not yet reconciling anything +(2026-09-21).** The custodian's read-only check found no ArgoCD on +railiance01 (namespace and CRDs NotFound; RPF-WP-0043-T01 evidence). The +founder then decided ArchitectureBlueprint §5.6 as option 1, adopt properly, +and Argo CD Core v3.5.3 was installed there (phase A). As of this record it +has 0 AppProjects, 0 Applications and 0 repository Secrets: the bootstrap in +`argocd/bootstrap/` is **not applied**, so `railiance-apps-root` does not yet +sync anything on railiance01. The records calling the controller "coulombcore +ArgoCD" were right about the past; coulombcore's ArgoCD still reads +`argocd/applications/` until phase C retires it. -- The only record of ArgoCD's health is the RPF-WP-0022 inbox evidence from - 2026-06: "installed and healthy on `railiance01`", with zero Applications. - That is from before the 2026-07-02 naming correction. Until then, "railiance01 - cluster" often meant coulombcore. -- The later records in this file and in - `argocd/platform-addons/openbao-secretstore/kustomization.yaml` name the - ArgoCD that runs these Applications "coulombcore ArgoCD". They say the - railiance01 stores are "not managed by coulombcore ArgoCD". -- Every Application uses `destination.server: https://kubernetes.default.svc`, - so it reconciles whichever cluster ArgoCD itself runs in. -- No evidence file in this repository records a sync of `railiance-apps-root` - on railiance01 (92.205.62.239). - -Confirming it takes an attended **read-only** check against railiance01, not -coulombcore: check the node IP, the `argocd` namespace, and the `railiance-apps-root` -sync and health status, for example `make argocd-status` with a kubeconfig whose -server is verified to be 92.205.62.239. Record the result as evidence. If ArgoCD -is not running there, the production row has no working path on railiance01 -yet. Installing ArgoCD belongs to the cluster layer and needs its own go-ahead -from the founder. This is RPF-WP-0043-T01. +**Do not apply the bootstrap as committed.** The root app and all four +children declare `automated` sync with prune and self-heal, so applying it +would adopt all four live workloads at once. Phase B (`RPF-WP-0044`) first +neutralises automated sync, then adopts one application at a time, each with +its own founder go-ahead. Read-only diff evidence: +`docs/evidence/argocd-phase-b-adoption-diff-2026-09-21.json`. ## Bootstrap Layout diff --git a/docs/evidence/argocd-phase-b-adoption-diff-2026-09-21.json b/docs/evidence/argocd-phase-b-adoption-diff-2026-09-21.json new file mode 100644 index 0000000..e1f6b9f --- /dev/null +++ b/docs/evidence/argocd-phase-b-adoption-diff-2026-09-21.json @@ -0,0 +1,87 @@ +{ + "record": "argocd-phase-b-adoption-diff", + "date": "2026-09-21", + "workplan": "RPF-WP-0044", + "mode": "read-only", + "evidence_boundary": "target-audited", + "performed_by": "railiance-platform agent session (Claude Code), founder-authorised session", + "cluster": { + "node": "239.62.205.92.host.secureserver.net", + "internal_ip": "92.205.62.239", + "k3s": "v1.35.1+k3s1", + "access": "ssh railiance01 'kubectl ...'; helm --kubeconfig /etc/rancher/k3s/k3s.yaml for get/history/list" + }, + "commands_used": [ + "kubectl get / kubectl diff (client-side and --server-side dry run; nothing persisted)", + "helm list -A -a, helm history, helm get values (user-supplied values only)", + "local: helm template, kubectl kustomize, rapp-issue-core tools/render.py into a scratch directory" + ], + "argocd_live": { + "version": "v3.5.3 core (headless)", + "workloads": "application-controller sts 1/1, repo-server 1/1, redis 1/1, applicationset-controller 1/1", + "applications": 0, + "appprojects": 0, + "repository_secrets": 0, + "tracking_method": "default (annotation) - argocd-cm sets no application.resourceTrackingMethod", + "argocd_cli_on_node": false + }, + "apps": { + "openbao-secretstore": { + "source": "railiance-platform.git main argocd/platform-addons/openbao-secretstore (kustomize)", + "rendered": ["ClusterSecretStore/openbao"], + "diff_server_side": "empty (rc=0)", + "diff_client_side": "empty (rc=0)", + "live_managers": ["railiance-platform", "external-secrets"], + "would_change_live": "no spec change; ArgoCD adds its tracking annotation only", + "would_prune": "nothing - no live object carries an ArgoCD tracking annotation; the other 24 ClusterSecretStores are not in the kustomization and stay untracked", + "repository_credential": "none needed - coulomb/railiance-platform is public (Forgejo API private=false, anonymous ls-remote works)", + "blockers": [] + }, + "issue-core": { + "source_declared": "issue-core.git main k8s/railiance", + "source_status": "PATH DOES NOT EXIST - removed by issue-core 51440da (2026-08-19, ISSUE-WP-0007); runtime moved to coulomb/rapp-issue-core", + "effective_source": "rapp-issue-core.git main manifests/ (a78f38c), image pinned in the file", + "rendered": ["Namespace", "ConfigMap", "ExternalSecret", "ServiceAccount", "Service", "Deployment", "NetworkPolicy x2"], + "diff_server_side": "empty for both the raw manifests/ directory and the tools/render.py output", + "live_image": "forgejo.coulomb.social/coulomb/issue-core@sha256:a56c80ccbd1690bc0bcea384165b1967704f0fdbaf1c506ce1082f56190b92cf", + "live_managers": ["rapp-issue-core (server-side apply)", "kubectl-rollout", "k3s"], + "would_change_live": "as declared: nothing (ComparisonError, path missing). Repointed to rapp-issue-core: no spec change, tracking annotation only", + "would_prune": "nothing", + "repository_credential": "NEW: coulomb/rapp-issue-core is private (anonymous API returns nothing). Needs a template rapp-issue-core.repository.sops.yaml.template and OpenBao path platform/operators/argocd/repositories/rapp-issue-core. The existing issue-core template is obsolete (issue-core is public and no longer the source).", + "blockers": ["Application must be repointed", "credential CCR", "rapp-issue-core make deploy must stop being a production path (would fight self-heal)"] + }, + "target-revenue": { + "source": "target-revenue.git main k8s/railiance (f1109d5, kustomize)", + "rendered": ["Deployment", "Service", "Ingress", "ExternalSecret", "postgresql.cnpg.io/Cluster", "Job/target-revenue-migrate (Sync hook, wave 1)", "Job/target-revenue-bootstrap-binky (Sync hook, wave 2)"], + "diff_server_side": "Deployment, Service, Ingress, ExternalSecret, Cluster: empty. Only the two Jobs differ: absent live (completed, TTL-deleted), would be created", + "diff_client_side": "same result", + "live_image": "forgejo.coulomb.social/coulomb/target-revenue:0.1.4 (tag, not digest)", + "live_managers": ["kubectl-client-side-apply", "k3s"], + "would_change_live": "yes: every sync with the default hook strategy runs the migrate Job against the production database and the binky bootstrap Job (commented idempotent)", + "would_prune": "nothing", + "project_whitelist": "BLOCKER: postgresql.cnpg.io/Cluster is not in railiance-tenants namespaceResourceWhitelist; the sync would be refused", + "repository_credential": "none needed - coulomb/target-revenue is public; no template exists", + "blockers": ["CNPG Cluster whitelist", "Sync-hook Jobs"] + }, + "external-secrets": { + "source": "https://charts.external-secrets.io external-secrets 0.16.1, values installCRDs=true, serviceAccount.create=true name=external-secrets", + "rendered_kinds": {"CustomResourceDefinition": 20, "ClusterRole": 5, "ClusterRoleBinding": 2, "Deployment": 3, "ServiceAccount": 3, "ValidatingWebhookConfiguration": 2, "Role": 1, "RoleBinding": 1, "Secret": 1, "Service": 1}, + "diff_server_side": "empty (rc=0), rendered with --include-crds --kube-version 1.35.1", + "live_helm_release": "external-secrets rev 7 status pending-upgrade since 2026-07-07 22:32; rev 6/7 user values identical to the Application values", + "live_managers": ["helm", "kubectl-rollout", "k3s"], + "would_change_live": "no spec change in the dry run; ArgoCD adds tracking annotations to 39 objects including 20 CRDs. Watch caBundle fields injected by cert-controller for drift", + "would_prune": "nothing now. Later, pruning a CRD would delete every ExternalSecret and SecretStore cluster-wide", + "repository_credential": "none needed - public Helm repository", + "blockers": ["cluster-wide blast radius", "CRDs need Prune=false before prune is ever enabled", "stale helm release pending-upgrade"] + } + }, + "sync_policy_found": { + "railiance-apps-root": "automated prune+selfHeal", + "children": "all four: automated prune+selfHeal, no finalizers", + "consequence": "applying the bootstrap as committed adopts all four at once" + }, + "not_verified": [ + "repo-server egress to forgejo.coulomb.social and charts.external-secrets.io from inside railiance01", + "state of coulombcore's ArgoCD, which still reads argocd/applications on main (outside this session's read-only railiance01 scope)" + ] +} diff --git a/workplans/RPF-WP-0044-argocd-phase-b-adopt-existing-applications.md b/workplans/RPF-WP-0044-argocd-phase-b-adopt-existing-applications.md new file mode 100644 index 0000000..341fc57 --- /dev/null +++ b/workplans/RPF-WP-0044-argocd-phase-b-adopt-existing-applications.md @@ -0,0 +1,271 @@ +--- +id: RPF-WP-0044 +type: workplan +title: "ArgoCD phase B: adopt the four existing Applications on railiance01" +domain: financials +repo: railiance-platform +status: ready +flavor: planning +owner: railiance-platform +topic_slug: railiance +created: "2026-09-21" +updated: "2026-09-21" +related: [RPF-WP-0043, RPF-WP-0022] +--- + +The founder, Bernd Worsch, decided ArchitectureBlueprint §5.6 on 2026-09-21, +exercising `GOVERN @ estate`: option 1, **adopt properly**. Phase A installed +Argo CD Core v3.5.3 on railiance01 (record: +`the-custodian/docs/kubernetes-change-gate-decision.md`). This plan is phase B: +bring the four Applications in `argocd/applications/` under that ArgoCD, **one +at a time, each with its own founder go-ahead**. Terms follow SecurityCanon +Mode of Authority v0.2.0 (draft). + +**This plan changes nothing live.** Every task that would change railiance01 +is `wait` on the founder. Each live step is `ADMINISTER @ realm:kubernetes/railiance01` +(bootstrap) or `CONSTRUCT @ railiance-platform` with the merge as the gate, +both `activation=APPROVED`. + +Related, kept separate: `RPF-WP-0043` onboards `rapp-policy-nexus`, which is +not one of these four. It needs T02 of this plan (a working root app on +railiance01) before its own T04. Phase C, retiring coulombcore's ArgoCD, is +not planned here (T08). + +## Read-only evidence, 2026-09-21 + +Full record: `docs/evidence/argocd-phase-b-adoption-diff-2026-09-21.json`. +Rendered locally (`helm template`, `kubectl kustomize`), then `kubectl diff` +client-side and `--server-side` dry run over `ssh railiance01`. Nothing was +applied. + +| App | Live spec diff | Would prune | Repo credential | Blocker | +| --- | --- | --- | --- | --- | +| openbao-secretstore | none (1 ClusterSecretStore) | nothing | none, public | none | +| issue-core | none **after repointing** | nothing | **new**: `rapp-issue-core` is private | declared path `issue-core.git k8s/railiance` no longer exists | +| target-revenue | none on Deployment/Service/Ingress/ExternalSecret/Cluster | nothing | none, public | CNPG `Cluster` not in `railiance-tenants` whitelist; two Sync-hook Jobs | +| external-secrets | none (39 objects, 20 CRDs) | nothing now | none, public Helm repo | cluster-wide blast radius; CRDs must never be pruned | + +Nothing is pruned at adoption because no live object carries an ArgoCD +tracking annotation; ArgoCD only prunes what it tracks. The one change every +adoption makes is that tracking annotation, on metadata only, so no rollout. + +Sync settings found: `railiance-apps-root` and all four children declare +`automated: {prune: true, selfHeal: true}`; none has a finalizer. Applying +`argocd/bootstrap/` as committed would therefore adopt all four at once. T01 +removes that before anything is applied. + +Also found: the `external-secrets` Helm release is stuck at revision 7, +`pending-upgrade`, since 2026-07-07 (values identical to the Application); +`issue-core` is now server-side applied by `rapp-issue-core`; +`target-revenue` runs a mutable image tag (`0.1.4`). ArgoCD on railiance01 +has 0 AppProjects, 0 Applications, 0 repository Secrets, and there is no +`argocd` CLI on the node, so the commands below use `kubectl` against the +Application resource. Not verified: repo-server egress to Forgejo and +`charts.external-secrets.io`; T02 proves it. + +## Neutralise automated sync and fix the declared state + +```task +id: RPF-WP-0044-T01 +status: wait +priority: high +``` + +**Waits on the founder's choice of layout.** Repository-only; applying +nothing. But coulombcore's ArgoCD still syncs `argocd/applications/` on +`main`, so an edit there is a live change on coulombcore. + +- **Option A (recommended):** a railiance01-only source path, + `argocd/railiance01/applications/`, starting empty, and the railiance01 root + pointing at it. `argocd/applications/` is left untouched for coulombcore + until phase C. No commit in this plan then reaches coulombcore. +- **Option B:** edit `argocd/applications/` in place. Every change also lands + on coulombcore's ArgoCD; needs its own go-ahead and a coulombcore read first. + +Declared changes (Option A): + +1. Root app for railiance01 with **no `automated` block** and path + `argocd/railiance01/applications`. +2. `railiance-tenants`: add `postgresql.cnpg.io/Cluster` to + `namespaceResourceWhitelist` (target-revenue). +3. Child Application drafts kept in this workplan until their task; each + without `automated`, without finalizers, with `targetRevision` pinned to a + commit or chart version. +4. `issue-core` draft repointed to `rapp-issue-core.git`, path `manifests`. +5. Add `argocd/repositories/rapp-issue-core.repository.sops.yaml.template` + (no value). Mark `issue-core.repository.sops.yaml.template` obsolete. + +AppProjects and the root are applied by hand (T02), not synced by the root, +so these edits change nothing live until T02. + +## Apply the AppProjects and the root app, automated sync off (live) + +```task +id: RPF-WP-0044-T02 +status: wait +priority: high +``` + +**Waits on the founder's go-ahead.** Depends on T01. + +```bash +ssh railiance01 'kubectl apply --dry-run=server -f -' < <(kubectl kustomize argocd/bootstrap) +ssh railiance01 'kubectl apply -f -' < <(kubectl kustomize argocd/bootstrap) +ssh railiance01 'kubectl -n argocd get appprojects,applications' +``` + +Expect three AppProjects and `railiance-apps-root` Synced with zero children +(the path is empty). A `ComparisonError` means the repo-server cannot reach +Forgejo: stop and fix egress before any app task. No repository Secret is +needed; `coulomb/railiance-platform` is public. + +Rollback: `kubectl -n argocd delete application railiance-apps-root` (no +finalizer, no children), then delete the three AppProjects. Nothing else is +touched. + +## Per-app procedure (T03–T06) + +Every adoption task follows the same steps. `` and `` vary. + +1. **Diff.** Re-run the evidence render and `kubectl diff --server-side` at + the commit or chart version being pinned. Any spec change, any missing + whitelist kind, or a different image: stop. +2. **Merge** the child Application (no `automated`) into the railiance01 + path, then sync the root by hand so it creates the child object only: + ```bash + ssh railiance01 'kubectl -n argocd patch application railiance-apps-root --type merge -p "{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"\",\"prune\":false}}}"' + ``` +3. **ArgoCD's own diff:** read + `kubectl -n argocd get application -o jsonpath='{.status.sync.status} {.status.health.status}'` + and `.status.resources`. Only metadata differences are acceptable. +4. **Manual sync, prune off, hooks skipped** (`apply` strategy): + ```bash + ssh railiance01 'kubectl -n argocd patch application --type merge -p "{\"operation\":{\"initiatedBy\":{\"username\":\"founder-approved\"},\"sync\":{\"revision\":\"\",\"prune\":false,\"syncStrategy\":{\"apply\":{}}}}}"' + ``` + Then verify Synced/Healthy and run the owner's live check. +5. **Prove it:** Synced and Healthy for at least 24 hours (7 days for + external-secrets). Only then, with a **second** go-ahead, a merge adds + `automated: {selfHeal: true}`; `prune: true` is a third step once the + app's tracked set is confirmed complete. +6. Record `docs/evidence/--argocd-adoption.json`. + +Rollback: + +- **R1, before automated sync:** revert the merge, then + `kubectl -n argocd delete application `. No finalizer, so nothing + cascades; the workload keeps running. Tracking annotations stay and are + harmless. +- **R2, after automated sync:** revert the automated-sync commit first and + sync the root by hand, then R1. Never hand-patch a child while the root + self-heals; that is `BREAK_GLASS`, recorded and reconciled into git. +- **Workload restore:** only once ArgoCD no longer manages the app, with the + previous deploy method named in the task. + +## Adopt openbao-secretstore (lowest risk) + +```task +id: RPF-WP-0044-T03 +status: wait +priority: high +``` + +**Waits on the founder's go-ahead.** Depends on T02. One object, +`ClusterSecretStore/openbao`, zero diff, public source. It is the store +issue-core's `ExternalSecret` reads, so adopt it before issue-core. The 24 +other ClusterSecretStores on railiance01 are not in the kustomization and stay +untracked (declared gap, RPF-WP-0043-T05). Live check: +`kubectl get clustersecretstore openbao` stays `Valid`, and +`externalsecret/issue-core-runtime` stays `SecretSynced`. Workload restore: +`kubectl apply -f argocd/platform-addons/openbao-secretstore/openbao.clustersecretstore.yaml`. + +## Adopt issue-core (from rapp-issue-core) + +```task +id: RPF-WP-0044-T04 +status: wait +priority: high +``` + +**Waits on the founder's go-ahead**, and on two preconditions: + +- `rapp-issue-core` agrees (hub message) that ArgoCD owns production and its + `make deploy` stops being a production path; it would fight self-heal. + Image releases become commits to `manifests/30-runtime.yaml`. +- A CCR for the ArgoCD repository credential, source at + `platform/operators/argocd/repositories/rapp-issue-core`, read-only deploy + token. Applied with `make argocd-repo-apply`; no value is handled in this + plan. + +Zero diff against live for both the raw `manifests/` and the rendered output +(digest `sha256:a56c80cc…b92cf`). All kinds are in the `railiance-tenants` +whitelist. Live check: `rapp-issue-core make verify-live`. Workload restore: +`rapp-issue-core make rollback`. + +## Adopt target-revenue + +```task +id: RPF-WP-0044-T05 +status: wait +priority: medium +``` + +**Waits on the founder's go-ahead.** Preconditions: T01 item 2 (CNPG +`Cluster` whitelisted) applied to the AppProject by hand, and `target-revenue` +confirms the two Sync-hook Jobs (`target-revenue-migrate`, +`target-revenue-bootstrap-binky`) are safe to re-run against production. They +are absent live and would run on every sync with the default hook strategy. +The adoption sync uses the `apply` strategy, which skips them. Once automated +sync is enabled they run on each sync, so the second go-ahead depends on that +confirmation. Ask `target-revenue` to pin an image digest instead of `0.1.4`. +Live check: `https://revenue.coulomb.social` health and the CNPG cluster +healthy. Workload restore: `kubectl apply -k k8s/railiance` from the previous +target-revenue commit. + +## Adopt external-secrets (highest risk) + +```task +id: RPF-WP-0044-T06 +status: wait +priority: medium +``` + +**Waits on the founder's go-ahead.** Zero diff, but ESO serves every +`ExternalSecret` on the cluster (25 ClusterSecretStores). Before the merge, +add `crds.annotations: {argocd.argoproj.io/sync-options: "Prune=false,Delete=false"}` +to the values and re-run the diff; pruning a CRD would delete every +`ExternalSecret` and store. Adopt with `ServerSideApply=true` as declared. +Watch for drift on `caBundle` fields injected by the cert-controller; if +ArgoCD shows it, add `ignoreDifferences` before enabling self-heal. After +adoption, retire the stale `pending-upgrade` Helm release record (its +`sh.helm.release.v1.external-secrets.v*` Secrets) under a separate go-ahead; +ArgoCD does not track them. Live check: all ClusterSecretStores keep their +current `Valid` state (two, `openbao-activity-core` and +`openbao-email-connect`, are already `ValidationFailed`). Workload restore: +`helm template` 0.16.1 with the same values, `kubectl apply --server-side`. + +## Turn the root back to the production lane + +```task +id: RPF-WP-0044-T07 +status: wait +priority: medium +``` + +**Waits on the founder's go-ahead.** After T03–T06 are proven, restore +`automated: {prune: true, selfHeal: true}` on `railiance-apps-root`, as the +change-gate decision's production row expects. From then on a merge to the +railiance01 path is a live production change, and `RPF-WP-0043-T04` +(policy-nexus) can use it. + +## Plan phase C: retire coulombcore's ArgoCD + +```task +id: RPF-WP-0044-T08 +status: todo +priority: low +``` + +Planning only. Needs a read-only check of coulombcore's ArgoCD, outside this +session's scope. Under Option A, retiring it also removes `argocd/applications/`. +Also hand back to the cluster layer: the phase A install is not declared in +any repository and its pods have no resource requests (BestEffort).