Close verified incident task and finish local workplan loose ends
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a0e3c3-621b-7350-9f77-50a8d3ee7657
This commit is contained in:
codex
parent
5781d34b3b
commit
debf981097
22 changed files with 1210 additions and 62 deletions
62
docs/argocd-coulombcore-retirement.md
Normal file
62
docs/argocd-coulombcore-retirement.md
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
# Coulombcore ArgoCD retirement preparation
|
||||
|
||||
Existing owner task: RPF-WP-0044-T08. Prepared September 27, 2026; blocked
|
||||
pending a readable live inventory. No controller, application or workload was
|
||||
changed. This document is the phase C preparation within the existing workplan.
|
||||
|
||||
The host SSH lane works. Kubernetes returned Unauthorized for both the normal
|
||||
kubectl context and `sudo -n k3s kubectl --kubeconfig
|
||||
/etc/rancher/k3s/k3s.yaml get nodes`. The infrastructure/cluster owner must
|
||||
restore accepted read access; do not rotate cluster credentials or restart k3s
|
||||
as an incidental inventory fix. Current workload ownership cannot be inferred
|
||||
from the old manifests.
|
||||
|
||||
## Inventory and decision inputs
|
||||
|
||||
Capture node/cluster identity, ArgoCD deployments/statefulsets and installed
|
||||
version, Applications and AppProjects, each Application's destination, pinned
|
||||
revision, tracked resource set, hooks, automated sync and finalizers. Read only
|
||||
repository Secret metadata, never data. Identify external cluster destinations:
|
||||
an old controller can still manage a remote cluster. Search platform and tenant
|
||||
source for references to `argocd/applications/` and `argocd/bootstrap/`, including
|
||||
Make entry points, and match each live application to its accepting owner.
|
||||
|
||||
The railiance01 root uses `argocd/railiance01/applications`; the legacy root
|
||||
uses `argocd/applications`. Do not remove the legacy tree while the old controller
|
||||
can reconcile it with prune enabled. Keep evidence of each workload's current
|
||||
replicas, readiness and image before any retirement execution.
|
||||
|
||||
## Ordered execution after inventory and owner acceptance
|
||||
|
||||
1. Have the cluster owner pin the installed railiance01 ArgoCD version and
|
||||
reviewed resource requests in its canonical source. Inspect current requests;
|
||||
the original phase A BestEffort observation is historical, not a fresh check.
|
||||
2. Classify each old tracked resource: already adopted on railiance01, retained
|
||||
on coulombcore with another owner, or separately approved for retirement.
|
||||
An application name match does not prove matching cluster/resource identity.
|
||||
3. Freeze the old root and child reconciliation through the cluster owner's
|
||||
reviewed procedure. Confirm no running sync operation and no second writer.
|
||||
Preserve the old Application specs and controller configuration in protected
|
||||
recovery storage; repository credentials stay under existing custody.
|
||||
4. Detach only inventory-approved Application objects without cascading workload
|
||||
deletion. Review resource finalizers first; never delete the ArgoCD namespace
|
||||
or CRDs as a shortcut. Verify every retained workload is still healthy and
|
||||
each replacement owner can reconcile its accepted resource set.
|
||||
5. Disable the old controller through its actual installation owner. Prove it
|
||||
no longer reconciles and that no unrelated system uses its repository/auth
|
||||
resources. Revoke retired credentials only through their custody owners.
|
||||
6. Once the old controller is inert, remove the legacy source directories and
|
||||
retire or repoint their callers in one reviewed platform change. Render the
|
||||
railiance01 bootstrap and children and verify no legacy reference remains.
|
||||
|
||||
Stop for missing inventory, ambiguous tracking, active operations, unknown
|
||||
finalizers, missing acceptance or degraded retained workloads. Before detachment,
|
||||
rollback restores the recorded sync configuration. After replacement ownership,
|
||||
keep the old controller stopped until the replacement writer is explicitly
|
||||
suspended; rollback must never run two reconcilers against one workload. Package
|
||||
or data deletion needs its own exact approved disposition.
|
||||
|
||||
Completion evidence must include the inventory, accepting owners, source/caller
|
||||
changes, controller shutdown and retained-workload checks. T08's planning closure
|
||||
still requires the live read-only inventory. Execution remains with the existing
|
||||
cluster/infra owners; no new task or workplan is created here.
|
||||
492
docs/evidence/2026-09-27-argocd-loose-end-status.json
Normal file
492
docs/evidence/2026-09-27-argocd-loose-end-status.json
Normal file
|
|
@ -0,0 +1,492 @@
|
|||
{
|
||||
"observed_at": "2026-09-27T16:53:28.268968+00:00",
|
||||
"applications": [
|
||||
{
|
||||
"name": "activity-core",
|
||||
"sync": {
|
||||
"comparedTo": {
|
||||
"destination": {
|
||||
"namespace": "activity-core",
|
||||
"server": "https://kubernetes.default.svc"
|
||||
},
|
||||
"source": {
|
||||
"path": "k8s/gitops",
|
||||
"repoURL": "https://forgejo.coulomb.social/coulomb/activity-core.git",
|
||||
"targetRevision": "a12f1169f9d130058ce767f5b26de0606997916c"
|
||||
}
|
||||
},
|
||||
"revision": "a12f1169f9d130058ce767f5b26de0606997916c",
|
||||
"status": "Synced"
|
||||
},
|
||||
"health": {
|
||||
"lastTransitionTime": "2026-09-27T14:06:22Z",
|
||||
"status": "Healthy"
|
||||
},
|
||||
"reconciledAt": "2026-09-27T16:50:21Z",
|
||||
"operation": {
|
||||
"phase": "Succeeded",
|
||||
"startedAt": "2026-09-27T14:06:19Z",
|
||||
"finishedAt": "2026-09-27T14:06:20Z"
|
||||
},
|
||||
"automated": null,
|
||||
"conditions": [],
|
||||
"resources": [
|
||||
{
|
||||
"kind": "ConfigMap",
|
||||
"name": "actcore-external-activity-definitions",
|
||||
"namespace": "activity-core",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"kind": "ConfigMap",
|
||||
"name": "actcore-ops-service-inventory",
|
||||
"namespace": "activity-core",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"kind": "ConfigMap",
|
||||
"name": "actcore-report-schemas",
|
||||
"namespace": "activity-core",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"kind": "ConfigMap",
|
||||
"name": "actcore-runtime-config",
|
||||
"namespace": "activity-core",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"kind": "Service",
|
||||
"name": "actcore-api",
|
||||
"namespace": "activity-core",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"kind": "Service",
|
||||
"name": "actcore-worker-metrics",
|
||||
"namespace": "activity-core",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "apps",
|
||||
"kind": "Deployment",
|
||||
"name": "actcore-api",
|
||||
"namespace": "activity-core",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "apps",
|
||||
"kind": "Deployment",
|
||||
"name": "actcore-event-router",
|
||||
"namespace": "activity-core",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "apps",
|
||||
"kind": "Deployment",
|
||||
"name": "actcore-worker",
|
||||
"namespace": "activity-core",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "bao-notice",
|
||||
"sync": {
|
||||
"comparedTo": {
|
||||
"destination": {
|
||||
"namespace": "bao-notice",
|
||||
"server": "https://kubernetes.default.svc"
|
||||
},
|
||||
"source": {
|
||||
"path": "argocd/platform-addons/bao-notice",
|
||||
"repoURL": "https://forgejo.coulomb.social/coulomb/railiance-platform.git",
|
||||
"targetRevision": "9ee20967ca7b50a8926c99ae86fa027f8bfb0fa3"
|
||||
}
|
||||
},
|
||||
"revision": "9ee20967ca7b50a8926c99ae86fa027f8bfb0fa3",
|
||||
"status": "OutOfSync"
|
||||
},
|
||||
"health": {
|
||||
"lastTransitionTime": "2026-09-23T22:56:40Z",
|
||||
"status": "Healthy"
|
||||
},
|
||||
"reconciledAt": "2026-09-27T16:52:12Z",
|
||||
"operation": {
|
||||
"phase": "Succeeded",
|
||||
"startedAt": "2026-09-23T22:56:34Z",
|
||||
"finishedAt": "2026-09-23T22:56:36Z"
|
||||
},
|
||||
"automated": null,
|
||||
"conditions": [
|
||||
{
|
||||
"lastTransitionTime": "2026-09-23T22:55:19Z",
|
||||
"message": "Application has 1 orphaned resources",
|
||||
"type": "OrphanedResourceWarning"
|
||||
}
|
||||
],
|
||||
"resources": [
|
||||
{
|
||||
"kind": "ConfigMap",
|
||||
"name": "bao-notice-conf-4f5g9kc7c2",
|
||||
"namespace": "bao-notice",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"kind": "ConfigMap",
|
||||
"name": "bao-notice-conf-dg4hmf2dg4",
|
||||
"namespace": "bao-notice",
|
||||
"requiresPruning": true,
|
||||
"status": "OutOfSync",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"kind": "ConfigMap",
|
||||
"name": "bao-notice-html-6mm6h6mgkf",
|
||||
"namespace": "bao-notice",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"kind": "Namespace",
|
||||
"name": "bao-notice",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"kind": "Service",
|
||||
"name": "bao-notice",
|
||||
"namespace": "bao-notice",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "apps",
|
||||
"kind": "Deployment",
|
||||
"name": "bao-notice",
|
||||
"namespace": "bao-notice",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "networking.k8s.io",
|
||||
"kind": "Ingress",
|
||||
"name": "bao-notice",
|
||||
"namespace": "bao-notice",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "networking.k8s.io",
|
||||
"kind": "Ingress",
|
||||
"name": "bao-notice-http-redirect",
|
||||
"namespace": "bao-notice",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "networking.k8s.io",
|
||||
"kind": "NetworkPolicy",
|
||||
"name": "bao-notice-acme-solver",
|
||||
"namespace": "bao-notice",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "networking.k8s.io",
|
||||
"kind": "NetworkPolicy",
|
||||
"name": "bao-notice-isolation",
|
||||
"namespace": "bao-notice",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "traefik.io",
|
||||
"kind": "Middleware",
|
||||
"name": "redirect-https",
|
||||
"namespace": "bao-notice",
|
||||
"status": "Synced",
|
||||
"version": "v1alpha1"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "eso-token-renewer",
|
||||
"sync": {
|
||||
"comparedTo": {
|
||||
"destination": {
|
||||
"namespace": "external-secrets",
|
||||
"server": "https://kubernetes.default.svc"
|
||||
},
|
||||
"source": {
|
||||
"path": "argocd/platform-addons/eso-token-renewer",
|
||||
"repoURL": "https://forgejo.coulomb.social/coulomb/railiance-platform.git",
|
||||
"targetRevision": "b2ebe108495c4d06ebb6ca33893ea4a2a23c9725"
|
||||
}
|
||||
},
|
||||
"revision": "b2ebe108495c4d06ebb6ca33893ea4a2a23c9725",
|
||||
"status": "Synced"
|
||||
},
|
||||
"health": {
|
||||
"lastTransitionTime": "2026-09-23T22:38:19Z",
|
||||
"status": "Healthy"
|
||||
},
|
||||
"reconciledAt": "2026-09-27T16:50:17Z",
|
||||
"operation": {
|
||||
"phase": "Succeeded",
|
||||
"startedAt": "2026-09-23T22:38:19Z",
|
||||
"finishedAt": "2026-09-23T22:38:19Z"
|
||||
},
|
||||
"automated": null,
|
||||
"conditions": [
|
||||
{
|
||||
"lastTransitionTime": "2026-09-23T22:37:58Z",
|
||||
"message": "Application has 28 orphaned resources",
|
||||
"type": "OrphanedResourceWarning"
|
||||
}
|
||||
],
|
||||
"resources": [
|
||||
{
|
||||
"kind": "ConfigMap",
|
||||
"name": "eso-token-renewer-worker-5c86dt7fm7",
|
||||
"namespace": "external-secrets",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"kind": "ServiceAccount",
|
||||
"name": "eso-token-renewer",
|
||||
"namespace": "external-secrets",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "batch",
|
||||
"kind": "CronJob",
|
||||
"name": "eso-token-renewer",
|
||||
"namespace": "external-secrets",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "openbao-secretstore",
|
||||
"sync": {
|
||||
"comparedTo": {
|
||||
"destination": {
|
||||
"namespace": "external-secrets",
|
||||
"server": "https://kubernetes.default.svc"
|
||||
},
|
||||
"source": {
|
||||
"path": "argocd/platform-addons/openbao-secretstore",
|
||||
"repoURL": "https://forgejo.coulomb.social/coulomb/railiance-platform.git",
|
||||
"targetRevision": "d2dbc19c254247652c49fda8721c80d53bca206a"
|
||||
}
|
||||
},
|
||||
"revision": "d2dbc19c254247652c49fda8721c80d53bca206a",
|
||||
"status": "Synced"
|
||||
},
|
||||
"health": {
|
||||
"lastTransitionTime": "2026-09-21T17:09:00Z",
|
||||
"status": "Healthy"
|
||||
},
|
||||
"reconciledAt": "2026-09-27T16:52:14Z",
|
||||
"operation": {
|
||||
"phase": "Succeeded",
|
||||
"startedAt": "2026-09-21T17:09:16Z",
|
||||
"finishedAt": "2026-09-21T17:09:18Z"
|
||||
},
|
||||
"automated": null,
|
||||
"conditions": [
|
||||
{
|
||||
"lastTransitionTime": "2026-09-23T18:14:58Z",
|
||||
"message": "Application has 28 orphaned resources",
|
||||
"type": "OrphanedResourceWarning"
|
||||
}
|
||||
],
|
||||
"resources": [
|
||||
{
|
||||
"group": "external-secrets.io",
|
||||
"kind": "ClusterSecretStore",
|
||||
"name": "openbao",
|
||||
"status": "Synced",
|
||||
"version": "v1beta1"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "railiance-apps-root",
|
||||
"sync": {
|
||||
"comparedTo": {
|
||||
"destination": {
|
||||
"namespace": "argocd",
|
||||
"server": "https://kubernetes.default.svc"
|
||||
},
|
||||
"source": {
|
||||
"path": "argocd/railiance01/applications",
|
||||
"repoURL": "https://forgejo.coulomb.social/coulomb/railiance-platform.git",
|
||||
"targetRevision": "main"
|
||||
}
|
||||
},
|
||||
"revision": "5781d34b3b9a6e3779b913de200f0eaf63cabacd",
|
||||
"status": "Synced"
|
||||
},
|
||||
"health": {
|
||||
"lastTransitionTime": "2026-09-21T17:08:13Z",
|
||||
"status": "Healthy"
|
||||
},
|
||||
"reconciledAt": "2026-09-27T16:51:31Z",
|
||||
"operation": {
|
||||
"phase": "Succeeded",
|
||||
"startedAt": "2026-09-27T14:05:47Z",
|
||||
"finishedAt": "2026-09-27T14:05:49Z"
|
||||
},
|
||||
"automated": null,
|
||||
"conditions": [
|
||||
{
|
||||
"lastTransitionTime": "2026-09-21T17:08:13Z",
|
||||
"message": "Application has 1 orphaned resources",
|
||||
"type": "OrphanedResourceWarning"
|
||||
}
|
||||
],
|
||||
"resources": [
|
||||
{
|
||||
"group": "argoproj.io",
|
||||
"kind": "Application",
|
||||
"name": "activity-core",
|
||||
"namespace": "argocd",
|
||||
"status": "Synced",
|
||||
"version": "v1alpha1"
|
||||
},
|
||||
{
|
||||
"group": "argoproj.io",
|
||||
"kind": "Application",
|
||||
"name": "bao-notice",
|
||||
"namespace": "argocd",
|
||||
"status": "Synced",
|
||||
"syncWave": 10,
|
||||
"version": "v1alpha1"
|
||||
},
|
||||
{
|
||||
"group": "argoproj.io",
|
||||
"kind": "Application",
|
||||
"name": "eso-token-renewer",
|
||||
"namespace": "argocd",
|
||||
"status": "Synced",
|
||||
"syncWave": 2,
|
||||
"version": "v1alpha1"
|
||||
},
|
||||
{
|
||||
"group": "argoproj.io",
|
||||
"kind": "Application",
|
||||
"name": "openbao-secretstore",
|
||||
"namespace": "argocd",
|
||||
"status": "Synced",
|
||||
"syncWave": 1,
|
||||
"version": "v1alpha1"
|
||||
},
|
||||
{
|
||||
"group": "argoproj.io",
|
||||
"kind": "Application",
|
||||
"name": "target-revenue",
|
||||
"namespace": "argocd",
|
||||
"status": "Synced",
|
||||
"syncWave": 10,
|
||||
"version": "v1alpha1"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "target-revenue",
|
||||
"sync": {
|
||||
"comparedTo": {
|
||||
"destination": {
|
||||
"namespace": "target-revenue",
|
||||
"server": "https://kubernetes.default.svc"
|
||||
},
|
||||
"source": {
|
||||
"path": "k8s/railiance",
|
||||
"repoURL": "https://forgejo.coulomb.social/coulomb/target-revenue.git",
|
||||
"targetRevision": "f1109d54eeda9f187daa215cf1c7163610d35d0a"
|
||||
}
|
||||
},
|
||||
"revision": "f1109d54eeda9f187daa215cf1c7163610d35d0a",
|
||||
"status": "Synced"
|
||||
},
|
||||
"health": {
|
||||
"lastTransitionTime": "2026-09-21T17:13:49Z",
|
||||
"status": "Healthy"
|
||||
},
|
||||
"reconciledAt": "2026-09-27T16:50:17Z",
|
||||
"operation": {
|
||||
"phase": "Succeeded",
|
||||
"startedAt": "2026-09-21T17:14:01Z",
|
||||
"finishedAt": "2026-09-21T17:14:10Z"
|
||||
},
|
||||
"automated": null,
|
||||
"conditions": [
|
||||
{
|
||||
"lastTransitionTime": "2026-09-21T17:13:49Z",
|
||||
"message": "Application has 5 orphaned resources",
|
||||
"type": "OrphanedResourceWarning"
|
||||
}
|
||||
],
|
||||
"resources": [
|
||||
{
|
||||
"kind": "Service",
|
||||
"name": "target-revenue",
|
||||
"namespace": "target-revenue",
|
||||
"status": "Synced",
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "apps",
|
||||
"kind": "Deployment",
|
||||
"name": "target-revenue",
|
||||
"namespace": "target-revenue",
|
||||
"status": "Synced",
|
||||
"syncWave": 3,
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "external-secrets.io",
|
||||
"kind": "ExternalSecret",
|
||||
"name": "target-revenue-runtime",
|
||||
"namespace": "target-revenue",
|
||||
"status": "Synced",
|
||||
"version": "v1beta1"
|
||||
},
|
||||
{
|
||||
"group": "networking.k8s.io",
|
||||
"kind": "Ingress",
|
||||
"name": "target-revenue",
|
||||
"namespace": "target-revenue",
|
||||
"status": "Synced",
|
||||
"syncWave": 4,
|
||||
"version": "v1"
|
||||
},
|
||||
{
|
||||
"group": "postgresql.cnpg.io",
|
||||
"kind": "Cluster",
|
||||
"name": "target-revenue-pg",
|
||||
"namespace": "target-revenue",
|
||||
"status": "Synced",
|
||||
"syncWave": -2,
|
||||
"version": "v1"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"coulombcore_inventory": "unavailable: Kubernetes Unauthorized, including explicit local k3s kubeconfig; no credential or controller mutation"
|
||||
}
|
||||
12
docs/evidence/2026-09-27-eso-adoption-preparation.json
Normal file
12
docs/evidence/2026-09-27-eso-adoption-preparation.json
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
{
|
||||
"observed_at": "2026-09-27T16:59:10.112128+00:00",
|
||||
"chart": "0.16.1",
|
||||
"draft_sha256": "a0b349cddd90aa707f080efd8556c8379d5b178d93b4c9e408f3b7f439bbf064",
|
||||
"render_sha256": "55236645afc9f2d9d376c28e73aed70bd31727d0e6f166664b3ebd568d5f136f",
|
||||
"objects": 39,
|
||||
"protected_crds": 20,
|
||||
"server_diff_exit_code": 1,
|
||||
"applied": false,
|
||||
"diff_summary": "Exactly 20 CRD metadata annotation additions; no spec, workload or other object changes",
|
||||
"diff_sha256": "3c34ad673fc374442adc13eeaf17c5216fce7b904103e63014d10e4ba6272965"
|
||||
}
|
||||
33
docs/evidence/2026-09-27-keycape-incident-owner-return.json
Normal file
33
docs/evidence/2026-09-27-keycape-incident-owner-return.json
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
{
|
||||
"schema": "platform.incident-owner-return.v1",
|
||||
"reviewed_at": "2026-09-27T17:01:20.402024+00:00",
|
||||
"task": "RPF-WP-0027-T05",
|
||||
"sources": [
|
||||
{
|
||||
"repository": "net-kingdom",
|
||||
"path": "workplans/NK-WP-0033-keycape-secret-exposure-rotation.md",
|
||||
"commit": "1da6e5457ad86fff1173bd2ff6174d70719b487d",
|
||||
"sha256": "8268aa86f7fa9f2a3848adfe2dc3022bcf2e9dc9197328993445f4138a40e513"
|
||||
},
|
||||
{
|
||||
"repository": "key-cape",
|
||||
"path": "workplans/KEY-WP-0011-live-secret-exposure-recovery.md",
|
||||
"commit": "6a996bd71e5e36d7483e7a79b3301bd895907644",
|
||||
"sha256": "3c5458180fe81a04e357f8db737e4287f79640b5ab02682bcadc595b73f846cd"
|
||||
}
|
||||
],
|
||||
"operator_ruling_date": "2026-09-23",
|
||||
"operator": "Bernd Worsch",
|
||||
"owner_check": "reconcile-lldap-resolver-live.sh --check --predecessor-unavailable",
|
||||
"owner_receipt": {
|
||||
"resolver_lookup": "PASS",
|
||||
"privacyidea_mfa": "PASS",
|
||||
"predecessor_denial": "NOT-PROVEN",
|
||||
"readiness": "PASS",
|
||||
"health": "PASS",
|
||||
"cleanup": "PASS"
|
||||
},
|
||||
"disposition": "Operator explicitly accepted the August 27 observations; unrecoverable predecessor is not a required new test. NetKingdom marks incident closed.",
|
||||
"live_action_performed_in_this_review": false,
|
||||
"custody_handoff_complete": false
|
||||
}
|
||||
192
docs/evidence/2026-09-27-service-assurance.json
Normal file
192
docs/evidence/2026-09-27-service-assurance.json
Normal file
|
|
@ -0,0 +1,192 @@
|
|||
{
|
||||
"observation": {
|
||||
"schema": "railiance-platform.observation.v1",
|
||||
"cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786",
|
||||
"captured_at": "2026-09-27T16:53:44.173824+00:00",
|
||||
"signals": {
|
||||
"apps-pg.ready": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:28.888062+00:00"
|
||||
},
|
||||
"apps-pg.backup": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T02:15:16Z"
|
||||
},
|
||||
"apps-pg.wal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:28.888098+00:00"
|
||||
},
|
||||
"apps-pg.headroom": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:16Z"
|
||||
},
|
||||
"platform-pg.ready": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:32.163508+00:00"
|
||||
},
|
||||
"platform-pg.backup": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T02:15:18Z"
|
||||
},
|
||||
"platform-pg.wal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:32.163550+00:00"
|
||||
},
|
||||
"platform-pg.headroom": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:32Z"
|
||||
},
|
||||
"platform-pg-2.ready": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:35.169803+00:00"
|
||||
},
|
||||
"platform-pg-2.backup": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T02:15:14Z"
|
||||
},
|
||||
"platform-pg-2.wal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:35.169827+00:00"
|
||||
},
|
||||
"platform-pg-2.headroom": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:22Z"
|
||||
},
|
||||
"openbao.seal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:42.333471+00:00"
|
||||
},
|
||||
"eso.ready": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:53:43.311315+00:00"
|
||||
},
|
||||
"eso.refresh": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T16:06:45Z"
|
||||
},
|
||||
"eso.token-renewal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-27T02:40:09Z"
|
||||
},
|
||||
"apps-pg.restore": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T22:30:45.208512+00:00"
|
||||
},
|
||||
"forgejo-db.restore": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T22:55:54.893587+00:00"
|
||||
},
|
||||
"openbao.snapshot": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-08-22T22:29:21Z"
|
||||
}
|
||||
}
|
||||
},
|
||||
"evaluation": {
|
||||
"schema": "railiance-platform.assurance-signal.v1",
|
||||
"cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786",
|
||||
"evaluated_at": "2026-09-27T16:53:44.173824+00:00",
|
||||
"signals": {
|
||||
"apps-pg.ready": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.backup": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.wal": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.restore": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.headroom": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"platform-pg.ready": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.backup": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.wal": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.restore": {
|
||||
"state": "missing",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.headroom": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.ready": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.backup": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.wal": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.restore": {
|
||||
"state": "missing",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.headroom": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"openbao.seal": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"openbao.snapshot": {
|
||||
"state": "stale",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"openbao.restore": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"offsite.upload": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"offsite.restore": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"eso.ready": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"eso.refresh": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"forgejo-db.restore": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"eso.token-renewal": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
}
|
||||
},
|
||||
"transport": "unmonitored",
|
||||
"guarantees": "unsupported",
|
||||
"threshold_status": "local-diagnostic-only",
|
||||
"healthy": false
|
||||
}
|
||||
}
|
||||
|
|
@ -3,8 +3,8 @@
|
|||
Historical resolver lanes: **draft / blocked**.
|
||||
KeyCape factor service lane: **active**, established and verified 2026-09-13 (below).
|
||||
Incident: `KEYCAPE-EXPOSURE-20260823-01`
|
||||
Consumer procedure: NetKingdom `NK-WP-0033`, resolver reconciliation revision
|
||||
`eec7007` / checkout `f2e578c`
|
||||
Consumer procedure: NetKingdom `NK-WP-0033`; latest attended check used
|
||||
checkout `6096c395` (script `4a38511`) on 2026-09-23.
|
||||
|
||||
This document defines the Railiance-side contract without containing or
|
||||
deriving any credential value. It is not an authorization to fetch, export,
|
||||
|
|
@ -49,6 +49,17 @@ Operator decision 2026-09-15: leave both historical resolver lanes blocked.
|
|||
Do not invent mount, path, or field names. The KeyCape factor service lane
|
||||
below is separate and does not close this gate.
|
||||
|
||||
September 27 evidence review: NetKingdom's September 23 operator ruling and
|
||||
green attended `--check --predecessor-unavailable` receipt close the incident
|
||||
verification obligation (RPF-WP-0027-T05). The predecessor remains NOT-PROVEN;
|
||||
the operator explicitly accepted its unavailable disposition. The owner records
|
||||
human custody at `operators/lldap/admin` and `operators/privacyidea/pi-admin`,
|
||||
including KV v2 LLDAP version 1 and withheld delete under `operator-custody`.
|
||||
These are confirmed owner coordinates, but they do not by themselves establish
|
||||
either historical route's complete field, auth, expiry and handoff contract.
|
||||
T03/T06 and the non-resolvable historical routes therefore remain blocked.
|
||||
See `docs/evidence/2026-09-27-keycape-incident-owner-return.json`.
|
||||
|
||||
## KeyCape factor service lane — authorized setup, 2026-09-13
|
||||
|
||||
RPF-WP-0040 / CCR-2026-0023 establish a new dedicated service lane. The user,
|
||||
|
|
|
|||
|
|
@ -61,13 +61,12 @@ CronJob status timestamps only. It fails when a newer scheduled run has not
|
|||
succeeded within an hour, and it goes stale after 36h. The tokens lapse after
|
||||
7 days without renewal.
|
||||
|
||||
The following remain missing until a native value-safe adapter and acceptance
|
||||
exist: validated isolated restore receipts,
|
||||
OpenBao snapshot/restore proof, and offsite upload/restore receipts. Missing
|
||||
adapters are not inferred healthy from pod readiness. The local producer is
|
||||
not the Q2 standard; railiance-telemetry has no implemented receiving contract
|
||||
in the reviewed checkout. Integration, routing and scheduled delivery remain
|
||||
T04, and no notification was sent during implementation.
|
||||
Current gaps include accepted platform-pg/platform-pg-2 and OpenBao isolated
|
||||
restore samples, fresh OpenBao snapshots, and dated full-archive receipts.
|
||||
Missing evidence is not inferred healthy from pod readiness. RTEL-WP-0002
|
||||
implements the Q2 reference contract and local delivery tests; its T04 still
|
||||
owns production mapping, recipient and controlled failure/absence acceptance.
|
||||
Integration, routing and scheduled delivery remain T04 here.
|
||||
|
||||
## Service records and evidence inventory
|
||||
|
||||
|
|
@ -87,7 +86,7 @@ provider invalidation/replacement recovery for the shared offsite lane.
|
|||
| OpenBao snapshot | WARDEN-WP-0027 preparation receipt, 2026-08-23 | Snapshot/encrypted off-host preparation, not isolated restore |
|
||||
| OpenBao restore | Existing `openbao-validate-restore-evidence.sh` and package procedure | Example receipt cannot pass as a fresh execution |
|
||||
| Recovery exercise | RPF-WP-0015-T02/T03 | Separate windows, synthetic driver/quorum and abort operator required |
|
||||
| Logical/Forgejo offsite | activity-core backup definitions, existing helper/runbooks | No new upload/restore performed; RPF-WP-0029 remains open |
|
||||
| Logical/Forgejo offsite | activity-core backup definitions, existing helper/runbooks | WP-0029 is finished; WP-0038 retains recurring primary/secondary activation |
|
||||
|
||||
## Admission and disclosure drift
|
||||
|
||||
|
|
@ -134,8 +133,26 @@ Decryption now retains `platform.forgejo-primary-decryption.v1` and its own
|
|||
operation times; older decryption receipts used the transfer schema through an
|
||||
overwrite bug. The restore tool explicitly accepts both forms, with verified
|
||||
hash/decryption flags. Existing historical receipts are unchanged. These producer
|
||||
fixes enable future dated archive evidence; automatic archive adapters, fresh
|
||||
end-to-end receipts and recurring execution are still pending.
|
||||
fixes enable dated archive evidence. The full primary archive adapter is now
|
||||
implemented; fresh end-to-end receipts and recurring execution remain pending.
|
||||
|
||||
For `offsite.upload`, add a reviewed entry with `signal`, `path` and `sha256`
|
||||
to the recovery index. It must name a full-profile Scaleway archive transfer
|
||||
with completed multipart upload, version-pinned GET, matching byte counts/hash,
|
||||
the application-archive destination and ordered timezone-aware timestamps.
|
||||
For `offsite.restore`, the entry additionally names `decryption` and `transfer`,
|
||||
each with `path` and `sha256`. The restore must attest database import, application
|
||||
health, repository verification, all package blobs and successful scratch cleanup.
|
||||
The receipt hashes, ciphertext identity, destination, profile and operation order
|
||||
must match across all three receipts. Only the distinct decryption schema is
|
||||
accepted for automatic assurance. Essentials and Nextcloud-only proofs cannot
|
||||
substitute for this primary full-application recovery signal. Other supported
|
||||
services still need their own evidence; one Forgejo receipt does not close T03.
|
||||
|
||||
No historical index entry was added: the September 6 archive receipts lack
|
||||
operation timestamps. They remain manual evidence. Tests use synthetic receipt
|
||||
chains and prove expiry, provenance rejection and rejection of the real undated
|
||||
receipt; those fixtures do not assert a new live recovery.
|
||||
|
||||
The OpenBao snapshot adapter also accepts the reviewed, hash-pinned receipt in
|
||||
`reviews/`. It requires the expected source cluster identity, encrypted off-host
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue