From 9e231b984d01801e44e44e40a17d832b116d9738 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 3 Oct 2026 11:52:48 +0200 Subject: [PATCH] Prepare platform recovery scheduling contract and refresh blockers Verify the repaired ESO lane, separate actor ownership from scheduling, and define the first bounded recovery slice and owner contract. Keep role acceptance and live operation admission explicit. Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a100de-2e1c-7e23-9ae9-2da0e0499411 --- docs/platform-recovery-scheduling-plan.md | 110 ++++++++++++++++++ ...form-backup-restore-evidence-scheduling.md | 51 +++++++- 2 files changed, 156 insertions(+), 5 deletions(-) create mode 100644 docs/platform-recovery-scheduling-plan.md diff --git a/docs/platform-recovery-scheduling-plan.md b/docs/platform-recovery-scheduling-plan.md new file mode 100644 index 0000000..bd52605 --- /dev/null +++ b/docs/platform-recovery-scheduling-plan.md @@ -0,0 +1,110 @@ +# Platform recovery scheduling — reviewed proposal + +Reviewed 2026-10-03 for ACTIVITY-WP-0042. This is a proposed delivery contract, +not admission of a new operation or permission to run it. + +## Recommended decision + +Accept **scheduling ownership only** for activity-core: Temporal triggers, +bounded dispatch, run correlation, missed-run visibility, and normalized result +evidence. Railiance Platform retains recovery implementations, acceptance +validators, cadence/freshness policy, destinations, retention, and credential +custody. Existing ACT-ADR-007 operations remain unchanged. + +Approve offline design and tests of the invocation contract now. Admit each +new mutating operation separately after its owner supplies the contract and +ACT-ADR-007 is amended or superseded. No new generic shell executor, restore +schedule, retention action, or production rollout follows from role acceptance. + +## What the investigation established + +- The current definition `activity-definitions/daily-cnpg-option-a-backup.md` + selects three logical Nextcloud targets at 02:30 UTC. It does not produce + Scaleway physical recovery or Forgejo archive recovery evidence. Its old + host kubeconfig names and RPO wording are not proof of current cluster + routing or an accepted guarantee; review those before reusing the wiring. +- `src/activity_core/bounded_operations.py` permits exactly three operations. + CNPG backup has one Temporal attempt because no durable remote idempotency + receipt is available. New restore dispatch must not inherit retry defaults. +- The stale-token blocker was resolved under RPF-WP-0045. Read-only inspection + on 2026-10-03 confirmed node IP 92.205.62.239, ClusterSecretStore + `openbao-activity-core`: Ready=True/Valid, and ExternalSecret + `activity-core/actcore-backup-offsite`: Ready=True/SecretSynced, last refresh + 2026-10-03T09:29:02Z. No Secret object or value was read. This verifies the + existing Nextcloud lane, not permission or credentials for a new restore. +- `warden route show railiance-backup-offsite-lane --json` names + railiance-platform as credential owner and reports workload resolution + unknown. Recovery escrow must stay outside the ordinary backup worker. +- Platform's `scripts/verify_scaleway_primary_restore.py` supports apps-pg + and forgejo-db, checks the cluster/destination, restores into a scratch + namespace, validates database contents, and attempts cleanup. Its CLI has + no dry-run option, approval-reference input, or durable idempotency key. + Do not register it directly as an unattended bounded operation. +- Platform's `scripts/restore_forgejo_offsite_locally.py` validates a transfer + receipt and supports full/essentials profiles. Its archive/decryption input + and local container execution need a separately scoped owner contract. +- RPF-WP-0038 retains the September 15 prohibition on backup expiration and + scheduled secondary cutover pending caller/inventory/quota gates. It is not + lifted by this proposal. RPF-WP-0036 retains cadence and evidence acceptance. +- Ops Warden's `registry/flex-auth/production_registry_snapshot.json` calls + `atm-backup-daily` an "Example nightly automation actor" and associates it + with workload `backup-daily`, but records resolution as absent. That is not + evidence that activity-core owns or uses it. T03 remains independent. + +Platform message `0638c13b-cf6f-4342-9714-dfb1b6c67fc7` already consents to +the scheduling-only boundary and commits to supplying the credential route once +the operation contract exists. It specifies per-run approval as the interim; +there is no standing grant. This proposal resolves its obsolete token-expiry +prerequisite and makes the remaining contract explicit. + +## First delivery slice + +Start with a single manually triggered, disabled-by-default **apps-pg physical +recovery evidence run** against Scaleway, using a platform-owned isolated +runner. This is a proposal for the first target, not an expansion of the +existing logical-backup job. Add recurring triggers only after one admitted +run and cleanup receipt have passed. Forgejo archive tiers and retention +remain later slices with separate input/receipt contracts. + +Before coding a production adapter, the platform owner must supply: + +| Contract item | Required evidence | +| --- | --- | +| Fixed operation and implementation | Owner-reviewed version/digest; no caller-provided command or import path | +| Targets | Exact cluster identity, source apps-pg, scratch namespace policy; production never a restore destination | +| Admission | Explicit dry-run semantics and per-run authority checked by the executing enforcement point; an approval reference alone is insufficient | +| Time and capacity | Agreed timeout and CPU/memory/storage ceilings including cleanup; no new numbers inferred from historical tests | +| Credentials | Scoped workload route and recovery custody; no borrowed host-admin kubeconfig or escrow added to activity-core | +| Replay and overlap | Stable run/operation ID, durable owner receipt lookup and single-flight behavior; one dispatch attempt until retry safety is proven | +| Cleanup | Confirmed scratch removal; timeout or ambiguous result requires owner reconciliation before another mutation | +| Receipt | Schema/version, run ID, implementation revision, target identity, start/finish times, terminal result, validation and cleanup booleans, authoritative receipt hash/reference | + +The scheduler retains only allowlisted receipt fields. Raw subprocess output, +SQL/application contents, archive locations, credentials and provider payloads +stay out of State Hub evidence. Failed, missing, stale and unverified receipts +must remain distinguishable. A scheduled fire or successful upload is not a +successful restore. + +## Implementation and verification sequence + +1. Founder accepts the scheduling-only boundary (T01). This resolves only + ACTIVITY-WP-0042's role choice, not the other decisions bundled in D2. +2. Agree the fixed runner contract above with railiance-platform (T05). +3. Build offline adapter/admission tests within T02. Cover unknown operation, + target drift, absent/expired/wrong-scope authority, duplicate dispatch, + timeouts, malformed/mismatched/stale receipts, failed cleanup and secret + canaries in child output. Use a fake owner runner; no live restore required. +4. Review the ADR change and exact credential/evidence contract before + registering the operation. Keep definitions disabled pending release. +5. Platform owner performs an approved isolated canary. Record evidence and + cleanup, then agree cadence/freshness/retention policy and rollout separately. +6. Enable the agreed Temporal schedule through the governed release path + (ACTIVITY-WP-0041), and verify missed-run and failed-evidence reporting. + +## Independent identity question + +Do not assign `atm-backup-daily` to activity-core from its name. Obtain an +issuance/deployment record identifying the actual execution unit, or a founder +ruling that it is an unused example to retire. Scheduling development does not +need to claim this historical actor. No owner message has been sent by this +review; the contract and identity questions are recorded as workplan tasks. diff --git a/workplans/ACTIVITY-WP-0042-platform-backup-restore-evidence-scheduling.md b/workplans/ACTIVITY-WP-0042-platform-backup-restore-evidence-scheduling.md index 617b645..feea171 100644 --- a/workplans/ACTIVITY-WP-0042-platform-backup-restore-evidence-scheduling.md +++ b/workplans/ACTIVITY-WP-0042-platform-backup-restore-evidence-scheduling.md @@ -27,8 +27,11 @@ Facts from the repo (2026-10-03): - ACT-ADR-007 allowlist is closed. A restore-evidence or Scaleway-backup run needs a new code-registered operation plus an ADR successor, tests, a credential route and an evidence contract. A definition string cannot add one. -- The `actcore-backup-offsite` lane depends on ClusterSecretStore - `openbao-activity-core`, reported failing since 2026-09-19 (expired token). +- The old expired-token finding is superseded: RPF-WP-0045 replaced static + tokens with Kubernetes auth. On 2026-10-03, read-only railiance01 inspection + confirmed `openbao-activity-core` Ready/Valid and `actcore-backup-offsite` + Ready/SecretSynced (refresh 09:29:02Z). This verifies the existing Nextcloud + lane only; new recovery authority and credential scope remain unapproved. ## Decide whether activity-core accepts the scheduling role @@ -62,15 +65,15 @@ founder confirmation. activity-core replied with the same position. ```task id: ACTIVITY-WP-0042-T02 status: wait -depends_on: [ACTIVITY-WP-0042-T01] -blocking_reason: "Depends on T01: no bounded operation or credential lane until the founder accepts the role." +depends_on: [ACTIVITY-WP-0042-T01, ACTIVITY-WP-0042-T05] +blocking_reason: "Requires role acceptance (T01) and the agreed platform runner contract (T05) before operation or credential admission." priority: medium state_hub_task_id: "250c6e8e-7281-5dec-9a92-06081b8c96bb" ``` Blocked on T01. Successor to ACT-ADR-007, registry entry (target bounds, timeout, idempotency, dry_run explicit), tests, credential route via OpenBao -(owner railiance-platform) with a repaired `openbao-activity-core` store, and +(owner railiance-platform; existing store health verified), and non-secret evidence mode. Scheduled caller satisfies RPF-WP-0038-T04 only once recurring tiered scheduling and retention are accepted by their owners. @@ -94,3 +97,41 @@ otherwise the declaring owner is whoever holds the actor's issuance (railiance-platform was also asked). Never infer membership. 2026-10-03: human gate routed to hub decision 89f1058f-33bc-47a8-8a49-0b44d426c67e (D2: ArgoCD owns production: go-ahead sequence for railiance-apps-root, issue-core, policy-nexus and activity-core automation) by CUST-WP-0075; resolving that decision raises C-40 here. + + +## Prepare the concrete scheduling and runner proposal + +```task +id: ACTIVITY-WP-0042-T04 +status: done +priority: high +``` + +User requested investigation and forward progress on 2026-10-03. Reviewed the +existing definition, bounded-operation registry, platform recovery entry points, +credential route and actor declaration. Removed the stale ESO blocker after a +fresh metadata-only check. Concrete recommendation and staged acceptance: +`docs/platform-recovery-scheduling-plan.md`. + +Recommend scheduling-only ownership; platform retains the recovery runner and +policy. First proposed slice is one disabled/manual apps-pg recovery-evidence +operation, with offline contract tests before any live admission. T01 is still +an explicit role choice; this investigation does not record acceptance of it +or resolve the broader D2 decision. The actor ownership question is separate. + +## Agree the platform runner contract + +```task +id: ACTIVITY-WP-0042-T05 +status: wait +priority: high +depends_on: [ACTIVITY-WP-0042-T01] +blocking_reason: "Scheduling role acceptance and platform agreement on the fixed runner contract are needed before adapter admission." +``` + +Use the contract table in `docs/platform-recovery-scheduling-plan.md` for an +owner review: authority verification, exact target, implementation version, +durable run receipts, overlap/retry behavior, cleanup, resource ceilings and +credential scope. Existing restore CLIs are not an admitted unattended API. +This task records the dependency; no external acceptance or message is claimed. +T02 implements/adopts the agreed contract; T03 does not gate offline design.