diff --git a/docs/platform-recovery-scheduling-plan.md b/docs/platform-recovery-scheduling-plan.md new file mode 100644 index 0000000..bd52605 --- /dev/null +++ b/docs/platform-recovery-scheduling-plan.md @@ -0,0 +1,110 @@ +# Platform recovery scheduling — reviewed proposal + +Reviewed 2026-10-03 for ACTIVITY-WP-0042. This is a proposed delivery contract, +not admission of a new operation or permission to run it. + +## Recommended decision + +Accept **scheduling ownership only** for activity-core: Temporal triggers, +bounded dispatch, run correlation, missed-run visibility, and normalized result +evidence. Railiance Platform retains recovery implementations, acceptance +validators, cadence/freshness policy, destinations, retention, and credential +custody. Existing ACT-ADR-007 operations remain unchanged. + +Approve offline design and tests of the invocation contract now. Admit each +new mutating operation separately after its owner supplies the contract and +ACT-ADR-007 is amended or superseded. No new generic shell executor, restore +schedule, retention action, or production rollout follows from role acceptance. + +## What the investigation established + +- The current definition `activity-definitions/daily-cnpg-option-a-backup.md` + selects three logical Nextcloud targets at 02:30 UTC. It does not produce + Scaleway physical recovery or Forgejo archive recovery evidence. Its old + host kubeconfig names and RPO wording are not proof of current cluster + routing or an accepted guarantee; review those before reusing the wiring. +- `src/activity_core/bounded_operations.py` permits exactly three operations. + CNPG backup has one Temporal attempt because no durable remote idempotency + receipt is available. New restore dispatch must not inherit retry defaults. +- The stale-token blocker was resolved under RPF-WP-0045. Read-only inspection + on 2026-10-03 confirmed node IP 92.205.62.239, ClusterSecretStore + `openbao-activity-core`: Ready=True/Valid, and ExternalSecret + `activity-core/actcore-backup-offsite`: Ready=True/SecretSynced, last refresh + 2026-10-03T09:29:02Z. No Secret object or value was read. This verifies the + existing Nextcloud lane, not permission or credentials for a new restore. +- `warden route show railiance-backup-offsite-lane --json` names + railiance-platform as credential owner and reports workload resolution + unknown. Recovery escrow must stay outside the ordinary backup worker. +- Platform's `scripts/verify_scaleway_primary_restore.py` supports apps-pg + and forgejo-db, checks the cluster/destination, restores into a scratch + namespace, validates database contents, and attempts cleanup. Its CLI has + no dry-run option, approval-reference input, or durable idempotency key. + Do not register it directly as an unattended bounded operation. +- Platform's `scripts/restore_forgejo_offsite_locally.py` validates a transfer + receipt and supports full/essentials profiles. Its archive/decryption input + and local container execution need a separately scoped owner contract. +- RPF-WP-0038 retains the September 15 prohibition on backup expiration and + scheduled secondary cutover pending caller/inventory/quota gates. It is not + lifted by this proposal. RPF-WP-0036 retains cadence and evidence acceptance. +- Ops Warden's `registry/flex-auth/production_registry_snapshot.json` calls + `atm-backup-daily` an "Example nightly automation actor" and associates it + with workload `backup-daily`, but records resolution as absent. That is not + evidence that activity-core owns or uses it. T03 remains independent. + +Platform message `0638c13b-cf6f-4342-9714-dfb1b6c67fc7` already consents to +the scheduling-only boundary and commits to supplying the credential route once +the operation contract exists. It specifies per-run approval as the interim; +there is no standing grant. This proposal resolves its obsolete token-expiry +prerequisite and makes the remaining contract explicit. + +## First delivery slice + +Start with a single manually triggered, disabled-by-default **apps-pg physical +recovery evidence run** against Scaleway, using a platform-owned isolated +runner. This is a proposal for the first target, not an expansion of the +existing logical-backup job. Add recurring triggers only after one admitted +run and cleanup receipt have passed. Forgejo archive tiers and retention +remain later slices with separate input/receipt contracts. + +Before coding a production adapter, the platform owner must supply: + +| Contract item | Required evidence | +| --- | --- | +| Fixed operation and implementation | Owner-reviewed version/digest; no caller-provided command or import path | +| Targets | Exact cluster identity, source apps-pg, scratch namespace policy; production never a restore destination | +| Admission | Explicit dry-run semantics and per-run authority checked by the executing enforcement point; an approval reference alone is insufficient | +| Time and capacity | Agreed timeout and CPU/memory/storage ceilings including cleanup; no new numbers inferred from historical tests | +| Credentials | Scoped workload route and recovery custody; no borrowed host-admin kubeconfig or escrow added to activity-core | +| Replay and overlap | Stable run/operation ID, durable owner receipt lookup and single-flight behavior; one dispatch attempt until retry safety is proven | +| Cleanup | Confirmed scratch removal; timeout or ambiguous result requires owner reconciliation before another mutation | +| Receipt | Schema/version, run ID, implementation revision, target identity, start/finish times, terminal result, validation and cleanup booleans, authoritative receipt hash/reference | + +The scheduler retains only allowlisted receipt fields. Raw subprocess output, +SQL/application contents, archive locations, credentials and provider payloads +stay out of State Hub evidence. Failed, missing, stale and unverified receipts +must remain distinguishable. A scheduled fire or successful upload is not a +successful restore. + +## Implementation and verification sequence + +1. Founder accepts the scheduling-only boundary (T01). This resolves only + ACTIVITY-WP-0042's role choice, not the other decisions bundled in D2. +2. Agree the fixed runner contract above with railiance-platform (T05). +3. Build offline adapter/admission tests within T02. Cover unknown operation, + target drift, absent/expired/wrong-scope authority, duplicate dispatch, + timeouts, malformed/mismatched/stale receipts, failed cleanup and secret + canaries in child output. Use a fake owner runner; no live restore required. +4. Review the ADR change and exact credential/evidence contract before + registering the operation. Keep definitions disabled pending release. +5. Platform owner performs an approved isolated canary. Record evidence and + cleanup, then agree cadence/freshness/retention policy and rollout separately. +6. Enable the agreed Temporal schedule through the governed release path + (ACTIVITY-WP-0041), and verify missed-run and failed-evidence reporting. + +## Independent identity question + +Do not assign `atm-backup-daily` to activity-core from its name. Obtain an +issuance/deployment record identifying the actual execution unit, or a founder +ruling that it is an unused example to retire. Scheduling development does not +need to claim this historical actor. No owner message has been sent by this +review; the contract and identity questions are recorded as workplan tasks. diff --git a/workplans/ACTIVITY-WP-0042-platform-backup-restore-evidence-scheduling.md b/workplans/ACTIVITY-WP-0042-platform-backup-restore-evidence-scheduling.md index 617b645..feea171 100644 --- a/workplans/ACTIVITY-WP-0042-platform-backup-restore-evidence-scheduling.md +++ b/workplans/ACTIVITY-WP-0042-platform-backup-restore-evidence-scheduling.md @@ -27,8 +27,11 @@ Facts from the repo (2026-10-03): - ACT-ADR-007 allowlist is closed. A restore-evidence or Scaleway-backup run needs a new code-registered operation plus an ADR successor, tests, a credential route and an evidence contract. A definition string cannot add one. -- The `actcore-backup-offsite` lane depends on ClusterSecretStore - `openbao-activity-core`, reported failing since 2026-09-19 (expired token). +- The old expired-token finding is superseded: RPF-WP-0045 replaced static + tokens with Kubernetes auth. On 2026-10-03, read-only railiance01 inspection + confirmed `openbao-activity-core` Ready/Valid and `actcore-backup-offsite` + Ready/SecretSynced (refresh 09:29:02Z). This verifies the existing Nextcloud + lane only; new recovery authority and credential scope remain unapproved. ## Decide whether activity-core accepts the scheduling role @@ -62,15 +65,15 @@ founder confirmation. activity-core replied with the same position. ```task id: ACTIVITY-WP-0042-T02 status: wait -depends_on: [ACTIVITY-WP-0042-T01] -blocking_reason: "Depends on T01: no bounded operation or credential lane until the founder accepts the role." +depends_on: [ACTIVITY-WP-0042-T01, ACTIVITY-WP-0042-T05] +blocking_reason: "Requires role acceptance (T01) and the agreed platform runner contract (T05) before operation or credential admission." priority: medium state_hub_task_id: "250c6e8e-7281-5dec-9a92-06081b8c96bb" ``` Blocked on T01. Successor to ACT-ADR-007, registry entry (target bounds, timeout, idempotency, dry_run explicit), tests, credential route via OpenBao -(owner railiance-platform) with a repaired `openbao-activity-core` store, and +(owner railiance-platform; existing store health verified), and non-secret evidence mode. Scheduled caller satisfies RPF-WP-0038-T04 only once recurring tiered scheduling and retention are accepted by their owners. @@ -94,3 +97,41 @@ otherwise the declaring owner is whoever holds the actor's issuance (railiance-platform was also asked). Never infer membership. 2026-10-03: human gate routed to hub decision 89f1058f-33bc-47a8-8a49-0b44d426c67e (D2: ArgoCD owns production: go-ahead sequence for railiance-apps-root, issue-core, policy-nexus and activity-core automation) by CUST-WP-0075; resolving that decision raises C-40 here. + + +## Prepare the concrete scheduling and runner proposal + +```task +id: ACTIVITY-WP-0042-T04 +status: done +priority: high +``` + +User requested investigation and forward progress on 2026-10-03. Reviewed the +existing definition, bounded-operation registry, platform recovery entry points, +credential route and actor declaration. Removed the stale ESO blocker after a +fresh metadata-only check. Concrete recommendation and staged acceptance: +`docs/platform-recovery-scheduling-plan.md`. + +Recommend scheduling-only ownership; platform retains the recovery runner and +policy. First proposed slice is one disabled/manual apps-pg recovery-evidence +operation, with offline contract tests before any live admission. T01 is still +an explicit role choice; this investigation does not record acceptance of it +or resolve the broader D2 decision. The actor ownership question is separate. + +## Agree the platform runner contract + +```task +id: ACTIVITY-WP-0042-T05 +status: wait +priority: high +depends_on: [ACTIVITY-WP-0042-T01] +blocking_reason: "Scheduling role acceptance and platform agreement on the fixed runner contract are needed before adapter admission." +``` + +Use the contract table in `docs/platform-recovery-scheduling-plan.md` for an +owner review: authority verification, exact target, implementation version, +durable run receipts, overlap/retry behavior, cleanup, resource ceilings and +credential scope. Existing restore CLIs are not an admitted unattended API. +This task records the dependency; no external acceptance or message is claimed. +T02 implements/adopts the agreed contract; T03 does not gate offline design.