activity-core/docs/platform-recovery-scheduling-plan.md

Platform recovery scheduling — reviewed proposal

Reviewed 2026-10-03 for ACTIVITY-WP-0042. This is a proposed delivery contract, not admission of a new operation or permission to run it.

Recommended decision

Accept scheduling ownership only for activity-core: Temporal triggers, bounded dispatch, run correlation, missed-run visibility, and normalized result evidence. Railiance Platform retains recovery implementations, acceptance validators, cadence/freshness policy, destinations, retention, and credential custody. Existing ACT-ADR-007 operations remain unchanged.

Approve offline design and tests of the invocation contract now. Admit each new mutating operation separately after its owner supplies the contract and ACT-ADR-007 is amended or superseded. No new generic shell executor, restore schedule, retention action, or production rollout follows from role acceptance.

What the investigation established

• The current definition activity-definitions/daily-cnpg-option-a-backup.md selects three logical Nextcloud targets at 02:30 UTC. It does not produce Scaleway physical recovery or Forgejo archive recovery evidence. Its old host kubeconfig names and RPO wording are not proof of current cluster routing or an accepted guarantee; review those before reusing the wiring.
• src/activity_core/bounded_operations.py permits exactly three operations. CNPG backup has one Temporal attempt because no durable remote idempotency receipt is available. New restore dispatch must not inherit retry defaults.
• The stale-token blocker was resolved under RPF-WP-0045. Read-only inspection on 2026-10-03 confirmed node IP 92.205.62.239, ClusterSecretStore openbao-activity-core: Ready=True/Valid, and ExternalSecret activity-core/actcore-backup-offsite: Ready=True/SecretSynced, last refresh 2026-10-03T09:29:02Z. No Secret object or value was read. This verifies the existing Nextcloud lane, not permission or credentials for a new restore.
• warden route show railiance-backup-offsite-lane --json names railiance-platform as credential owner and reports workload resolution unknown. Recovery escrow must stay outside the ordinary backup worker.
• Platform's scripts/verify_scaleway_primary_restore.py supports apps-pg and forgejo-db, checks the cluster/destination, restores into a scratch namespace, validates database contents, and attempts cleanup. Its CLI has no dry-run option, approval-reference input, or durable idempotency key. Do not register it directly as an unattended bounded operation.
• Platform's scripts/restore_forgejo_offsite_locally.py validates a transfer receipt and supports full/essentials profiles. Its archive/decryption input and local container execution need a separately scoped owner contract.
• RPF-WP-0038 retains the September 15 prohibition on backup expiration and scheduled secondary cutover pending caller/inventory/quota gates. It is not lifted by this proposal. RPF-WP-0036 retains cadence and evidence acceptance.
• Ops Warden's registry/flex-auth/production_registry_snapshot.json calls atm-backup-daily an "Example nightly automation actor" and associates it with workload backup-daily, but records resolution as absent. That is not evidence that activity-core owns or uses it. T03 remains independent.

Platform message 0638c13b-cf6f-4342-9714-dfb1b6c67fc7 already consents to the scheduling-only boundary and commits to supplying the credential route once the operation contract exists. It specifies per-run approval as the interim; there is no standing grant. This proposal resolves its obsolete token-expiry prerequisite and makes the remaining contract explicit.

First delivery slice

Start with a single manually triggered, disabled-by-default apps-pg physical recovery evidence run against Scaleway, using a platform-owned isolated runner. This is a proposal for the first target, not an expansion of the existing logical-backup job. Add recurring triggers only after one admitted run and cleanup receipt have passed. Forgejo archive tiers and retention remain later slices with separate input/receipt contracts.

Before coding a production adapter, the platform owner must supply:

Contract item	Required evidence
Fixed operation and implementation	Owner-reviewed version/digest; no caller-provided command or import path
Targets	Exact cluster identity, source apps-pg, scratch namespace policy; production never a restore destination
Admission	Explicit dry-run semantics and per-run authority checked by the executing enforcement point; an approval reference alone is insufficient
Time and capacity	Agreed timeout and CPU/memory/storage ceilings including cleanup; no new numbers inferred from historical tests
Credentials	Scoped workload route and recovery custody; no borrowed host-admin kubeconfig or escrow added to activity-core
Replay and overlap	Stable run/operation ID, durable owner receipt lookup and single-flight behavior; one dispatch attempt until retry safety is proven
Cleanup	Confirmed scratch removal; timeout or ambiguous result requires owner reconciliation before another mutation
Receipt	Schema/version, run ID, implementation revision, target identity, start/finish times, terminal result, validation and cleanup booleans, authoritative receipt hash/reference

The scheduler retains only allowlisted receipt fields. Raw subprocess output, SQL/application contents, archive locations, credentials and provider payloads stay out of State Hub evidence. Failed, missing, stale and unverified receipts must remain distinguishable. A scheduled fire or successful upload is not a successful restore.

Implementation and verification sequence

1. Founder accepts the scheduling-only boundary (T01). This resolves only ACTIVITY-WP-0042's role choice, not the other decisions bundled in D2.
2. Agree the fixed runner contract above with railiance-platform (T05).
3. Build offline adapter/admission tests within T02. Cover unknown operation, target drift, absent/expired/wrong-scope authority, duplicate dispatch, timeouts, malformed/mismatched/stale receipts, failed cleanup and secret canaries in child output. Use a fake owner runner; no live restore required.
4. Review the ADR change and exact credential/evidence contract before registering the operation. Keep definitions disabled pending release.
5. Platform owner performs an approved isolated canary. Record evidence and cleanup, then agree cadence/freshness/retention policy and rollout separately.
6. Enable the agreed Temporal schedule through the governed release path (ACTIVITY-WP-0041), and verify missed-run and failed-evidence reporting.

Independent identity question

Do not assign atm-backup-daily to activity-core from its name. Obtain an issuance/deployment record identifying the actual execution unit, or a founder ruling that it is an unused example to retire. Scheduling development does not need to claim this historical actor. No owner message has been sent by this review; the contract and identity questions are recorded as workplan tasks.