activity-core/docs/platform-recovery-scheduling-plan.md

# Platform recovery scheduling — reviewed proposal

Reviewed 2026-10-03 for ACTIVITY-WP-0042. This is a proposed delivery contract,
not admission of a new operation or permission to run it.

## Recommended decision

Accept **scheduling ownership only** for activity-core: Temporal triggers,
bounded dispatch, run correlation, missed-run visibility, and normalized result
evidence. Railiance Platform retains recovery implementations, acceptance
validators, cadence/freshness policy, destinations, retention, and credential
custody. Existing ACT-ADR-007 operations remain unchanged.

Approve offline design and tests of the invocation contract now. Admit each
new mutating operation separately after its owner supplies the contract and
ACT-ADR-007 is amended or superseded. No new generic shell executor, restore
schedule, retention action, or production rollout follows from role acceptance.

## What the investigation established

- The current definition `activity-definitions/daily-cnpg-option-a-backup.md`
  selects three logical Nextcloud targets at 02:30 UTC. It does not produce
  Scaleway physical recovery or Forgejo archive recovery evidence. Its old
  host kubeconfig names and RPO wording are not proof of current cluster
  routing or an accepted guarantee; review those before reusing the wiring.
- `src/activity_core/bounded_operations.py` permits exactly three operations.
  CNPG backup has one Temporal attempt because no durable remote idempotency
  receipt is available. New restore dispatch must not inherit retry defaults.
- The stale-token blocker was resolved under RPF-WP-0045. Read-only inspection
  on 2026-10-03 confirmed node IP 92.205.62.239, ClusterSecretStore
  `openbao-activity-core`: Ready=True/Valid, and ExternalSecret
  `activity-core/actcore-backup-offsite`: Ready=True/SecretSynced, last refresh
  2026-10-03T09:29:02Z. No Secret object or value was read. This verifies the
  existing Nextcloud lane, not permission or credentials for a new restore.
- `warden route show railiance-backup-offsite-lane --json` names
  railiance-platform as credential owner and reports workload resolution
  unknown. Recovery escrow must stay outside the ordinary backup worker.
- Platform's `scripts/verify_scaleway_primary_restore.py` supports apps-pg
  and forgejo-db, checks the cluster/destination, restores into a scratch
  namespace, validates database contents, and attempts cleanup. Its CLI has
  no dry-run option, approval-reference input, or durable idempotency key.
  Do not register it directly as an unattended bounded operation.
- Platform's `scripts/restore_forgejo_offsite_locally.py` validates a transfer
  receipt and supports full/essentials profiles. Its archive/decryption input
  and local container execution need a separately scoped owner contract.
- RPF-WP-0038 retains the September 15 prohibition on backup expiration and
  scheduled secondary cutover pending caller/inventory/quota gates. It is not
  lifted by this proposal. RPF-WP-0036 retains cadence and evidence acceptance.
- Ops Warden's `registry/flex-auth/production_registry_snapshot.json` calls
  `atm-backup-daily` an "Example nightly automation actor" and associates it
  with workload `backup-daily`, but records resolution as absent. That is not
  evidence that activity-core owns or uses it. T03 remains independent.

Platform message `0638c13b-cf6f-4342-9714-dfb1b6c67fc7` already consents to
the scheduling-only boundary and commits to supplying the credential route once
the operation contract exists. It specifies per-run approval as the interim;
there is no standing grant. This proposal resolves its obsolete token-expiry
prerequisite and makes the remaining contract explicit.

## First delivery slice

Start with a single manually triggered, disabled-by-default **apps-pg physical
recovery evidence run** against Scaleway, using a platform-owned isolated
runner. This is a proposal for the first target, not an expansion of the
existing logical-backup job. Add recurring triggers only after one admitted
run and cleanup receipt have passed. Forgejo archive tiers and retention
remain later slices with separate input/receipt contracts.

Before coding a production adapter, the platform owner must supply:

| Contract item | Required evidence |
| --- | --- |
| Fixed operation and implementation | Owner-reviewed version/digest; no caller-provided command or import path |
| Targets | Exact cluster identity, source apps-pg, scratch namespace policy; production never a restore destination |
| Admission | Explicit dry-run semantics and per-run authority checked by the executing enforcement point; an approval reference alone is insufficient |
| Time and capacity | Agreed timeout and CPU/memory/storage ceilings including cleanup; no new numbers inferred from historical tests |
| Credentials | Scoped workload route and recovery custody; no borrowed host-admin kubeconfig or escrow added to activity-core |
| Replay and overlap | Stable run/operation ID, durable owner receipt lookup and single-flight behavior; one dispatch attempt until retry safety is proven |
| Cleanup | Confirmed scratch removal; timeout or ambiguous result requires owner reconciliation before another mutation |
| Receipt | Schema/version, run ID, implementation revision, target identity, start/finish times, terminal result, validation and cleanup booleans, authoritative receipt hash/reference |

The scheduler retains only allowlisted receipt fields. Raw subprocess output,
SQL/application contents, archive locations, credentials and provider payloads
stay out of State Hub evidence. Failed, missing, stale and unverified receipts
must remain distinguishable. A scheduled fire or successful upload is not a
successful restore.

## Implementation and verification sequence

1. Founder accepts the scheduling-only boundary (T01). This resolves only
   ACTIVITY-WP-0042's role choice, not the other decisions bundled in D2.
2. Agree the fixed runner contract above with railiance-platform (T05).
3. Build offline adapter/admission tests within T02. Cover unknown operation,
   target drift, absent/expired/wrong-scope authority, duplicate dispatch,
   timeouts, malformed/mismatched/stale receipts, failed cleanup and secret
   canaries in child output. Use a fake owner runner; no live restore required.
4. Review the ADR change and exact credential/evidence contract before
   registering the operation. Keep definitions disabled pending release.
5. Platform owner performs an approved isolated canary. Record evidence and
   cleanup, then agree cadence/freshness/retention policy and rollout separately.
6. Enable the agreed Temporal schedule through the governed release path
   (ACTIVITY-WP-0041), and verify missed-run and failed-evidence reporting.

## Independent identity question

Do not assign `atm-backup-daily` to activity-core from its name. Obtain an
issuance/deployment record identifying the actual execution unit, or a founder
ruling that it is an unused example to retire. Scheduling development does not
need to claim this historical actor. No owner message has been sent by this
review; the contract and identity questions are recorded as workplan tasks.