Prepare platform recovery scheduling contract and refresh blockers
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

Verify the repaired ESO lane, separate actor ownership from scheduling, and define the first bounded recovery slice and owner contract. Keep role acceptance and live operation admission explicit.

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a100de-2e1c-7e23-9ae9-2da0e0499411
This commit is contained in:
tegwick 2026-10-03 11:52:48 +02:00
parent e5ddd9ef5d
commit 9e231b984d
2 changed files with 156 additions and 5 deletions

View file

@ -0,0 +1,110 @@
# Platform recovery scheduling — reviewed proposal
Reviewed 2026-10-03 for ACTIVITY-WP-0042. This is a proposed delivery contract,
not admission of a new operation or permission to run it.
## Recommended decision
Accept **scheduling ownership only** for activity-core: Temporal triggers,
bounded dispatch, run correlation, missed-run visibility, and normalized result
evidence. Railiance Platform retains recovery implementations, acceptance
validators, cadence/freshness policy, destinations, retention, and credential
custody. Existing ACT-ADR-007 operations remain unchanged.
Approve offline design and tests of the invocation contract now. Admit each
new mutating operation separately after its owner supplies the contract and
ACT-ADR-007 is amended or superseded. No new generic shell executor, restore
schedule, retention action, or production rollout follows from role acceptance.
## What the investigation established
- The current definition `activity-definitions/daily-cnpg-option-a-backup.md`
selects three logical Nextcloud targets at 02:30 UTC. It does not produce
Scaleway physical recovery or Forgejo archive recovery evidence. Its old
host kubeconfig names and RPO wording are not proof of current cluster
routing or an accepted guarantee; review those before reusing the wiring.
- `src/activity_core/bounded_operations.py` permits exactly three operations.
CNPG backup has one Temporal attempt because no durable remote idempotency
receipt is available. New restore dispatch must not inherit retry defaults.
- The stale-token blocker was resolved under RPF-WP-0045. Read-only inspection
on 2026-10-03 confirmed node IP 92.205.62.239, ClusterSecretStore
`openbao-activity-core`: Ready=True/Valid, and ExternalSecret
`activity-core/actcore-backup-offsite`: Ready=True/SecretSynced, last refresh
2026-10-03T09:29:02Z. No Secret object or value was read. This verifies the
existing Nextcloud lane, not permission or credentials for a new restore.
- `warden route show railiance-backup-offsite-lane --json` names
railiance-platform as credential owner and reports workload resolution
unknown. Recovery escrow must stay outside the ordinary backup worker.
- Platform's `scripts/verify_scaleway_primary_restore.py` supports apps-pg
and forgejo-db, checks the cluster/destination, restores into a scratch
namespace, validates database contents, and attempts cleanup. Its CLI has
no dry-run option, approval-reference input, or durable idempotency key.
Do not register it directly as an unattended bounded operation.
- Platform's `scripts/restore_forgejo_offsite_locally.py` validates a transfer
receipt and supports full/essentials profiles. Its archive/decryption input
and local container execution need a separately scoped owner contract.
- RPF-WP-0038 retains the September 15 prohibition on backup expiration and
scheduled secondary cutover pending caller/inventory/quota gates. It is not
lifted by this proposal. RPF-WP-0036 retains cadence and evidence acceptance.
- Ops Warden's `registry/flex-auth/production_registry_snapshot.json` calls
`atm-backup-daily` an "Example nightly automation actor" and associates it
with workload `backup-daily`, but records resolution as absent. That is not
evidence that activity-core owns or uses it. T03 remains independent.
Platform message `0638c13b-cf6f-4342-9714-dfb1b6c67fc7` already consents to
the scheduling-only boundary and commits to supplying the credential route once
the operation contract exists. It specifies per-run approval as the interim;
there is no standing grant. This proposal resolves its obsolete token-expiry
prerequisite and makes the remaining contract explicit.
## First delivery slice
Start with a single manually triggered, disabled-by-default **apps-pg physical
recovery evidence run** against Scaleway, using a platform-owned isolated
runner. This is a proposal for the first target, not an expansion of the
existing logical-backup job. Add recurring triggers only after one admitted
run and cleanup receipt have passed. Forgejo archive tiers and retention
remain later slices with separate input/receipt contracts.
Before coding a production adapter, the platform owner must supply:
| Contract item | Required evidence |
| --- | --- |
| Fixed operation and implementation | Owner-reviewed version/digest; no caller-provided command or import path |
| Targets | Exact cluster identity, source apps-pg, scratch namespace policy; production never a restore destination |
| Admission | Explicit dry-run semantics and per-run authority checked by the executing enforcement point; an approval reference alone is insufficient |
| Time and capacity | Agreed timeout and CPU/memory/storage ceilings including cleanup; no new numbers inferred from historical tests |
| Credentials | Scoped workload route and recovery custody; no borrowed host-admin kubeconfig or escrow added to activity-core |
| Replay and overlap | Stable run/operation ID, durable owner receipt lookup and single-flight behavior; one dispatch attempt until retry safety is proven |
| Cleanup | Confirmed scratch removal; timeout or ambiguous result requires owner reconciliation before another mutation |
| Receipt | Schema/version, run ID, implementation revision, target identity, start/finish times, terminal result, validation and cleanup booleans, authoritative receipt hash/reference |
The scheduler retains only allowlisted receipt fields. Raw subprocess output,
SQL/application contents, archive locations, credentials and provider payloads
stay out of State Hub evidence. Failed, missing, stale and unverified receipts
must remain distinguishable. A scheduled fire or successful upload is not a
successful restore.
## Implementation and verification sequence
1. Founder accepts the scheduling-only boundary (T01). This resolves only
ACTIVITY-WP-0042's role choice, not the other decisions bundled in D2.
2. Agree the fixed runner contract above with railiance-platform (T05).
3. Build offline adapter/admission tests within T02. Cover unknown operation,
target drift, absent/expired/wrong-scope authority, duplicate dispatch,
timeouts, malformed/mismatched/stale receipts, failed cleanup and secret
canaries in child output. Use a fake owner runner; no live restore required.
4. Review the ADR change and exact credential/evidence contract before
registering the operation. Keep definitions disabled pending release.
5. Platform owner performs an approved isolated canary. Record evidence and
cleanup, then agree cadence/freshness/retention policy and rollout separately.
6. Enable the agreed Temporal schedule through the governed release path
(ACTIVITY-WP-0041), and verify missed-run and failed-evidence reporting.
## Independent identity question
Do not assign `atm-backup-daily` to activity-core from its name. Obtain an
issuance/deployment record identifying the actual execution unit, or a founder
ruling that it is an unused example to retire. Scheduling development does not
need to claim this historical actor. No owner message has been sent by this
review; the contract and identity questions are recorded as workplan tasks.