railiance-platform/history/2026-09-06-telemetry-scaleway-recovery.md

Telemetry Scaleway recovery — 2026-09-06

The user authorized the next telemetry backup/recovery step. The private service now has a verified essentials archive on Scaleway and an isolated application restore. RAPP-TELEMETRY-WP-0001-T03 can close for private installation/recovery proof. This does not grant public production admission or close RPF-WP-0036.

Accepted evidence

• 3,748,952 bytes encrypted with the established Railiance age recipient.
• Prefix platform-pg/application-archives/telemetry/ in the existing Scaleway railiance-platform-pg-backup bucket, separate from Barman/Forgejo. Exact version 1788728086280183 was downloaded and SHA-256 matched the source ciphertext.
• Preserves full Grafana data/plugins, Alertmanager persistent snapshots, ConfigMaps, Alertmanager configuration and a source Git archive. Omits disposable Prometheus TSDB; production rebuilds its seven-day metrics history. No telemetry upload to Nextcloud occurred.
• Grafana and Alertmanager replicas were fenced for 7.53 seconds for a consistent volume copy, then both rollout checks passed. This is the replica-fence window, not a measured endpoint-availability SLA. Final production health is 12/12 scrape targets up.
• The fetched copy was decrypted with existing governed escrow in a contained attended session. Same-digest Grafana and Alertmanager started on an internal local Docker network, no published ports, with alert delivery disabled.
• SQLite integrity passed; native admin login worked; anonymous and forged-header requests were denied; 20 dashboards, two datasources and the exact inert Alertmanager silence were recovered.
• Temporary containers, networks and plaintext were removed. The inert production silence was expired and verified. The encrypted Scaleway recovery point remains.

Problems resolved during the drill

The first content probe assumed Grafana's legacy dashboard table was authoritative. Grafana 13 migrated dashboards to unified storage; API acceptance is the correct check, and the legacy table count of zero was not evidence of loss. Next, Docker's internal network did not provide the assumed host port bindings, so the probe now runs inside that network with no published ports. Grafana required its production emptyDir search mount to be reconstructed as writable temporary storage. Finally, its archived provisioning ConfigMaps needed to be mounted and its search index allowed to rebuild. The final unmodified run of the corrected tool passed in about 21 seconds with images cached; do not call this a full disaster-recovery RTO. All failed attempts were isolated and cleaned up; no production recovery rollback was performed. Fixed diagnostic classifications contain no secret log output.

The credential catalog id backup-object-storage is unresolved, as the existing CCR-2026-0012 already records. Transfer used that CCR's established databases Secret delivery within the platform owner tool; no new namespace grant or workload key copy was created. Decryption used the established attended operator route, and each session self-revoked. No credential values entered receipts.

Remaining production work

Bind daily capture/upload and missed-run reporting to a durable, scoped executor; these tools establish an attended repeatable path, not an automatic backup SLA. Keep independent recovery-key/unseal availability explicit: reading current OpenBao during this drill is not a total-cluster-loss escrow proof. Existing provider retention remains owned by reef-storage/resource-control; no lifecycle or deletion policy was changed. Nextcloud remains essentials-only, with no TSDB.

Operator SSO, actual signal delivery/receipt, an outside-node watchdog and public production admission remain under T04. Package evidence is in ../rapp-telemetry/evidence/live/2026-09-06-isolated-restore.json and procedures in ../rapp-telemetry/docs/recovery.md.