Verify telemetry essentials recovery from versioned Scaleway archive

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883

history/2026-09-06-telemetry-scaleway-recovery.md

@@ -0,0 +1,63 @@
+# Telemetry Scaleway recovery — 2026-09-06
+
+The user authorized the next telemetry backup/recovery step. The private service
+now has a verified essentials archive on Scaleway and an isolated application
+restore. RAPP-TELEMETRY-WP-0001-T03 can close for private installation/recovery
+proof. This does not grant public production admission or close RPF-WP-0036.
+
+## Accepted evidence
+
+- 3,748,952 bytes encrypted with the established Railiance age recipient.
+- Prefix platform-pg/application-archives/telemetry/ in the existing Scaleway
+  railiance-platform-pg-backup bucket, separate from Barman/Forgejo. Exact version
+  1788728086280183 was downloaded and SHA-256 matched the source ciphertext.
+- Preserves full Grafana data/plugins, Alertmanager persistent snapshots,
+  ConfigMaps, Alertmanager configuration and a source Git archive. Omits disposable
+  Prometheus TSDB; production rebuilds its seven-day metrics history. No telemetry
+  upload to Nextcloud occurred.
+- Grafana and Alertmanager replicas were fenced for 7.53 seconds for a consistent
+  volume copy, then both rollout checks passed. This is the replica-fence window,
+  not a measured endpoint-availability SLA. Final production health is 12/12
+  scrape targets up.
+- The fetched copy was decrypted with existing governed escrow in a contained
+  attended session. Same-digest Grafana and Alertmanager started on an internal
+  local Docker network, no published ports, with alert delivery disabled.
+- SQLite integrity passed; native admin login worked; anonymous and forged-header
+  requests were denied; 20 dashboards, two datasources and the exact inert
+  Alertmanager silence were recovered.
+- Temporary containers, networks and plaintext were removed. The inert production
+  silence was expired and verified. The encrypted Scaleway recovery point remains.
+
+## Problems resolved during the drill
+
+The first content probe assumed Grafana's legacy dashboard table was authoritative.
+Grafana 13 migrated dashboards to unified storage; API acceptance is the correct
+check, and the legacy table count of zero was not evidence of loss. Next, Docker's
+internal network did not provide the assumed host port bindings, so the probe now
+runs inside that network with no published ports. Grafana required its production
+emptyDir search mount to be reconstructed as writable temporary storage. Finally,
+its archived provisioning ConfigMaps needed to be mounted and its search index
+allowed to rebuild. The final unmodified run of the corrected tool passed in
+about 21 seconds with images cached; do not call this a full disaster-recovery RTO.
+All failed attempts were isolated and cleaned up; no production recovery rollback
+was performed. Fixed diagnostic classifications contain no secret log output.
+
+The credential catalog id backup-object-storage is unresolved, as the existing
+CCR-2026-0012 already records. Transfer used that CCR's established databases
+Secret delivery within the platform owner tool; no new namespace grant or
+workload key copy was created. Decryption used the established attended operator
+route, and each session self-revoked. No credential values entered receipts.
+
+## Remaining production work
+
+Bind daily capture/upload and missed-run reporting to a durable, scoped executor;
+these tools establish an attended repeatable path, not an automatic backup SLA.
+Keep independent recovery-key/unseal availability explicit: reading current
+OpenBao during this drill is not a total-cluster-loss escrow proof. Existing
+provider retention remains owned by reef-storage/resource-control; no lifecycle
+or deletion policy was changed. Nextcloud remains essentials-only, with no TSDB.
+
+Operator SSO, actual signal delivery/receipt, an outside-node watchdog and public
+production admission remain under T04. Package evidence is in
+../rapp-telemetry/evidence/live/2026-09-06-isolated-restore.json and procedures in
+../rapp-telemetry/docs/recovery.md.