Validate OpenBao snapshot evidence and record assurance closure gates
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883
This commit is contained in:
parent
a867ec269a
commit
31386c8e63
7 changed files with 115 additions and 3 deletions
|
|
@ -10,6 +10,11 @@
|
|||
"signal": "forgejo-db.restore",
|
||||
"path": "docs/evidence/forgejo-scaleway-restore-2026-09-06.json",
|
||||
"sha256": "071a732318a55a8ca152d90b7a4cd70644728f94ce2753ab73d1c94ba2a26114"
|
||||
},
|
||||
{
|
||||
"signal": "openbao.snapshot",
|
||||
"path": "reviews/WARDEN-WP-0027-T02-DRILL-20260822-01-openbao-snapshot-receipt.json",
|
||||
"sha256": "20e7f5daada384937fb5c1cde8366cb472d552006581a6966f336a6e39836e6f"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -129,3 +129,11 @@ overwrite bug. The restore tool explicitly accepts both forms, with verified
|
|||
hash/decryption flags. Existing historical receipts are unchanged. These producer
|
||||
fixes enable future dated archive evidence; automatic archive adapters, fresh
|
||||
end-to-end receipts and recurring execution are still pending.
|
||||
|
||||
The OpenBao snapshot adapter also accepts the reviewed, hash-pinned receipt in
|
||||
`reviews/`. It requires the expected source cluster identity, encrypted off-host
|
||||
custody, verified hashes and creation time. That time is a conservative age
|
||||
anchor for snapshot creation, not proof of a later restore or renewed custody.
|
||||
The August 22 receipt evaluates stale under the 36-hour budget. Updating the
|
||||
index or reading the file cannot renew it; an isolated OpenBao restore remains
|
||||
a separate obligation.
|
||||
|
|
|
|||
37
history/2026-09-06-WP-0036-closure-gates.md
Normal file
37
history/2026-09-06-WP-0036-closure-gates.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# WP-0036 closure review — 2026-09-06
|
||||
|
||||
Result: not eligible for completion. T01/T02/T05/T07 are done; T03/T04/T06
|
||||
retain unmet acceptance criteria. No task criteria were relaxed or moved into
|
||||
new plans to create an apparent closure.
|
||||
|
||||
| Task | Verified local result | Remaining acceptance |
|
||||
| --- | --- | --- |
|
||||
| T03 | Native apps-pg/forgejo-db adapters; archive receipt producers; encrypted off-host OpenBao snapshot adapter | Fresh isolated recovery for all supported services, independently available recovery custody, approved installed cadence with execution receipts and operator delivery |
|
||||
| T04 | Bounded metadata capture and local validation | Q2 receiving contract, named recipient, controlled failure delivery and missing-emission detection |
|
||||
| T06 | Exact compatibility inventory and dated retention through October 5 | Scoped legacy alias repair and generated brief matching source; retain compatibility until owner acceptance or the recorded retention decision is reviewed |
|
||||
|
||||
New source checks:
|
||||
|
||||
- `railiance-telemetry/README.md` explicitly says seeded, no implementation;
|
||||
its only local workplan is proposed RTELE-WP-0001. A receiving contract/runtime
|
||||
is not available in that checkout. S3 must not build a replacement Q2 plane
|
||||
merely to satisfy this plan. Owner response is needed for any newer service.
|
||||
- `rapp-postgres/docs/evidence/backup-restore-20260813T111651Z-remote.json`
|
||||
explicitly identifies same-node scratch MinIO, not the governed off-host
|
||||
target. Its successful drill cannot satisfy the current production recovery
|
||||
acceptance criterion.
|
||||
- The August 22 OpenBao snapshot receipt is encrypted, hash-verified and copied
|
||||
off-host. Added its exact SHA-256 to the recovery index and a narrow adapter
|
||||
that validates source identity and custody flags. It reports the original
|
||||
creation time and therefore stale, never healthy by rereading. It does not
|
||||
establish isolated restore or current independent quorum access.
|
||||
|
||||
The prepared requests in `docs/platform-ownership-handoffs.md` remain ready for
|
||||
telemetry, repo-manager/State Hub and compatibility owners. Authorization to
|
||||
send them was requested separately; no message was sent during this review.
|
||||
Sending a request alone would not fulfill the receiving owner's acceptance.
|
||||
|
||||
Next closure sequence: obtain the Q2 contract and accepted execution/recipient
|
||||
binding; run fresh owner-authorized recovery and cadence proofs; verify failure
|
||||
and missing-emission delivery; reconcile aliases and regenerate orientation.
|
||||
Existing WP-0015 outage exercises remain separate, with their own prerequisites.
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
"""Hash-pinned native recovery receipts, with original completion timestamps."""
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
from service_assurance import timestamp
|
||||
|
||||
|
|
@ -14,17 +15,39 @@ def recovery_signals(now, root=ROOT):
|
|||
signals = {}
|
||||
for entry in index['receipts']:
|
||||
signal = entry['signal']
|
||||
if signal in signals or signal not in ('apps-pg.restore', 'forgejo-db.restore'):
|
||||
if signal in signals or signal not in ('apps-pg.restore', 'forgejo-db.restore', 'openbao.snapshot'):
|
||||
raise ValueError('unexpected recovery signal')
|
||||
sample = {'result': 'unavailable', 'observed_at': now.isoformat()}
|
||||
try:
|
||||
path = (root / entry['path']).resolve()
|
||||
if not path.is_relative_to((root / 'docs/evidence').resolve()):
|
||||
allowed = [root / 'docs/evidence']
|
||||
if signal == 'openbao.snapshot':
|
||||
allowed.append(root / 'reviews')
|
||||
if not any(path.is_relative_to(directory.resolve()) for directory in allowed):
|
||||
raise ValueError('receipt outside evidence directory')
|
||||
raw = path.read_bytes()
|
||||
if hashlib.sha256(raw).hexdigest() != entry['sha256']:
|
||||
raise ValueError('receipt drift')
|
||||
receipt = json.loads(raw)
|
||||
if signal == 'openbao.snapshot':
|
||||
required = ('snapshot_created', 'source_initialized', 'source_unsealed',
|
||||
'snapshot_encrypted', 'encrypted_copy_off_host',
|
||||
'encryption_verified', 'hash_verified', 'no_secret_material_recorded')
|
||||
if (receipt.get('receipt_version') != 1
|
||||
or receipt.get('source_cluster') != 'railiance01'
|
||||
or receipt.get('source_namespace') != 'openbao'
|
||||
or receipt.get('cluster_id') != 'fd28df5d-98ec-57dd-42ec-9b3e4f4e53bf'
|
||||
or not all(receipt.get(key) is True for key in required)
|
||||
or not receipt.get('encrypted_location_ref', '').startswith('offhost-custody:')):
|
||||
raise ValueError('snapshot not accepted')
|
||||
for key in ('snapshot_sha256', 'encrypted_snapshot_sha256'):
|
||||
value = receipt.get(key, '')
|
||||
if not re.fullmatch(r'sha256:[0-9a-f]{64}', value):
|
||||
raise ValueError('snapshot hash missing')
|
||||
if timestamp(receipt['created_at']) > now:
|
||||
raise ValueError('future snapshot')
|
||||
signals[signal] = {'result': 'pass', 'observed_at': receipt['created_at']}
|
||||
continue
|
||||
cell = signal.removesuffix('.restore')
|
||||
if (receipt['schema'] != 'platform.scaleway-primary-restore.v1'
|
||||
or receipt['primary_destination'] != f's3://railiance-platform-pg-backup/platform-pg/{cell}/'
|
||||
|
|
|
|||
|
|
@ -41,3 +41,32 @@ def test_invalid_receipt_is_unavailable(tmp_path, change):
|
|||
(tmp_path / 'assurance').mkdir()
|
||||
(tmp_path / 'assurance/recovery-evidence.json').write_text(json.dumps(index))
|
||||
assert recovery_signals(NOW, tmp_path)['apps-pg.restore']['result'] == 'unavailable'
|
||||
|
||||
|
||||
def test_snapshot_is_stale_and_never_substitutes_for_restore():
|
||||
signals = recovery_signals(NOW)
|
||||
assert signals['openbao.snapshot']['observed_at'] == '2026-08-22T22:29:21Z'
|
||||
assert 'openbao.restore' not in signals
|
||||
contract = {'cluster_uid': 'test', 'capture_max_age_seconds': 900,
|
||||
'signals': {'openbao.snapshot': {'owner': 'platform', 'max_age_seconds': 129600}}}
|
||||
result = evaluate(contract, {'schema': 'railiance-platform.observation.v1',
|
||||
'cluster_uid': 'test', 'captured_at': NOW.isoformat(),
|
||||
'signals': {'openbao.snapshot': signals['openbao.snapshot']}}, NOW)
|
||||
assert result['signals']['openbao.snapshot']['state'] == 'stale'
|
||||
|
||||
|
||||
@pytest.mark.parametrize('key,value', [('encrypted_copy_off_host', False),
|
||||
('hash_verified', False), ('cluster_id', 'other'), ('snapshot_sha256', 'invalid'),
|
||||
('created_at', '2027-01-01T00:00:00Z')])
|
||||
def test_snapshot_rejects_unverified_or_wrong_scope(tmp_path, key, value):
|
||||
index = json.loads((ROOT / 'assurance/recovery-evidence.json').read_text())
|
||||
entry = next(e for e in index['receipts'] if e['signal'] == 'openbao.snapshot')
|
||||
receipt = json.loads((ROOT / entry['path']).read_text())
|
||||
receipt[key] = value
|
||||
path = tmp_path / entry['path']; path.parent.mkdir(parents=True)
|
||||
path.write_text(json.dumps(receipt))
|
||||
entry['sha256'] = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
(tmp_path / 'assurance').mkdir()
|
||||
index['receipts'] = [entry]
|
||||
(tmp_path / 'assurance/recovery-evidence.json').write_text(json.dumps(index))
|
||||
assert recovery_signals(NOW, tmp_path)['openbao.snapshot']['result'] == 'unavailable'
|
||||
|
|
|
|||
|
|
@ -148,7 +148,7 @@ class CollectorTests(unittest.TestCase):
|
|||
with patch.object(self.collector, 'query', side_effect=query), patch.object(self.collector, 'admission', return_value=baseline):
|
||||
observation = self.collector.capture()
|
||||
for name, sample in observation['signals'].items():
|
||||
if name not in ('apps-pg.restore', 'forgejo-db.restore'):
|
||||
if name not in ('apps-pg.restore', 'forgejo-db.restore', 'openbao.snapshot'):
|
||||
self.assertEqual(sample['result'], 'unavailable')
|
||||
# Recorded recovery evidence is independent of failed live status reads.
|
||||
self.assertEqual(observation['signals']['apps-pg.restore']['result'], 'pass')
|
||||
|
|
|
|||
|
|
@ -291,3 +291,13 @@ automatic recovery adapters pending reviewed fresh archive receipts and adapter
|
|||
implementation. Validation and residual gates are recorded in
|
||||
`history/2026-09-06-archive-receipt-review.md`; T03 remains waiting on its wider
|
||||
cadence/custody/acceptance gates.
|
||||
|
||||
## Closure review — 2026-09-06
|
||||
|
||||
User requested completion. Source review still finds T03/T04/T06 acceptance
|
||||
unmet; see `history/2026-09-06-WP-0036-closure-gates.md`. Telemetry's checkout has
|
||||
no receiver implementation, and the older platform-pg scratch MinIO drill does
|
||||
not establish production off-host recovery. Added the verified August 22
|
||||
OpenBao snapshot to the hash-pinned evidence adapter: it reports stale using
|
||||
its original creation time, separately from the still-missing isolated restore.
|
||||
No criteria were weakened, external acceptance inferred or live window reused.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue