Validate OpenBao snapshot evidence and record assurance closure gates
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883
This commit is contained in:
parent
a867ec269a
commit
31386c8e63
7 changed files with 115 additions and 3 deletions
|
|
@ -10,6 +10,11 @@
|
||||||
"signal": "forgejo-db.restore",
|
"signal": "forgejo-db.restore",
|
||||||
"path": "docs/evidence/forgejo-scaleway-restore-2026-09-06.json",
|
"path": "docs/evidence/forgejo-scaleway-restore-2026-09-06.json",
|
||||||
"sha256": "071a732318a55a8ca152d90b7a4cd70644728f94ce2753ab73d1c94ba2a26114"
|
"sha256": "071a732318a55a8ca152d90b7a4cd70644728f94ce2753ab73d1c94ba2a26114"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"signal": "openbao.snapshot",
|
||||||
|
"path": "reviews/WARDEN-WP-0027-T02-DRILL-20260822-01-openbao-snapshot-receipt.json",
|
||||||
|
"sha256": "20e7f5daada384937fb5c1cde8366cb472d552006581a6966f336a6e39836e6f"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -129,3 +129,11 @@ overwrite bug. The restore tool explicitly accepts both forms, with verified
|
||||||
hash/decryption flags. Existing historical receipts are unchanged. These producer
|
hash/decryption flags. Existing historical receipts are unchanged. These producer
|
||||||
fixes enable future dated archive evidence; automatic archive adapters, fresh
|
fixes enable future dated archive evidence; automatic archive adapters, fresh
|
||||||
end-to-end receipts and recurring execution are still pending.
|
end-to-end receipts and recurring execution are still pending.
|
||||||
|
|
||||||
|
The OpenBao snapshot adapter also accepts the reviewed, hash-pinned receipt in
|
||||||
|
`reviews/`. It requires the expected source cluster identity, encrypted off-host
|
||||||
|
custody, verified hashes and creation time. That time is a conservative age
|
||||||
|
anchor for snapshot creation, not proof of a later restore or renewed custody.
|
||||||
|
The August 22 receipt evaluates stale under the 36-hour budget. Updating the
|
||||||
|
index or reading the file cannot renew it; an isolated OpenBao restore remains
|
||||||
|
a separate obligation.
|
||||||
|
|
|
||||||
37
history/2026-09-06-WP-0036-closure-gates.md
Normal file
37
history/2026-09-06-WP-0036-closure-gates.md
Normal file
|
|
@ -0,0 +1,37 @@
|
||||||
|
# WP-0036 closure review — 2026-09-06
|
||||||
|
|
||||||
|
Result: not eligible for completion. T01/T02/T05/T07 are done; T03/T04/T06
|
||||||
|
retain unmet acceptance criteria. No task criteria were relaxed or moved into
|
||||||
|
new plans to create an apparent closure.
|
||||||
|
|
||||||
|
| Task | Verified local result | Remaining acceptance |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| T03 | Native apps-pg/forgejo-db adapters; archive receipt producers; encrypted off-host OpenBao snapshot adapter | Fresh isolated recovery for all supported services, independently available recovery custody, approved installed cadence with execution receipts and operator delivery |
|
||||||
|
| T04 | Bounded metadata capture and local validation | Q2 receiving contract, named recipient, controlled failure delivery and missing-emission detection |
|
||||||
|
| T06 | Exact compatibility inventory and dated retention through October 5 | Scoped legacy alias repair and generated brief matching source; retain compatibility until owner acceptance or the recorded retention decision is reviewed |
|
||||||
|
|
||||||
|
New source checks:
|
||||||
|
|
||||||
|
- `railiance-telemetry/README.md` explicitly says seeded, no implementation;
|
||||||
|
its only local workplan is proposed RTELE-WP-0001. A receiving contract/runtime
|
||||||
|
is not available in that checkout. S3 must not build a replacement Q2 plane
|
||||||
|
merely to satisfy this plan. Owner response is needed for any newer service.
|
||||||
|
- `rapp-postgres/docs/evidence/backup-restore-20260813T111651Z-remote.json`
|
||||||
|
explicitly identifies same-node scratch MinIO, not the governed off-host
|
||||||
|
target. Its successful drill cannot satisfy the current production recovery
|
||||||
|
acceptance criterion.
|
||||||
|
- The August 22 OpenBao snapshot receipt is encrypted, hash-verified and copied
|
||||||
|
off-host. Added its exact SHA-256 to the recovery index and a narrow adapter
|
||||||
|
that validates source identity and custody flags. It reports the original
|
||||||
|
creation time and therefore stale, never healthy by rereading. It does not
|
||||||
|
establish isolated restore or current independent quorum access.
|
||||||
|
|
||||||
|
The prepared requests in `docs/platform-ownership-handoffs.md` remain ready for
|
||||||
|
telemetry, repo-manager/State Hub and compatibility owners. Authorization to
|
||||||
|
send them was requested separately; no message was sent during this review.
|
||||||
|
Sending a request alone would not fulfill the receiving owner's acceptance.
|
||||||
|
|
||||||
|
Next closure sequence: obtain the Q2 contract and accepted execution/recipient
|
||||||
|
binding; run fresh owner-authorized recovery and cadence proofs; verify failure
|
||||||
|
and missing-emission delivery; reconcile aliases and regenerate orientation.
|
||||||
|
Existing WP-0015 outage exercises remain separate, with their own prerequisites.
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
"""Hash-pinned native recovery receipts, with original completion timestamps."""
|
"""Hash-pinned native recovery receipts, with original completion timestamps."""
|
||||||
import hashlib
|
import hashlib
|
||||||
import json
|
import json
|
||||||
|
import re
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from service_assurance import timestamp
|
from service_assurance import timestamp
|
||||||
|
|
||||||
|
|
@ -14,17 +15,39 @@ def recovery_signals(now, root=ROOT):
|
||||||
signals = {}
|
signals = {}
|
||||||
for entry in index['receipts']:
|
for entry in index['receipts']:
|
||||||
signal = entry['signal']
|
signal = entry['signal']
|
||||||
if signal in signals or signal not in ('apps-pg.restore', 'forgejo-db.restore'):
|
if signal in signals or signal not in ('apps-pg.restore', 'forgejo-db.restore', 'openbao.snapshot'):
|
||||||
raise ValueError('unexpected recovery signal')
|
raise ValueError('unexpected recovery signal')
|
||||||
sample = {'result': 'unavailable', 'observed_at': now.isoformat()}
|
sample = {'result': 'unavailable', 'observed_at': now.isoformat()}
|
||||||
try:
|
try:
|
||||||
path = (root / entry['path']).resolve()
|
path = (root / entry['path']).resolve()
|
||||||
if not path.is_relative_to((root / 'docs/evidence').resolve()):
|
allowed = [root / 'docs/evidence']
|
||||||
|
if signal == 'openbao.snapshot':
|
||||||
|
allowed.append(root / 'reviews')
|
||||||
|
if not any(path.is_relative_to(directory.resolve()) for directory in allowed):
|
||||||
raise ValueError('receipt outside evidence directory')
|
raise ValueError('receipt outside evidence directory')
|
||||||
raw = path.read_bytes()
|
raw = path.read_bytes()
|
||||||
if hashlib.sha256(raw).hexdigest() != entry['sha256']:
|
if hashlib.sha256(raw).hexdigest() != entry['sha256']:
|
||||||
raise ValueError('receipt drift')
|
raise ValueError('receipt drift')
|
||||||
receipt = json.loads(raw)
|
receipt = json.loads(raw)
|
||||||
|
if signal == 'openbao.snapshot':
|
||||||
|
required = ('snapshot_created', 'source_initialized', 'source_unsealed',
|
||||||
|
'snapshot_encrypted', 'encrypted_copy_off_host',
|
||||||
|
'encryption_verified', 'hash_verified', 'no_secret_material_recorded')
|
||||||
|
if (receipt.get('receipt_version') != 1
|
||||||
|
or receipt.get('source_cluster') != 'railiance01'
|
||||||
|
or receipt.get('source_namespace') != 'openbao'
|
||||||
|
or receipt.get('cluster_id') != 'fd28df5d-98ec-57dd-42ec-9b3e4f4e53bf'
|
||||||
|
or not all(receipt.get(key) is True for key in required)
|
||||||
|
or not receipt.get('encrypted_location_ref', '').startswith('offhost-custody:')):
|
||||||
|
raise ValueError('snapshot not accepted')
|
||||||
|
for key in ('snapshot_sha256', 'encrypted_snapshot_sha256'):
|
||||||
|
value = receipt.get(key, '')
|
||||||
|
if not re.fullmatch(r'sha256:[0-9a-f]{64}', value):
|
||||||
|
raise ValueError('snapshot hash missing')
|
||||||
|
if timestamp(receipt['created_at']) > now:
|
||||||
|
raise ValueError('future snapshot')
|
||||||
|
signals[signal] = {'result': 'pass', 'observed_at': receipt['created_at']}
|
||||||
|
continue
|
||||||
cell = signal.removesuffix('.restore')
|
cell = signal.removesuffix('.restore')
|
||||||
if (receipt['schema'] != 'platform.scaleway-primary-restore.v1'
|
if (receipt['schema'] != 'platform.scaleway-primary-restore.v1'
|
||||||
or receipt['primary_destination'] != f's3://railiance-platform-pg-backup/platform-pg/{cell}/'
|
or receipt['primary_destination'] != f's3://railiance-platform-pg-backup/platform-pg/{cell}/'
|
||||||
|
|
|
||||||
|
|
@ -41,3 +41,32 @@ def test_invalid_receipt_is_unavailable(tmp_path, change):
|
||||||
(tmp_path / 'assurance').mkdir()
|
(tmp_path / 'assurance').mkdir()
|
||||||
(tmp_path / 'assurance/recovery-evidence.json').write_text(json.dumps(index))
|
(tmp_path / 'assurance/recovery-evidence.json').write_text(json.dumps(index))
|
||||||
assert recovery_signals(NOW, tmp_path)['apps-pg.restore']['result'] == 'unavailable'
|
assert recovery_signals(NOW, tmp_path)['apps-pg.restore']['result'] == 'unavailable'
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_is_stale_and_never_substitutes_for_restore():
|
||||||
|
signals = recovery_signals(NOW)
|
||||||
|
assert signals['openbao.snapshot']['observed_at'] == '2026-08-22T22:29:21Z'
|
||||||
|
assert 'openbao.restore' not in signals
|
||||||
|
contract = {'cluster_uid': 'test', 'capture_max_age_seconds': 900,
|
||||||
|
'signals': {'openbao.snapshot': {'owner': 'platform', 'max_age_seconds': 129600}}}
|
||||||
|
result = evaluate(contract, {'schema': 'railiance-platform.observation.v1',
|
||||||
|
'cluster_uid': 'test', 'captured_at': NOW.isoformat(),
|
||||||
|
'signals': {'openbao.snapshot': signals['openbao.snapshot']}}, NOW)
|
||||||
|
assert result['signals']['openbao.snapshot']['state'] == 'stale'
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('key,value', [('encrypted_copy_off_host', False),
|
||||||
|
('hash_verified', False), ('cluster_id', 'other'), ('snapshot_sha256', 'invalid'),
|
||||||
|
('created_at', '2027-01-01T00:00:00Z')])
|
||||||
|
def test_snapshot_rejects_unverified_or_wrong_scope(tmp_path, key, value):
|
||||||
|
index = json.loads((ROOT / 'assurance/recovery-evidence.json').read_text())
|
||||||
|
entry = next(e for e in index['receipts'] if e['signal'] == 'openbao.snapshot')
|
||||||
|
receipt = json.loads((ROOT / entry['path']).read_text())
|
||||||
|
receipt[key] = value
|
||||||
|
path = tmp_path / entry['path']; path.parent.mkdir(parents=True)
|
||||||
|
path.write_text(json.dumps(receipt))
|
||||||
|
entry['sha256'] = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||||
|
(tmp_path / 'assurance').mkdir()
|
||||||
|
index['receipts'] = [entry]
|
||||||
|
(tmp_path / 'assurance/recovery-evidence.json').write_text(json.dumps(index))
|
||||||
|
assert recovery_signals(NOW, tmp_path)['openbao.snapshot']['result'] == 'unavailable'
|
||||||
|
|
|
||||||
|
|
@ -148,7 +148,7 @@ class CollectorTests(unittest.TestCase):
|
||||||
with patch.object(self.collector, 'query', side_effect=query), patch.object(self.collector, 'admission', return_value=baseline):
|
with patch.object(self.collector, 'query', side_effect=query), patch.object(self.collector, 'admission', return_value=baseline):
|
||||||
observation = self.collector.capture()
|
observation = self.collector.capture()
|
||||||
for name, sample in observation['signals'].items():
|
for name, sample in observation['signals'].items():
|
||||||
if name not in ('apps-pg.restore', 'forgejo-db.restore'):
|
if name not in ('apps-pg.restore', 'forgejo-db.restore', 'openbao.snapshot'):
|
||||||
self.assertEqual(sample['result'], 'unavailable')
|
self.assertEqual(sample['result'], 'unavailable')
|
||||||
# Recorded recovery evidence is independent of failed live status reads.
|
# Recorded recovery evidence is independent of failed live status reads.
|
||||||
self.assertEqual(observation['signals']['apps-pg.restore']['result'], 'pass')
|
self.assertEqual(observation['signals']['apps-pg.restore']['result'], 'pass')
|
||||||
|
|
|
||||||
|
|
@ -291,3 +291,13 @@ automatic recovery adapters pending reviewed fresh archive receipts and adapter
|
||||||
implementation. Validation and residual gates are recorded in
|
implementation. Validation and residual gates are recorded in
|
||||||
`history/2026-09-06-archive-receipt-review.md`; T03 remains waiting on its wider
|
`history/2026-09-06-archive-receipt-review.md`; T03 remains waiting on its wider
|
||||||
cadence/custody/acceptance gates.
|
cadence/custody/acceptance gates.
|
||||||
|
|
||||||
|
## Closure review — 2026-09-06
|
||||||
|
|
||||||
|
User requested completion. Source review still finds T03/T04/T06 acceptance
|
||||||
|
unmet; see `history/2026-09-06-WP-0036-closure-gates.md`. Telemetry's checkout has
|
||||||
|
no receiver implementation, and the older platform-pg scratch MinIO drill does
|
||||||
|
not establish production off-host recovery. Added the verified August 22
|
||||||
|
OpenBao snapshot to the hash-pinned evidence adapter: it reports stale using
|
||||||
|
its original creation time, separately from the still-missing isolated restore.
|
||||||
|
No criteria were weakened, external acceptance inferred or live window reused.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue