coulomb/railiance-clock
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Harden authority sampling and health failures; prepare operations handoff

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e6f0-a6f6-7b93-9124-55ef584930e2
This commit is contained in:
tegwick 2026-09-28 10:08:52 +02:00
parent a6db15d202
commit c976c9c277
8 changed files with 372 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

17
docs/closeout-review-2026-09-28.md
View file

@ -70,3 +70,20 @@ reference tests, not native infrastructure acceptance. No host clock, credential
service, consumer configuration or owner repository was changed. Inbox messages
are processed by incorporating their rulings here; no additional owner request
is needed to restate existing gates.
## Local hardening follow-up
After the initial closeout, regression tests exposed the sample-finalization
continuity gap and malformed-health exceptions. The authority now checks the
exact send timestamp and health age before signing, latches detected steps, and
rejects invalid health shapes and noninteger/negative measurements. The collector
rejects malformed policy/helper data and replaces old health with an explicit
unusable record on collection failure.
The expanded local suite passes 78 tests and 4 subtests, including late forward
and backward steps, health expiry during preparation, stable HTTP health refusals,
body framing, connection-capacity recovery, collector failures, signer rotation
and revoked admission. The [draft operations runbook](operations-runbook.md)
prepares 0005-T06's handoff. All native/owner gates and workplan statuses above
remain unchanged. No live authority or credential was changed.

88
docs/operations-runbook.md Normal file
View file

@ -0,0 +1,88 @@
# Private authority operations — draft for owner acceptance
Prepared under RCLK-WP-0005-T06. This is a reviewable procedure, not deployment
authorization. S1 configuration and systemd units belong to railiance-infra's
`ansible/roles/railiance_clock`; signing custody and delivery belong to
railiance-platform under CMS v0.2. Telemetry/platform must accept alert routing,
response ownership and numerical budgets before operational handoff can finish.
## Observe and classify
Use the admitted private transport and owner service inventory. Record the source
commit, artifact digest, target, admission reference, epoch, key ID and policy ID.
Never collect private keys, trust credentials or bearer tokens into evidence.
| Observation | Meaning | Response |
| --- | --- | --- |
| `/healthz` unavailable | Service or transport unavailable | Check owner unit/transport state and sanitized logs; consumers refuse protected use |
| `/healthz` 200, `/readyz` 503 | Process alive but bounded time unavailable | Inspect health freshness, source/policy/boot binding and UTC continuity; do not declare a healthy authority from liveness |
| `/readyz` 200, client refuses | Local trust, timer, transport, sample or rollback gate failed | Preserve the refusal and existing rollback state; inspect sanitized reason and admitted coordinates |
| Unexpected epoch/key/policy | Continuity or trust changed | Stop adoption until owner admission; do not pin coordinates from an untrusted sample |
| Source loss, leap ambiguity or clock step | Bound/continuity cannot be assumed | Preserve fail-closed behavior and escalate to S1 owner; do not change corporate clocks |
Candidate budgets are documented in
[the implementation review](implementation-review-2026-09-15.md). They are not
an accepted SLO. Proposed alert inputs are readiness loss, stale/unusable host
health, sample RTT/width/age refusal, trust expiry, continuity latch, and unexpected
epoch changes. Telemetry must select collection, thresholds, routing, suppression
and acknowledgement policy. This repo does not claim those alerts are deployed.
## Restore source health or recover a continuity break
1. S1 owner diagnoses the admitted daemon, effective source and health exporter.
Keep the authority unavailable while the bound is unsupported. Source recovery
can restore readiness only if continuity remains valid.
2. A latched continuity failure requires a new authority process/epoch and explicit
client admission. Returning wall time to its old value does not clear the latch.
3. Before restarting, prepare the exact owner-source diff, pinned artifact/config,
consumer impact and rollback scope; obtain the applicable native authorization.
Apply through owner automation. Do not reset or step host clocks to clear an
application failure.
4. Verify host quality and both probes, then obtain independently authenticated
key/epoch/policy coordinates through the admitted owner path. Verify a fresh
signed exchange on a supported platform before consumer activation.
5. Preserve the client's persisted rollback floor. Missing/corrupt state or an
older epoch mapping requires reviewed recovery; deleting state is not recovery.
Record recovery approval and the disposition of the previous floor.
## Rotate or revoke signing trust
1. Run `warden plan` and route discovery before credential work. Platform owns
the exact custody path, delivery and revocation action. Prepare a concrete
rollout and affected-client inventory without reading secret values into logs.
2. For rotation, deliver the admitted key through owner automation and restart as
required by the authority, which loads its signing key at startup. Treat the
resulting epoch change as a new admission gate.
3. Refresh client trust out of band with exact authority/environment/key/epoch/
policy bindings and a boot-bound deadline. The maximum online trust session is
15 minutes; neither a sample nor restarting the client renews that deadline.
A running client refuses when its trust file changes and must be reacquired.
4. Verify old signer, revoked trust, stale admission and wrong binding refusals,
alongside one newly admitted positive sample. Local fixture tests prove these
code paths only; durable distribution and revocation visibility require native
evidence from platform and the consumer owner.
5. For suspected compromise, stop protected use and revoke through the owners.
Code rollback or an old public key is not a new trust admission. The owners
must define and prove how every affected client learns revocation before any
visibility deadline is claimed.
## Roll back code or configuration
Use a reviewed, pinned previous artifact/config through the S1 delivery path;
retain exact apply/readback and no-op convergence receipts. A service restart
creates a new epoch. Do not restore old trust deadlines, old health snapshots,
private-key backups or a lower rollback floor as part of code rollback. Re-run
the readiness, new-admission and negative checks before protected use resumes.
## Required rehearsal and handoff evidence
RAIL-HO-WP-0013 and RCLK-WP-0005-T05 retain disposable VM outage/recovery,
reboot, configuration drift, rebuild, rollback and convergence evidence. Record
the test target, exact source/artifact/config hashes, preconditions, expected and
actual refusals, recovery and cleanup. RCLK-WP-0004-T02 retains measured platform
and interval thresholds; T03 retains consumer identity/expiry/CAS negatives.
T06 can close only after named telemetry/platform recipients accept alert and
response ownership, measured budgets, custody/recovery procedures and the native
receipts. Single-host failure and compromise remain explicit limitations. No
backup authority, offline holdover or automatic trust renewal is promised.