railiance-clock/docs/operations-runbook.md

Private authority operations — draft for owner acceptance

Prepared under RCLK-WP-0005-T06. This is a reviewable procedure, not deployment authorization. S1 configuration and systemd units belong to railiance-infra's ansible/roles/railiance_clock; signing custody and delivery belong to railiance-platform under CMS v0.2. Telemetry/platform must accept alert routing, response ownership and numerical budgets before operational handoff can finish.

Observe and classify

Use the admitted private transport and owner service inventory. Record the source commit, artifact digest, target, admission reference, epoch, key ID and policy ID. Never collect private keys, trust credentials or bearer tokens into evidence.

Observation	Meaning	Response
/healthz unavailable	Service or transport unavailable	Check owner unit/transport state and sanitized logs; consumers refuse protected use
/healthz 200, /readyz 503	Process alive but bounded time unavailable	Inspect health freshness, source/policy/boot binding and UTC continuity; do not declare a healthy authority from liveness
/readyz 200, client refuses	Local trust, timer, transport, sample or rollback gate failed	Preserve the refusal and existing rollback state; inspect sanitized reason and admitted coordinates
Unexpected epoch/key/policy	Continuity or trust changed	Stop adoption until owner admission; do not pin coordinates from an untrusted sample
Source loss, leap ambiguity or clock step	Bound/continuity cannot be assumed	Preserve fail-closed behavior and escalate to S1 owner; do not change corporate clocks

Candidate budgets are documented in the implementation review. They are not an accepted SLO. Proposed alert inputs are readiness loss, stale/unusable host health, sample RTT/width/age refusal, trust expiry, continuity latch, and unexpected epoch changes. Telemetry must select collection, thresholds, routing, suppression and acknowledgement policy. This repo does not claim those alerts are deployed.

Restore source health or recover a continuity break

1. S1 owner diagnoses the admitted daemon, effective source and health exporter. Keep the authority unavailable while the bound is unsupported. Source recovery can restore readiness only if continuity remains valid.
2. A latched continuity failure requires a new authority process/epoch and explicit client admission. Returning wall time to its old value does not clear the latch.
3. Before restarting, prepare the exact owner-source diff, pinned artifact/config, consumer impact and rollback scope; obtain the applicable native authorization. Apply through owner automation. Do not reset or step host clocks to clear an application failure.
4. Verify host quality and both probes, then obtain independently authenticated key/epoch/policy coordinates through the admitted owner path. Verify a fresh signed exchange on a supported platform before consumer activation.
5. Preserve the client's persisted rollback floor. Missing/corrupt state or an older epoch mapping requires reviewed recovery; deleting state is not recovery. Record recovery approval and the disposition of the previous floor.

Rotate or revoke signing trust

1. Run warden plan and route discovery before credential work. Platform owns the exact custody path, delivery and revocation action. Prepare a concrete rollout and affected-client inventory without reading secret values into logs.
2. For rotation, deliver the admitted key through owner automation and restart as required by the authority, which loads its signing key at startup. Treat the resulting epoch change as a new admission gate.
3. Refresh client trust out of band with exact authority/environment/key/epoch/ policy bindings and a boot-bound deadline. The maximum online trust session is 15 minutes; neither a sample nor restarting the client renews that deadline. A running client refuses when its trust file changes and must be reacquired.
4. Verify old signer, revoked trust, stale admission and wrong binding refusals, alongside one newly admitted positive sample. Local fixture tests prove these code paths only; durable distribution and revocation visibility require native evidence from platform and the consumer owner.
5. For suspected compromise, stop protected use and revoke through the owners. Code rollback or an old public key is not a new trust admission. The owners must define and prove how every affected client learns revocation before any visibility deadline is claimed.

Roll back code or configuration

Use a reviewed, pinned previous artifact/config through the S1 delivery path; retain exact apply/readback and no-op convergence receipts. A service restart creates a new epoch. Do not restore old trust deadlines, old health snapshots, private-key backups or a lower rollback floor as part of code rollback. Re-run the readiness, new-admission and negative checks before protected use resumes.

Required rehearsal and handoff evidence

RAIL-HO-WP-0013 and RCLK-WP-0005-T05 retain disposable VM outage/recovery, reboot, configuration drift, rebuild, rollback and convergence evidence. Record the test target, exact source/artifact/config hashes, preconditions, expected and actual refusals, recovery and cleanup. RCLK-WP-0004-T02 retains measured platform and interval thresholds; T03 retains consumer identity/expiry/CAS negatives.

T06 can close only after named telemetry/platform recipients accept alert and response ownership, measured budgets, custody/recovery procedures and the native receipts. Single-host failure and compromise remain explicit limitations. No backup authority, offline holdover or automatic trust renewal is promised.