railiance-clock/docs/operations-runbook.md

# Private authority operations — draft for owner acceptance

Prepared under RCLK-WP-0005-T06. This is a reviewable procedure, not deployment
authorization. S1 configuration and systemd units belong to railiance-infra's
`ansible/roles/railiance_clock`; signing custody and delivery belong to
railiance-platform under CMS v0.2. Telemetry/platform must accept alert routing,
response ownership and numerical budgets before operational handoff can finish.

## Observe and classify

Use the admitted private transport and owner service inventory. Record the source
commit, artifact digest, target, admission reference, epoch, key ID and policy ID.
Never collect private keys, trust credentials or bearer tokens into evidence.

| Observation | Meaning | Response |
| --- | --- | --- |
| `/healthz` unavailable | Service or transport unavailable | Check owner unit/transport state and sanitized logs; consumers refuse protected use |
| `/healthz` 200, `/readyz` 503 | Process alive but bounded time unavailable | Inspect health freshness, source/policy/boot binding and UTC continuity; do not declare a healthy authority from liveness |
| `/readyz` 200, client refuses | Local trust, timer, transport, sample or rollback gate failed | Preserve the refusal and existing rollback state; inspect sanitized reason and admitted coordinates |
| Unexpected epoch/key/policy | Continuity or trust changed | Stop adoption until owner admission; do not pin coordinates from an untrusted sample |
| Source loss, leap ambiguity or clock step | Bound/continuity cannot be assumed | Preserve fail-closed behavior and escalate to S1 owner; do not change corporate clocks |

Candidate budgets are documented in
[the implementation review](implementation-review-2026-09-15.md). They are not
an accepted SLO. Proposed alert inputs are readiness loss, stale/unusable host
health, sample RTT/width/age refusal, trust expiry, continuity latch, and unexpected
epoch changes. Telemetry must select collection, thresholds, routing, suppression
and acknowledgement policy. This repo does not claim those alerts are deployed.

## Restore source health or recover a continuity break

1. S1 owner diagnoses the admitted daemon, effective source and health exporter.
   Keep the authority unavailable while the bound is unsupported. Source recovery
   can restore readiness only if continuity remains valid.
2. A latched continuity failure requires a new authority process/epoch and explicit
   client admission. Returning wall time to its old value does not clear the latch.
3. Before restarting, prepare the exact owner-source diff, pinned artifact/config,
   consumer impact and rollback scope; obtain the applicable native authorization.
   Apply through owner automation. Do not reset or step host clocks to clear an
   application failure.
4. Verify host quality and both probes, then obtain independently authenticated
   key/epoch/policy coordinates through the admitted owner path. Verify a fresh
   signed exchange on a supported platform before consumer activation.
5. Preserve the client's persisted rollback floor. Missing/corrupt state or an
   older epoch mapping requires reviewed recovery; deleting state is not recovery.
   Record recovery approval and the disposition of the previous floor.

## Rotate or revoke signing trust

1. Run `warden plan` and route discovery before credential work. Platform owns
   the exact custody path, delivery and revocation action. Prepare a concrete
   rollout and affected-client inventory without reading secret values into logs.
2. For rotation, deliver the admitted key through owner automation and restart as
   required by the authority, which loads its signing key at startup. Treat the
   resulting epoch change as a new admission gate.
3. Refresh client trust out of band with exact authority/environment/key/epoch/
   policy bindings and a boot-bound deadline. The maximum online trust session is
   15 minutes; neither a sample nor restarting the client renews that deadline.
   A running client refuses when its trust file changes and must be reacquired.
4. Verify old signer, revoked trust, stale admission and wrong binding refusals,
   alongside one newly admitted positive sample. Local fixture tests prove these
   code paths only; durable distribution and revocation visibility require native
   evidence from platform and the consumer owner.
5. For suspected compromise, stop protected use and revoke through the owners.
   Code rollback or an old public key is not a new trust admission. The owners
   must define and prove how every affected client learns revocation before any
   visibility deadline is claimed.

## Roll back code or configuration

Use a reviewed, pinned previous artifact/config through the S1 delivery path;
retain exact apply/readback and no-op convergence receipts. A service restart
creates a new epoch. Do not restore old trust deadlines, old health snapshots,
private-key backups or a lower rollback floor as part of code rollback. Re-run
the readiness, new-admission and negative checks before protected use resumes.

## Required rehearsal and handoff evidence

RAIL-HO-WP-0013 and RCLK-WP-0005-T05 retain disposable VM outage/recovery,
reboot, configuration drift, rebuild, rollback and convergence evidence. Record
the test target, exact source/artifact/config hashes, preconditions, expected and
actual refusals, recovery and cleanup. RCLK-WP-0004-T02 retains measured platform
and interval thresholds; T03 retains consumer identity/expiry/CAS negatives.

T06 can close only after named telemetry/platform recipients accept alert and
response ownership, measured budgets, custody/recovery procedures and the native
receipts. Single-host failure and compromise remain explicit limitations. No
backup authority, offline holdover or automatic trust renewal is promised.