Harden authority sampling and health failures; prepare operations handoff

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e6f0-a6f6-7b93-9124-55ef584930e2
This commit is contained in:
tegwick 2026-09-28 10:08:52 +02:00
parent a6db15d202
commit c976c9c277
8 changed files with 372 additions and 11 deletions

View file

@ -70,3 +70,20 @@ reference tests, not native infrastructure acceptance. No host clock, credential
service, consumer configuration or owner repository was changed. Inbox messages
are processed by incorporating their rulings here; no additional owner request
is needed to restate existing gates.
## Local hardening follow-up
After the initial closeout, regression tests exposed the sample-finalization
continuity gap and malformed-health exceptions. The authority now checks the
exact send timestamp and health age before signing, latches detected steps, and
rejects invalid health shapes and noninteger/negative measurements. The collector
rejects malformed policy/helper data and replaces old health with an explicit
unusable record on collection failure.
The expanded local suite passes 78 tests and 4 subtests, including late forward
and backward steps, health expiry during preparation, stable HTTP health refusals,
body framing, connection-capacity recovery, collector failures, signer rotation
and revoked admission. The [draft operations runbook](operations-runbook.md)
prepares 0005-T06's handoff. All native/owner gates and workplan statuses above
remain unchanged. No live authority or credential was changed.

View file

@ -0,0 +1,88 @@
# Private authority operations — draft for owner acceptance
Prepared under RCLK-WP-0005-T06. This is a reviewable procedure, not deployment
authorization. S1 configuration and systemd units belong to railiance-infra's
`ansible/roles/railiance_clock`; signing custody and delivery belong to
railiance-platform under CMS v0.2. Telemetry/platform must accept alert routing,
response ownership and numerical budgets before operational handoff can finish.
## Observe and classify
Use the admitted private transport and owner service inventory. Record the source
commit, artifact digest, target, admission reference, epoch, key ID and policy ID.
Never collect private keys, trust credentials or bearer tokens into evidence.
| Observation | Meaning | Response |
| --- | --- | --- |
| `/healthz` unavailable | Service or transport unavailable | Check owner unit/transport state and sanitized logs; consumers refuse protected use |
| `/healthz` 200, `/readyz` 503 | Process alive but bounded time unavailable | Inspect health freshness, source/policy/boot binding and UTC continuity; do not declare a healthy authority from liveness |
| `/readyz` 200, client refuses | Local trust, timer, transport, sample or rollback gate failed | Preserve the refusal and existing rollback state; inspect sanitized reason and admitted coordinates |
| Unexpected epoch/key/policy | Continuity or trust changed | Stop adoption until owner admission; do not pin coordinates from an untrusted sample |
| Source loss, leap ambiguity or clock step | Bound/continuity cannot be assumed | Preserve fail-closed behavior and escalate to S1 owner; do not change corporate clocks |
Candidate budgets are documented in
[the implementation review](implementation-review-2026-09-15.md). They are not
an accepted SLO. Proposed alert inputs are readiness loss, stale/unusable host
health, sample RTT/width/age refusal, trust expiry, continuity latch, and unexpected
epoch changes. Telemetry must select collection, thresholds, routing, suppression
and acknowledgement policy. This repo does not claim those alerts are deployed.
## Restore source health or recover a continuity break
1. S1 owner diagnoses the admitted daemon, effective source and health exporter.
Keep the authority unavailable while the bound is unsupported. Source recovery
can restore readiness only if continuity remains valid.
2. A latched continuity failure requires a new authority process/epoch and explicit
client admission. Returning wall time to its old value does not clear the latch.
3. Before restarting, prepare the exact owner-source diff, pinned artifact/config,
consumer impact and rollback scope; obtain the applicable native authorization.
Apply through owner automation. Do not reset or step host clocks to clear an
application failure.
4. Verify host quality and both probes, then obtain independently authenticated
key/epoch/policy coordinates through the admitted owner path. Verify a fresh
signed exchange on a supported platform before consumer activation.
5. Preserve the client's persisted rollback floor. Missing/corrupt state or an
older epoch mapping requires reviewed recovery; deleting state is not recovery.
Record recovery approval and the disposition of the previous floor.
## Rotate or revoke signing trust
1. Run `warden plan` and route discovery before credential work. Platform owns
the exact custody path, delivery and revocation action. Prepare a concrete
rollout and affected-client inventory without reading secret values into logs.
2. For rotation, deliver the admitted key through owner automation and restart as
required by the authority, which loads its signing key at startup. Treat the
resulting epoch change as a new admission gate.
3. Refresh client trust out of band with exact authority/environment/key/epoch/
policy bindings and a boot-bound deadline. The maximum online trust session is
15 minutes; neither a sample nor restarting the client renews that deadline.
A running client refuses when its trust file changes and must be reacquired.
4. Verify old signer, revoked trust, stale admission and wrong binding refusals,
alongside one newly admitted positive sample. Local fixture tests prove these
code paths only; durable distribution and revocation visibility require native
evidence from platform and the consumer owner.
5. For suspected compromise, stop protected use and revoke through the owners.
Code rollback or an old public key is not a new trust admission. The owners
must define and prove how every affected client learns revocation before any
visibility deadline is claimed.
## Roll back code or configuration
Use a reviewed, pinned previous artifact/config through the S1 delivery path;
retain exact apply/readback and no-op convergence receipts. A service restart
creates a new epoch. Do not restore old trust deadlines, old health snapshots,
private-key backups or a lower rollback floor as part of code rollback. Re-run
the readiness, new-admission and negative checks before protected use resumes.
## Required rehearsal and handoff evidence
RAIL-HO-WP-0013 and RCLK-WP-0005-T05 retain disposable VM outage/recovery,
reboot, configuration drift, rebuild, rollback and convergence evidence. Record
the test target, exact source/artifact/config hashes, preconditions, expected and
actual refusals, recovery and cleanup. RCLK-WP-0004-T02 retains measured platform
and interval thresholds; T03 retains consumer identity/expiry/CAS negatives.
T06 can close only after named telemetry/platform recipients accept alert and
response ownership, measured budgets, custody/recovery procedures and the native
receipts. Single-host failure and compromise remain explicit limitations. No
backup authority, offline holdover or automatic trust renewal is promised.

View file

@ -30,7 +30,7 @@ class Authority:
self.failed = False
self.lock = threading.Lock()
def health(self):
def health(self, *, wall_ns=None, monotonic_ns=None):
path = Path(self.config['health_file'])
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW)
with os.fdopen(fd, 'rb') as stream:
@ -38,15 +38,23 @@ class Authority:
if not stat.S_ISREG(info.st_mode) or info.st_uid != self.config.get('health_owner_uid', 0) or info.st_mode & 0o022:
raise ClockError('untrusted host health file')
health = strict_json(stream.read(8193))
if not isinstance(health, dict):
raise ClockError('invalid host health object')
if health.get('quality') != 'bounded' or health.get('boot_id') != self.boot_id or health.get('policy_id') != self.config['policy_id']:
raise ClockError('host quality unusable')
age = self.monotonic() - health['observed_monotonic_ns']
observed = bounded(health.get('observed_monotonic_ns'))
reported_error = bounded(health.get('error_ns'))
if wall_ns is None:
wall_ns = self.wall()
if monotonic_ns is None:
monotonic_ns = self.monotonic()
age = monotonic_ns - observed
if not 0 <= age < self.config['max_health_age_ns']:
raise ClockError('host health stale')
error = bounded(health['error_ns'] + (age * self.config['host_drift_ppm'] + 999999) // 1000000)
error = bounded(reported_error + (age * self.config['host_drift_ppm'] + 999999) // 1000000)
if error > self.config['max_error_ns']:
raise ClockError('host error bound exceeded')
if abs((self.wall() - self.start_wall) - (self.monotonic() - self.start_mono)) > self.config['max_continuity_error_ns']:
if abs((wall_ns - self.start_wall) - (monotonic_ns - self.start_mono)) > self.config['max_continuity_error_ns']:
self.failed = True
if self.failed:
raise ClockError('authority continuity lost; new epoch admission required')
@ -61,9 +69,13 @@ class Authority:
received = self.wall()
error = self.health()
sent = self.wall()
sent_mono = self.monotonic()
if sent < received:
self.failed = True
raise ClockError('authority clock moved backwards')
# Validate the exact timestamp being signed, including health aging
# during sample preparation. A later wall read could hide a transient step.
error = max(error, self.health(wall_ns=sent, monotonic_ns=sent_mono))
payload = dict(request, epoch=self.epoch, policy_id=self.config['policy_id'],
receive_unix_ns=str(received), send_unix_ns=str(sent), error_ns=str(error),
max_age_ns=str(self.config['max_sample_age_ns']), quality='bounded')

View file

@ -20,6 +20,15 @@ def properties(raw):
def assess(policy, kernel, clock, upstream):
# Explicitly conditional on trusted Linux discipline and admitted NTP source;
# this is not independent UTC consensus or Byzantine-source detection.
if not all(isinstance(value, dict) for value in (policy, kernel, clock, upstream)):
raise ClockError('invalid health inputs')
for name in ('state', 'status', 'maxerror_us', 'esterror_us', 'tolerance_scaled_ppm'):
bounded(kernel.get(name))
for name in ('measurement_margin_ns', 'max_error_ns'):
bounded(policy.get(name))
sources = policy.get('upstreams')
if not isinstance(sources, list) or not sources or not all(isinstance(s, str) and s for s in sources):
raise ClockError('invalid upstream policy')
if clock.get('NTPSynchronized') != 'yes' or clock.get('NTP') != 'yes':
raise ClockError('host not synchronized')
if kernel['state'] != 0 or kernel['status'] & (0x10 | 0x20 | 0x40):
@ -27,11 +36,8 @@ def assess(policy, kernel, clock, upstream):
if upstream.get('ServerName') not in policy['upstreams']:
raise ClockError('unadmitted upstream')
message = upstream.get('NTPMessage','')
if not re.search(r'\bLeap=0,', message) or not re.search(r'\bIgnored=no,', message):
if not isinstance(message, str) or not re.search(r'\bLeap=0,', message) or not re.search(r'\bIgnored=no,', message):
raise ClockError('unusable NTP source observation')
for name in ('maxerror_us','esterror_us','tolerance_scaled_ppm'):
if type(kernel[name]) is not int or kernel[name] < 0:
raise ClockError('invalid kernel error accounting')
error = bounded(max(kernel['maxerror_us'],kernel['esterror_us']) * 1000 + policy['measurement_margin_ns'])
if error > policy['max_error_ns']:
raise ClockError('UTC error budget exceeded')
@ -40,20 +46,25 @@ def assess(policy, kernel, clock, upstream):
def collect(policy):
started = time.monotonic_ns()
try:
if policy.get('status') != 'admitted':
if not isinstance(policy, dict) or policy.get('status') != 'admitted':
raise ClockError('UTC policy not admitted')
bounded(policy.get('host_drift_ppm'))
if not isinstance(policy.get('kernel_helper'), str) or not policy['kernel_helper']:
raise ClockError('invalid kernel helper')
if not isinstance(policy.get('policy_id'), str) or not policy['policy_id']:
raise ClockError('invalid policy identity')
kernel = json.loads(command([policy['kernel_helper']]))
clock = properties(command(['timedatectl','show','--property=NTPSynchronized','--property=NTP']))
upstream = properties(command(['timedatectl','show-timesync','--all']))
error = assess(policy,kernel,clock,upstream)
elapsed = time.monotonic_ns() - started
elapsed = bounded(time.monotonic_ns() - started)
error += (elapsed * policy['host_drift_ppm'] + 999999) // 1000000
if error > policy['max_error_ns']:
raise ClockError('health collection exceeded error budget')
return {'quality':'bounded','boot_id':Path('/proc/sys/kernel/random/boot_id').read_text().strip(),
'policy_id':policy['policy_id'],'observed_monotonic_ns':started,'error_ns':error,
'model':'linux-disciplined-kernel-maxerror-v1','upstream':upstream['ServerName']}
except (OSError, ValueError, KeyError, subprocess.TimeoutExpired):
except (OSError, ValueError, KeyError, TypeError, subprocess.TimeoutExpired):
return {'quality':'unusable','reason':'host_quality_unavailable'}
def write_health(path, value):

View file

@ -63,3 +63,136 @@ def test_health_model_refusals():
for changed in [{**k,'state':5},{**k,'status':16},{**k,'maxerror_us':200_000},{**k,'maxerror_us':-1}]:
with pytest.raises(ClockError):assess(p,changed,c,u)
with pytest.raises(ClockError):assess(p,k,c,{**u,'ServerName':'unadmitted'})
@pytest.mark.parametrize('step', [100_000_000, -100_000_000])
def test_step_at_final_timestamp_latches_refusal(authority, step):
a, _, clocks = authority
original = clocks['wall']
# Receive and initial health are normal; only the timestamp to sign steps.
readings = iter([original, original, original + step])
a.wall = lambda: next(readings, original)
with pytest.raises(ClockError):
a.sample(json.dumps(request()).encode())
assert a.failed
# Returning to the old mapping must not restore the epoch's usability.
with pytest.raises(ClockError):
a.health()
def test_health_expires_during_sample_preparation(authority):
a, _, clocks = authority
original = a.health
calls = 0
def advancing_health(**kwargs):
nonlocal calls
error = original(**kwargs)
calls += 1
if calls == 1:
clocks['mono'] += a.config['max_health_age_ns']
clocks['wall'] += a.config['max_health_age_ns']
return error
a.health = advancing_health
with pytest.raises(ClockError, match='stale'):
a.sample(json.dumps(request()).encode())
@pytest.fixture
def http_authority(authority):
a, hp, clocks = authority
server = Server(('127.0.0.1', 0), a)
thread = threading.Thread(target=server.serve_forever)
thread.start()
try:
yield server, hp
finally:
server.shutdown(); server.server_close(); thread.join()
@pytest.mark.parametrize('bad', [None, [], True, 1, 'invalid',
{'error_ns': -1}, {'error_ns': True}, {'error_ns': '1000'},
{'observed_monotonic_ns': None}, {'observed_monotonic_ns': False}])
def test_malformed_health_is_stable_http_refusal(http_authority, bad):
server, hp = http_authority
if isinstance(bad, dict):
valid = json.loads(hp.read_text()); valid.update(bad); bad = valid
hp.write_text(json.dumps(bad))
base = f'http://127.0.0.1:{server.server_port}'
requests = [base + '/readyz', Request(base + '/v1/time-samples',
data=json.dumps(request()).encode(), headers={'Content-Type': 'application/json'})]
for req in requests:
with pytest.raises(HTTPError) as exc:
urlopen(req, timeout=2)
assert exc.value.code == 503
assert exc.value.read() == b'{"status":"unusable"}'
with urlopen(base + '/healthz', timeout=2) as response:
assert response.status == 200
@pytest.mark.parametrize('headers,body', [
('Content-Length: 2\r\nContent-Length: 2\r\n', b'{}'),
('Content-Length: 2\r\nTransfer-Encoding: chunked\r\n', b'{}'),
('Content-Length: 2\r\nContent-Encoding: gzip\r\n', b'{}'),
('Content-Length: 1025\r\n', b''),
('Content-Length: -1\r\n', b''),
('Content-Length: 5\r\n', b'{}'),
])
def test_http_framing_refused(http_authority, headers, body):
import socket
server, _ = http_authority
with socket.create_connection(server.server_address, timeout=2) as conn:
conn.sendall(('POST /v1/time-samples HTTP/1.0\r\n'
'Content-Type: application/json\r\n' + headers + '\r\n').encode() + body)
conn.shutdown(socket.SHUT_WR)
with conn.makefile('rb') as response:
raw = response.read()
assert raw.startswith(b'HTTP/1.0 503')
assert raw.endswith(b'{"status":"unusable"}')
def test_connection_limit_and_capacity_recovery(authority, monkeypatch):
import socket
from railiance_clock.authority import Handler
a, _, _ = authority
ready = threading.Event()
released = threading.Event()
count = 0
lock = threading.Lock()
original_setup = Handler.setup
original_worker = Server.process_request_thread
def setup(handler):
nonlocal count
original_setup(handler)
with lock:
count += 1
if count == 8:
ready.set()
def worker(server, request, address):
nonlocal count
try:
original_worker(server, request, address)
finally:
with lock:
count -= 1
if count == 0:
released.set()
monkeypatch.setattr(Handler, 'setup', setup)
monkeypatch.setattr(Server, 'process_request_thread', worker)
server = Server(('127.0.0.1', 0), a)
thread = threading.Thread(target=server.serve_forever)
thread.start()
clients = []
try:
for _ in range(8):
clients.append(socket.create_connection(server.server_address, timeout=2))
assert ready.wait(2)
with socket.create_connection(server.server_address, timeout=2) as extra:
assert extra.recv(1) == b''
for conn in clients:
conn.close()
assert released.wait(2)
with urlopen(f'http://127.0.0.1:{server.server_port}/readyz', timeout=2) as response:
assert response.status == 200
finally:
for conn in clients:
conn.close()
server.shutdown(); server.server_close(); thread.join()

View file

@ -112,3 +112,28 @@ def test_restart_preserves_floor(setup):
payload.update(receive_unix_ns='50000000000',send_unix_ns='50001000000')
return sign(payload,key,'test-key')
with pytest.raises(ClockError):Clock(FileTrust(path),timer=timer,transport=older).read()
def test_rotation_requires_fresh_client_and_matching_signer(setup):
path, key, timer, transport = setup
old = Clock(FileTrust(path), timer=timer, transport=transport)
old.read()
replacement = ec.generate_private_key(ec.SECP256R1())
data = json.loads(path.read_text())
data['kid'] = 'rotated-key'
data['public_key_pem'] = replacement.public_key().public_bytes(
serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo).decode()
path.write_text(json.dumps(data))
with pytest.raises(ClockError):
old.read()
# Loading new trust must still refuse the old signer.
with pytest.raises(ClockError):
Clock(FileTrust(path), timer=timer, transport=transport).read()
def rotated(request):
payload = verify(transport(request), key.public_key(), 'test-key')
return sign(payload, replacement, 'rotated-key')
assert Clock(FileTrust(path), timer=timer, transport=rotated).read().authority_id == 'railiance01'
data['status'] = 'revoked'
path.write_text(json.dumps(data))
with pytest.raises(ClockError):
FileTrust(path)

65
tests/test_health.py Normal file
View file

@ -0,0 +1,65 @@
import json
import subprocess
import pytest
from railiance_clock import health
@pytest.fixture
def policy():
return {'status': 'admitted', 'policy_id': 'test', 'kernel_helper': '/test/helper',
'host_drift_ppm': 1000, 'upstreams': ['ntp.test'],
'measurement_margin_ns': 1000, 'max_error_ns': 100_000_000}
@pytest.fixture
def commands(monkeypatch):
values = {
'/test/helper': json.dumps({'state': 0, 'status': 0, 'maxerror_us': 1000,
'esterror_us': 100, 'tolerance_scaled_ppm': 32768000}),
'show': 'NTP=yes\nNTPSynchronized=yes\n',
'show-timesync': 'ServerName=ntp.test\nNTPMessage={ Leap=0, Ignored=no, }\n',
}
def command(argv):
value = values[argv[0] if len(argv) == 1 else argv[1]]
if isinstance(value, Exception):
raise value
return value
monkeypatch.setattr(health, 'command', command)
return values
def test_collection_and_replacement_on_failure(tmp_path, policy, commands):
path = tmp_path / 'health.json'
good = health.collect(policy)
assert good['quality'] == 'bounded'
health.write_health(path, good)
commands['show'] = 'NTP=yes\nNTPSynchronized=no\n'
bad = health.collect(policy)
health.write_health(path, bad)
assert json.loads(path.read_text()) == {'quality': 'unusable', 'reason': 'host_quality_unavailable'}
assert path.stat().st_mode & 0o777 == 0o644
assert list(tmp_path.iterdir()) == [path]
@pytest.mark.parametrize('bad', [None, [], True, 'bad', {'status': 'admitted'},
{'host_drift_ppm': -1}, {'host_drift_ppm': True}, {'upstreams': 'ntp.test'},
{'measurement_margin_ns': -1}, {'max_error_ns': False}, {'kernel_helper': []}])
def test_bad_policy_refuses(policy, commands, bad):
if isinstance(bad, dict):
if bad == {'status': 'admitted'}:
policy = bad
else:
policy.update(bad)
else:
policy = bad
assert health.collect(policy)['quality'] == 'unusable'
@pytest.mark.parametrize('bad', ['null', '[]', 'true', '{',
'{"state":false,"status":0}', '{"state":0,"status":"0"}',
OSError('private details'), subprocess.TimeoutExpired('private command', 3)])
def test_helper_failures_are_sanitized(policy, commands, bad):
commands['/test/helper'] = bad
assert health.collect(policy) == {'quality': 'unusable', 'reason': 'host_quality_unavailable'}

View file

@ -102,6 +102,12 @@ trust bootstrap reviewed, least-privilege and no-redirect tests, config/source
hashes and exact deployment/rollback plan. Fail if dynamic trust resolution needs
usable Railiance time before the first sample with no independent bootstrap path.
Local follow-up 2026-09-28: authority sampling now checks continuity and health
age at the exact signed timestamp; malformed health refuses explicitly. Local
HTTP framing, capacity recovery, collection failure and trust-rotation tests
pass. These support C05/C13–C15, but do not close native admission or trust
delivery.
## Integrate the clock prerequisite into the new-estate bootstrap path
```task
@ -167,6 +173,10 @@ Keep all unresolved work as live records before finishing this plan. RCLK-WP-000
may adopt a protected consumer only after this infrastructure gate AND its own
explicit consumer review/negative tests pass. A deployed endpoint is not a grant.
Local follow-up 2026-09-28: the [operations runbook](../docs/operations-runbook.md)
now supplies reviewable alert, recovery, rotation and rollback procedures.
Telemetry/platform acceptance and native evidence remain the T06 blockers.
### Production deployment evidence — 2026-09-15
The operator approved CCR-2026-0028 for platform and Clock ownership and then