From c976c9c277033e9fca384b9e9fcd5d79ff776d43 Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 28 Sep 2026 10:08:52 +0200 Subject: [PATCH] Harden authority sampling and health failures; prepare operations handoff Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a0e6f0-a6f6-7b93-9124-55ef584930e2 --- docs/closeout-review-2026-09-28.md | 17 +++ docs/operations-runbook.md | 88 ++++++++++++ src/railiance_clock/authority.py | 20 ++- src/railiance_clock/health.py | 25 +++- tests/test_authority.py | 133 ++++++++++++++++++ tests/test_clock.py | 25 ++++ tests/test_health.py | 65 +++++++++ .../RCLK-WP-0005-infrastructure-as-code.md | 10 ++ 8 files changed, 372 insertions(+), 11 deletions(-) create mode 100644 docs/operations-runbook.md create mode 100644 tests/test_health.py diff --git a/docs/closeout-review-2026-09-28.md b/docs/closeout-review-2026-09-28.md index bcf97eb..850fe96 100644 --- a/docs/closeout-review-2026-09-28.md +++ b/docs/closeout-review-2026-09-28.md @@ -70,3 +70,20 @@ reference tests, not native infrastructure acceptance. No host clock, credential service, consumer configuration or owner repository was changed. Inbox messages are processed by incorporating their rulings here; no additional owner request is needed to restate existing gates. + + +## Local hardening follow-up + +After the initial closeout, regression tests exposed the sample-finalization +continuity gap and malformed-health exceptions. The authority now checks the +exact send timestamp and health age before signing, latches detected steps, and +rejects invalid health shapes and noninteger/negative measurements. The collector +rejects malformed policy/helper data and replaces old health with an explicit +unusable record on collection failure. + +The expanded local suite passes 78 tests and 4 subtests, including late forward +and backward steps, health expiry during preparation, stable HTTP health refusals, +body framing, connection-capacity recovery, collector failures, signer rotation +and revoked admission. The [draft operations runbook](operations-runbook.md) +prepares 0005-T06's handoff. All native/owner gates and workplan statuses above +remain unchanged. No live authority or credential was changed. diff --git a/docs/operations-runbook.md b/docs/operations-runbook.md new file mode 100644 index 0000000..bddca57 --- /dev/null +++ b/docs/operations-runbook.md @@ -0,0 +1,88 @@ +# Private authority operations — draft for owner acceptance + +Prepared under RCLK-WP-0005-T06. This is a reviewable procedure, not deployment +authorization. S1 configuration and systemd units belong to railiance-infra's +`ansible/roles/railiance_clock`; signing custody and delivery belong to +railiance-platform under CMS v0.2. Telemetry/platform must accept alert routing, +response ownership and numerical budgets before operational handoff can finish. + +## Observe and classify + +Use the admitted private transport and owner service inventory. Record the source +commit, artifact digest, target, admission reference, epoch, key ID and policy ID. +Never collect private keys, trust credentials or bearer tokens into evidence. + +| Observation | Meaning | Response | +| --- | --- | --- | +| `/healthz` unavailable | Service or transport unavailable | Check owner unit/transport state and sanitized logs; consumers refuse protected use | +| `/healthz` 200, `/readyz` 503 | Process alive but bounded time unavailable | Inspect health freshness, source/policy/boot binding and UTC continuity; do not declare a healthy authority from liveness | +| `/readyz` 200, client refuses | Local trust, timer, transport, sample or rollback gate failed | Preserve the refusal and existing rollback state; inspect sanitized reason and admitted coordinates | +| Unexpected epoch/key/policy | Continuity or trust changed | Stop adoption until owner admission; do not pin coordinates from an untrusted sample | +| Source loss, leap ambiguity or clock step | Bound/continuity cannot be assumed | Preserve fail-closed behavior and escalate to S1 owner; do not change corporate clocks | + +Candidate budgets are documented in +[the implementation review](implementation-review-2026-09-15.md). They are not +an accepted SLO. Proposed alert inputs are readiness loss, stale/unusable host +health, sample RTT/width/age refusal, trust expiry, continuity latch, and unexpected +epoch changes. Telemetry must select collection, thresholds, routing, suppression +and acknowledgement policy. This repo does not claim those alerts are deployed. + +## Restore source health or recover a continuity break + +1. S1 owner diagnoses the admitted daemon, effective source and health exporter. + Keep the authority unavailable while the bound is unsupported. Source recovery + can restore readiness only if continuity remains valid. +2. A latched continuity failure requires a new authority process/epoch and explicit + client admission. Returning wall time to its old value does not clear the latch. +3. Before restarting, prepare the exact owner-source diff, pinned artifact/config, + consumer impact and rollback scope; obtain the applicable native authorization. + Apply through owner automation. Do not reset or step host clocks to clear an + application failure. +4. Verify host quality and both probes, then obtain independently authenticated + key/epoch/policy coordinates through the admitted owner path. Verify a fresh + signed exchange on a supported platform before consumer activation. +5. Preserve the client's persisted rollback floor. Missing/corrupt state or an + older epoch mapping requires reviewed recovery; deleting state is not recovery. + Record recovery approval and the disposition of the previous floor. + +## Rotate or revoke signing trust + +1. Run `warden plan` and route discovery before credential work. Platform owns + the exact custody path, delivery and revocation action. Prepare a concrete + rollout and affected-client inventory without reading secret values into logs. +2. For rotation, deliver the admitted key through owner automation and restart as + required by the authority, which loads its signing key at startup. Treat the + resulting epoch change as a new admission gate. +3. Refresh client trust out of band with exact authority/environment/key/epoch/ + policy bindings and a boot-bound deadline. The maximum online trust session is + 15 minutes; neither a sample nor restarting the client renews that deadline. + A running client refuses when its trust file changes and must be reacquired. +4. Verify old signer, revoked trust, stale admission and wrong binding refusals, + alongside one newly admitted positive sample. Local fixture tests prove these + code paths only; durable distribution and revocation visibility require native + evidence from platform and the consumer owner. +5. For suspected compromise, stop protected use and revoke through the owners. + Code rollback or an old public key is not a new trust admission. The owners + must define and prove how every affected client learns revocation before any + visibility deadline is claimed. + +## Roll back code or configuration + +Use a reviewed, pinned previous artifact/config through the S1 delivery path; +retain exact apply/readback and no-op convergence receipts. A service restart +creates a new epoch. Do not restore old trust deadlines, old health snapshots, +private-key backups or a lower rollback floor as part of code rollback. Re-run +the readiness, new-admission and negative checks before protected use resumes. + +## Required rehearsal and handoff evidence + +RAIL-HO-WP-0013 and RCLK-WP-0005-T05 retain disposable VM outage/recovery, +reboot, configuration drift, rebuild, rollback and convergence evidence. Record +the test target, exact source/artifact/config hashes, preconditions, expected and +actual refusals, recovery and cleanup. RCLK-WP-0004-T02 retains measured platform +and interval thresholds; T03 retains consumer identity/expiry/CAS negatives. + +T06 can close only after named telemetry/platform recipients accept alert and +response ownership, measured budgets, custody/recovery procedures and the native +receipts. Single-host failure and compromise remain explicit limitations. No +backup authority, offline holdover or automatic trust renewal is promised. diff --git a/src/railiance_clock/authority.py b/src/railiance_clock/authority.py index e34d06b..fba3fba 100644 --- a/src/railiance_clock/authority.py +++ b/src/railiance_clock/authority.py @@ -30,7 +30,7 @@ class Authority: self.failed = False self.lock = threading.Lock() - def health(self): + def health(self, *, wall_ns=None, monotonic_ns=None): path = Path(self.config['health_file']) fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW) with os.fdopen(fd, 'rb') as stream: @@ -38,15 +38,23 @@ class Authority: if not stat.S_ISREG(info.st_mode) or info.st_uid != self.config.get('health_owner_uid', 0) or info.st_mode & 0o022: raise ClockError('untrusted host health file') health = strict_json(stream.read(8193)) + if not isinstance(health, dict): + raise ClockError('invalid host health object') if health.get('quality') != 'bounded' or health.get('boot_id') != self.boot_id or health.get('policy_id') != self.config['policy_id']: raise ClockError('host quality unusable') - age = self.monotonic() - health['observed_monotonic_ns'] + observed = bounded(health.get('observed_monotonic_ns')) + reported_error = bounded(health.get('error_ns')) + if wall_ns is None: + wall_ns = self.wall() + if monotonic_ns is None: + monotonic_ns = self.monotonic() + age = monotonic_ns - observed if not 0 <= age < self.config['max_health_age_ns']: raise ClockError('host health stale') - error = bounded(health['error_ns'] + (age * self.config['host_drift_ppm'] + 999999) // 1000000) + error = bounded(reported_error + (age * self.config['host_drift_ppm'] + 999999) // 1000000) if error > self.config['max_error_ns']: raise ClockError('host error bound exceeded') - if abs((self.wall() - self.start_wall) - (self.monotonic() - self.start_mono)) > self.config['max_continuity_error_ns']: + if abs((wall_ns - self.start_wall) - (monotonic_ns - self.start_mono)) > self.config['max_continuity_error_ns']: self.failed = True if self.failed: raise ClockError('authority continuity lost; new epoch admission required') @@ -61,9 +69,13 @@ class Authority: received = self.wall() error = self.health() sent = self.wall() + sent_mono = self.monotonic() if sent < received: self.failed = True raise ClockError('authority clock moved backwards') + # Validate the exact timestamp being signed, including health aging + # during sample preparation. A later wall read could hide a transient step. + error = max(error, self.health(wall_ns=sent, monotonic_ns=sent_mono)) payload = dict(request, epoch=self.epoch, policy_id=self.config['policy_id'], receive_unix_ns=str(received), send_unix_ns=str(sent), error_ns=str(error), max_age_ns=str(self.config['max_sample_age_ns']), quality='bounded') diff --git a/src/railiance_clock/health.py b/src/railiance_clock/health.py index c2a213c..cad9922 100644 --- a/src/railiance_clock/health.py +++ b/src/railiance_clock/health.py @@ -20,6 +20,15 @@ def properties(raw): def assess(policy, kernel, clock, upstream): # Explicitly conditional on trusted Linux discipline and admitted NTP source; # this is not independent UTC consensus or Byzantine-source detection. + if not all(isinstance(value, dict) for value in (policy, kernel, clock, upstream)): + raise ClockError('invalid health inputs') + for name in ('state', 'status', 'maxerror_us', 'esterror_us', 'tolerance_scaled_ppm'): + bounded(kernel.get(name)) + for name in ('measurement_margin_ns', 'max_error_ns'): + bounded(policy.get(name)) + sources = policy.get('upstreams') + if not isinstance(sources, list) or not sources or not all(isinstance(s, str) and s for s in sources): + raise ClockError('invalid upstream policy') if clock.get('NTPSynchronized') != 'yes' or clock.get('NTP') != 'yes': raise ClockError('host not synchronized') if kernel['state'] != 0 or kernel['status'] & (0x10 | 0x20 | 0x40): @@ -27,11 +36,8 @@ def assess(policy, kernel, clock, upstream): if upstream.get('ServerName') not in policy['upstreams']: raise ClockError('unadmitted upstream') message = upstream.get('NTPMessage','') - if not re.search(r'\bLeap=0,', message) or not re.search(r'\bIgnored=no,', message): + if not isinstance(message, str) or not re.search(r'\bLeap=0,', message) or not re.search(r'\bIgnored=no,', message): raise ClockError('unusable NTP source observation') - for name in ('maxerror_us','esterror_us','tolerance_scaled_ppm'): - if type(kernel[name]) is not int or kernel[name] < 0: - raise ClockError('invalid kernel error accounting') error = bounded(max(kernel['maxerror_us'],kernel['esterror_us']) * 1000 + policy['measurement_margin_ns']) if error > policy['max_error_ns']: raise ClockError('UTC error budget exceeded') @@ -40,20 +46,25 @@ def assess(policy, kernel, clock, upstream): def collect(policy): started = time.monotonic_ns() try: - if policy.get('status') != 'admitted': + if not isinstance(policy, dict) or policy.get('status') != 'admitted': raise ClockError('UTC policy not admitted') + bounded(policy.get('host_drift_ppm')) + if not isinstance(policy.get('kernel_helper'), str) or not policy['kernel_helper']: + raise ClockError('invalid kernel helper') + if not isinstance(policy.get('policy_id'), str) or not policy['policy_id']: + raise ClockError('invalid policy identity') kernel = json.loads(command([policy['kernel_helper']])) clock = properties(command(['timedatectl','show','--property=NTPSynchronized','--property=NTP'])) upstream = properties(command(['timedatectl','show-timesync','--all'])) error = assess(policy,kernel,clock,upstream) - elapsed = time.monotonic_ns() - started + elapsed = bounded(time.monotonic_ns() - started) error += (elapsed * policy['host_drift_ppm'] + 999999) // 1000000 if error > policy['max_error_ns']: raise ClockError('health collection exceeded error budget') return {'quality':'bounded','boot_id':Path('/proc/sys/kernel/random/boot_id').read_text().strip(), 'policy_id':policy['policy_id'],'observed_monotonic_ns':started,'error_ns':error, 'model':'linux-disciplined-kernel-maxerror-v1','upstream':upstream['ServerName']} - except (OSError, ValueError, KeyError, subprocess.TimeoutExpired): + except (OSError, ValueError, KeyError, TypeError, subprocess.TimeoutExpired): return {'quality':'unusable','reason':'host_quality_unavailable'} def write_health(path, value): diff --git a/tests/test_authority.py b/tests/test_authority.py index 794da9d..87f686e 100644 --- a/tests/test_authority.py +++ b/tests/test_authority.py @@ -63,3 +63,136 @@ def test_health_model_refusals(): for changed in [{**k,'state':5},{**k,'status':16},{**k,'maxerror_us':200_000},{**k,'maxerror_us':-1}]: with pytest.raises(ClockError):assess(p,changed,c,u) with pytest.raises(ClockError):assess(p,k,c,{**u,'ServerName':'unadmitted'}) + +@pytest.mark.parametrize('step', [100_000_000, -100_000_000]) +def test_step_at_final_timestamp_latches_refusal(authority, step): + a, _, clocks = authority + original = clocks['wall'] + # Receive and initial health are normal; only the timestamp to sign steps. + readings = iter([original, original, original + step]) + a.wall = lambda: next(readings, original) + with pytest.raises(ClockError): + a.sample(json.dumps(request()).encode()) + assert a.failed + # Returning to the old mapping must not restore the epoch's usability. + with pytest.raises(ClockError): + a.health() + + +def test_health_expires_during_sample_preparation(authority): + a, _, clocks = authority + original = a.health + calls = 0 + def advancing_health(**kwargs): + nonlocal calls + error = original(**kwargs) + calls += 1 + if calls == 1: + clocks['mono'] += a.config['max_health_age_ns'] + clocks['wall'] += a.config['max_health_age_ns'] + return error + a.health = advancing_health + with pytest.raises(ClockError, match='stale'): + a.sample(json.dumps(request()).encode()) + + +@pytest.fixture +def http_authority(authority): + a, hp, clocks = authority + server = Server(('127.0.0.1', 0), a) + thread = threading.Thread(target=server.serve_forever) + thread.start() + try: + yield server, hp + finally: + server.shutdown(); server.server_close(); thread.join() + + +@pytest.mark.parametrize('bad', [None, [], True, 1, 'invalid', + {'error_ns': -1}, {'error_ns': True}, {'error_ns': '1000'}, + {'observed_monotonic_ns': None}, {'observed_monotonic_ns': False}]) +def test_malformed_health_is_stable_http_refusal(http_authority, bad): + server, hp = http_authority + if isinstance(bad, dict): + valid = json.loads(hp.read_text()); valid.update(bad); bad = valid + hp.write_text(json.dumps(bad)) + base = f'http://127.0.0.1:{server.server_port}' + requests = [base + '/readyz', Request(base + '/v1/time-samples', + data=json.dumps(request()).encode(), headers={'Content-Type': 'application/json'})] + for req in requests: + with pytest.raises(HTTPError) as exc: + urlopen(req, timeout=2) + assert exc.value.code == 503 + assert exc.value.read() == b'{"status":"unusable"}' + with urlopen(base + '/healthz', timeout=2) as response: + assert response.status == 200 + + +@pytest.mark.parametrize('headers,body', [ + ('Content-Length: 2\r\nContent-Length: 2\r\n', b'{}'), + ('Content-Length: 2\r\nTransfer-Encoding: chunked\r\n', b'{}'), + ('Content-Length: 2\r\nContent-Encoding: gzip\r\n', b'{}'), + ('Content-Length: 1025\r\n', b''), + ('Content-Length: -1\r\n', b''), + ('Content-Length: 5\r\n', b'{}'), +]) +def test_http_framing_refused(http_authority, headers, body): + import socket + server, _ = http_authority + with socket.create_connection(server.server_address, timeout=2) as conn: + conn.sendall(('POST /v1/time-samples HTTP/1.0\r\n' + 'Content-Type: application/json\r\n' + headers + '\r\n').encode() + body) + conn.shutdown(socket.SHUT_WR) + with conn.makefile('rb') as response: + raw = response.read() + assert raw.startswith(b'HTTP/1.0 503') + assert raw.endswith(b'{"status":"unusable"}') + + +def test_connection_limit_and_capacity_recovery(authority, monkeypatch): + import socket + from railiance_clock.authority import Handler + a, _, _ = authority + ready = threading.Event() + released = threading.Event() + count = 0 + lock = threading.Lock() + original_setup = Handler.setup + original_worker = Server.process_request_thread + def setup(handler): + nonlocal count + original_setup(handler) + with lock: + count += 1 + if count == 8: + ready.set() + def worker(server, request, address): + nonlocal count + try: + original_worker(server, request, address) + finally: + with lock: + count -= 1 + if count == 0: + released.set() + monkeypatch.setattr(Handler, 'setup', setup) + monkeypatch.setattr(Server, 'process_request_thread', worker) + server = Server(('127.0.0.1', 0), a) + thread = threading.Thread(target=server.serve_forever) + thread.start() + clients = [] + try: + for _ in range(8): + clients.append(socket.create_connection(server.server_address, timeout=2)) + assert ready.wait(2) + with socket.create_connection(server.server_address, timeout=2) as extra: + assert extra.recv(1) == b'' + for conn in clients: + conn.close() + assert released.wait(2) + with urlopen(f'http://127.0.0.1:{server.server_port}/readyz', timeout=2) as response: + assert response.status == 200 + finally: + for conn in clients: + conn.close() + server.shutdown(); server.server_close(); thread.join() diff --git a/tests/test_clock.py b/tests/test_clock.py index 95c6628..2e702a1 100644 --- a/tests/test_clock.py +++ b/tests/test_clock.py @@ -112,3 +112,28 @@ def test_restart_preserves_floor(setup): payload.update(receive_unix_ns='50000000000',send_unix_ns='50001000000') return sign(payload,key,'test-key') with pytest.raises(ClockError):Clock(FileTrust(path),timer=timer,transport=older).read() + + +def test_rotation_requires_fresh_client_and_matching_signer(setup): + path, key, timer, transport = setup + old = Clock(FileTrust(path), timer=timer, transport=transport) + old.read() + replacement = ec.generate_private_key(ec.SECP256R1()) + data = json.loads(path.read_text()) + data['kid'] = 'rotated-key' + data['public_key_pem'] = replacement.public_key().public_bytes( + serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo).decode() + path.write_text(json.dumps(data)) + with pytest.raises(ClockError): + old.read() + # Loading new trust must still refuse the old signer. + with pytest.raises(ClockError): + Clock(FileTrust(path), timer=timer, transport=transport).read() + def rotated(request): + payload = verify(transport(request), key.public_key(), 'test-key') + return sign(payload, replacement, 'rotated-key') + assert Clock(FileTrust(path), timer=timer, transport=rotated).read().authority_id == 'railiance01' + data['status'] = 'revoked' + path.write_text(json.dumps(data)) + with pytest.raises(ClockError): + FileTrust(path) diff --git a/tests/test_health.py b/tests/test_health.py new file mode 100644 index 0000000..1e6f8ae --- /dev/null +++ b/tests/test_health.py @@ -0,0 +1,65 @@ +import json +import subprocess + +import pytest + +from railiance_clock import health + + +@pytest.fixture +def policy(): + return {'status': 'admitted', 'policy_id': 'test', 'kernel_helper': '/test/helper', + 'host_drift_ppm': 1000, 'upstreams': ['ntp.test'], + 'measurement_margin_ns': 1000, 'max_error_ns': 100_000_000} + + +@pytest.fixture +def commands(monkeypatch): + values = { + '/test/helper': json.dumps({'state': 0, 'status': 0, 'maxerror_us': 1000, + 'esterror_us': 100, 'tolerance_scaled_ppm': 32768000}), + 'show': 'NTP=yes\nNTPSynchronized=yes\n', + 'show-timesync': 'ServerName=ntp.test\nNTPMessage={ Leap=0, Ignored=no, }\n', + } + def command(argv): + value = values[argv[0] if len(argv) == 1 else argv[1]] + if isinstance(value, Exception): + raise value + return value + monkeypatch.setattr(health, 'command', command) + return values + + +def test_collection_and_replacement_on_failure(tmp_path, policy, commands): + path = tmp_path / 'health.json' + good = health.collect(policy) + assert good['quality'] == 'bounded' + health.write_health(path, good) + commands['show'] = 'NTP=yes\nNTPSynchronized=no\n' + bad = health.collect(policy) + health.write_health(path, bad) + assert json.loads(path.read_text()) == {'quality': 'unusable', 'reason': 'host_quality_unavailable'} + assert path.stat().st_mode & 0o777 == 0o644 + assert list(tmp_path.iterdir()) == [path] + + +@pytest.mark.parametrize('bad', [None, [], True, 'bad', {'status': 'admitted'}, + {'host_drift_ppm': -1}, {'host_drift_ppm': True}, {'upstreams': 'ntp.test'}, + {'measurement_margin_ns': -1}, {'max_error_ns': False}, {'kernel_helper': []}]) +def test_bad_policy_refuses(policy, commands, bad): + if isinstance(bad, dict): + if bad == {'status': 'admitted'}: + policy = bad + else: + policy.update(bad) + else: + policy = bad + assert health.collect(policy)['quality'] == 'unusable' + + +@pytest.mark.parametrize('bad', ['null', '[]', 'true', '{', + '{"state":false,"status":0}', '{"state":0,"status":"0"}', + OSError('private details'), subprocess.TimeoutExpired('private command', 3)]) +def test_helper_failures_are_sanitized(policy, commands, bad): + commands['/test/helper'] = bad + assert health.collect(policy) == {'quality': 'unusable', 'reason': 'host_quality_unavailable'} diff --git a/workplans/RCLK-WP-0005-infrastructure-as-code.md b/workplans/RCLK-WP-0005-infrastructure-as-code.md index 642fa78..918be47 100644 --- a/workplans/RCLK-WP-0005-infrastructure-as-code.md +++ b/workplans/RCLK-WP-0005-infrastructure-as-code.md @@ -102,6 +102,12 @@ trust bootstrap reviewed, least-privilege and no-redirect tests, config/source hashes and exact deployment/rollback plan. Fail if dynamic trust resolution needs usable Railiance time before the first sample with no independent bootstrap path. +Local follow-up 2026-09-28: authority sampling now checks continuity and health +age at the exact signed timestamp; malformed health refuses explicitly. Local +HTTP framing, capacity recovery, collection failure and trust-rotation tests +pass. These support C05/C13–C15, but do not close native admission or trust +delivery. + ## Integrate the clock prerequisite into the new-estate bootstrap path ```task @@ -167,6 +173,10 @@ Keep all unresolved work as live records before finishing this plan. RCLK-WP-000 may adopt a protected consumer only after this infrastructure gate AND its own explicit consumer review/negative tests pass. A deployed endpoint is not a grant. +Local follow-up 2026-09-28: the [operations runbook](../docs/operations-runbook.md) +now supplies reviewable alert, recovery, rotation and rollback procedures. +Telemetry/platform acceptance and native evidence remain the T06 blockers. + ### Production deployment evidence — 2026-09-15 The operator approved CCR-2026-0028 for platform and Clock ownership and then