feat: report bounded verified access dependency readiness
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / pytest-smoke (push) Failing after 3s

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e747-8f27-7242-8df8-8bc44f88c929
This commit is contained in:
tegwick 2026-09-28 14:32:36 +02:00
parent 7f0dc78607
commit c0383c9c2b
9 changed files with 294 additions and 17 deletions

View file

@ -67,7 +67,8 @@ or an assertion copied from token claims. Tests use synthetic owners explicitly.
The current CLI deliberately provides no fixture adapter or production bypass. The current CLI deliberately provides no fixture adapter or production bypass.
An explicit `SecuritySettings`/owner-facts composition now owns the HTTP client An explicit `SecuritySettings`/owner-facts composition now owns the HTTP client
and closes it with the runtime. Audit custody is probed per append. Real owner-facts and closes it with the runtime. Audit custody is probed per append. Real owner-facts
admission and additional owner dependency probes remain T02–T04 work; see the admission and independent owner probes remain T02–T04 work;
[dependency readiness](access-readiness.md) now reports recent verified operations; see the
[owner integration review](owner-access-integration.md). [owner integration review](owner-access-integration.md).
For this candidate all Hub resources are explicitly **platform-owned**. Other For this candidate all Hub resources are explicitly **platform-owned**. Other

56
docs/access-readiness.md Normal file
View file

@ -0,0 +1,56 @@
# Access dependency readiness
HUB-WP-0012 source candidate, 2026-09-28.
Protected `GET /readyz` now reports `access_identity`, `access_facts`,
`access_policy` and `access_audit`, plus their aggregate `access_profile`.
Controller configuration alone cannot mark these dependencies ready.
Each observation records the completion of a real verified dependency operation:
- Identity: the configured verifier returns a verified actor.
- Facts: the authoritative result matches that actor, includes evidence and is
fresh. Revoked or inactive state is a valid owner response, not an outage.
- Policy: the configured evaluator returns a verified decision. A signed denial
demonstrates usable policy evaluation while still denying the operation.
- Audit: the configured sink acknowledges authorization or refusal custody.
Committed-outcome delivery has its separate durable backlog readiness check.
Before an observation exists, its status is `unavailable`. Dependency errors or
cancellation mark it unavailable immediately. Successful observations expire to
`stale` after ten seconds using a monotonic clock. Invalid user-token refusals
neither poison a previously healthy identity sample nor refresh its age. Invalid
facts and malformed policy adapter results do not count as success. A fact that
expires while policy/audit run also becomes unavailable. The aggregate is `ok`
only when all four samples are fresh successes.
These process-local observations are diagnostics, never cached authorization.
Every request continues through token verification, current facts, signed policy
and durable audit. The controller itself now enforces a ten-second deadline,
including for direct SDK callers; HTTP/browser deadlines remain in place. No
credentials, owner URLs, token contents or raw exceptions appear in readiness.
`/readyz` remains protected: its own authorization refreshes the observations
before the handler reads them. If that authorization fails, the boundary returns
its normal 401/403/503 response instead of exposing dependency details to an
unauthorized caller. `AccessController.readiness_checks()` provides the same
read-only snapshot to an admitted host composition without making network calls.
Only minimal `/healthz` remains publicly available in the default composition.
This is **recent usability**, not an independent promise of owner reachability.
OIDC verification may use the admitted bounded JWKS cache; it does not prove that
the issuer is reachable at that instant. Flex-auth's reviewed `/healthz` reports
liveness only, and the facts adapter has no admitted independent health contract.
Hub therefore does not invent health endpoints, send synthetic root requests or
interpret liveness as an authorization grant. Independent owner probes, monitoring
admission and live outage/recovery receipts remain integration work.
Local tests cover cold/stale state, named failures and recovery, invalid-token
isolation, policy denial, bad facts/adapter results, cancellation/deadline behavior,
protected readiness and unchanged liveness. Conformance checks include access and
outcome-delivery dependencies when present, so correctly degraded readiness is
recognized rather than mistaken for a healthy dependency set.
Validation on 2026-09-28: 372 ordinary tests, five disposable PostgreSQL tests and
six owner-source Audit Core tests passed. Inventory, build and installed-wheel
checks passed. These are local receipts, not deployed owner acceptance.

View file

@ -245,6 +245,10 @@ class ConformanceHarness:
body = response.json() body = response.json()
checks = body.get("checks", {}) checks = body.get("checks", {})
dependency_keys = ("database", "repo_manager_projection", "workload_projection") dependency_keys = ("database", "repo_manager_projection", "workload_projection")
if "access_profile" in checks:
dependency_keys += ("access_profile", "access_identity", "access_facts", "access_policy", "access_audit")
if "outcome_delivery" in checks:
dependency_keys += ("outcome_delivery",)
dependency_values = {} dependency_values = {}
for required in dependency_keys: for required in dependency_keys:
if required not in checks: if required not in checks:

View file

@ -29,7 +29,7 @@ from hub_core.runtime.workload_projection import (
WorkloadProjectionService, WorkloadProjectionService,
) )
from hub_core.runtime.workload_projection_routes import create_workload_projection_router from hub_core.runtime.workload_projection_routes import create_workload_projection_router
from hub_core.security.boundary import AccessBoundary, AccessController, FactSource from hub_core.security.boundary import AccessBoundary, AccessController, FactSource, ACCESS_DEPENDENCIES
from hub_core.security.config import SecuritySettings from hub_core.security.config import SecuritySettings
from hub_core.security.browser import BrowserSettings, BrowserSessions, create_browser_router from hub_core.security.browser import BrowserSettings, BrowserSessions, create_browser_router
@ -173,7 +173,10 @@ def create_app(
**await workload_projection.readiness_checks(), **await workload_projection.readiness_checks(),
} }
if resolved_settings.enforce_access: if resolved_settings.enforce_access:
dependency_checks["access_profile"] = "ok" if access_controller else "unavailable" dependency_checks.update(access_controller.readiness_checks() if access_controller else {
**{"access_" + name: "unavailable" for name in ACCESS_DEPENDENCIES},
"access_profile": "unavailable",
})
if callable(getattr(resolved_store, "deliver_outcomes", None)): if callable(getattr(resolved_store, "deliver_outcomes", None)):
dependency_checks["outcome_delivery"] = (await resolved_store.outcome_readiness()) if outcome_delivery else "unavailable" dependency_checks["outcome_delivery"] = (await resolved_store.outcome_readiness()) if outcome_delivery else "unavailable"
ready = resolved_settings.is_ready(resolved_store.backend_name) and all( ready = resolved_settings.is_ready(resolved_store.backend_name) and all(

View file

@ -23,6 +23,9 @@ from hub_core.security.identity import AccessFailure, Actor
from hub_core.security.context import current_authorization from hub_core.security.context import current_authorization
PROFILE = "hub-core.access/1.0.0" PROFILE = "hub-core.access/1.0.0"
ACCESS_DEPENDENCIES = ("identity", "facts", "policy", "audit")
DEPENDENCY_OBSERVATION_TTL = 10
AUTHORIZATION_TIMEOUT = 10
@dataclass(frozen=True) @dataclass(frozen=True)
@ -107,10 +110,51 @@ class AccessController:
raise ValueError("immutable root identity is required") raise ValueError("immutable root identity is required")
self.identity, self.facts, self.policy, self.audit = identity, facts, policy, audit self.identity, self.facts, self.policy, self.audit = identity, facts, policy, audit
self.root_identity = (root_issuer, root_subject) self.root_identity = (root_issuer, root_subject)
self._observations: dict[str, tuple[str, float]] = {}
def readiness_checks(self) -> dict[str, str]:
"""Recent usable operations, not independent owner liveness or grants.
Diagnostic samples never participate in authorization. Missing adapters
or observations cannot be promoted to ready by configuration alone.
"""
now = time.monotonic()
checks = {}
for name in ACCESS_DEPENDENCIES:
sample = self._observations.get(name)
state = "unavailable" if sample is None else sample[0]
if state == "ok" and now - sample[1] > DEPENDENCY_OBSERVATION_TTL:
state = "stale"
checks["access_" + name] = state
checks["access_profile"] = "ok" if all(v == "ok" for v in checks.values()) else "unavailable"
return checks
async def _dependency(self, name, operation):
try:
result = await operation()
if name == "identity" and not isinstance(result, Actor):
raise ValueError("identity adapter returned no verified actor")
if name == "policy" and not isinstance(result, Decision):
raise ValueError("policy adapter returned no verified decision")
except BaseException as exc:
# A malformed user token does not prove an identity owner outage.
# Preserve any prior sample without refreshing its age.
if not (name == "identity" and isinstance(exc, AccessFailure) and exc.status == 401):
self._observations[name] = ("unavailable", time.monotonic())
raise
self._observations[name] = ("ok", time.monotonic())
return result
async def append_audit(self, record: dict) -> None:
await self._dependency("audit", lambda: self.audit.append(record))
async def authorize(self, token: str, action: str, resource: str, async def authorize(self, token: str, action: str, resource: str,
correlation_id: str, request_digest: str) -> Authorization: correlation_id: str, request_digest: str) -> Authorization:
actor = await self.identity.authenticate(token) async with asyncio.timeout(AUTHORIZATION_TIMEOUT):
return await self._authorize(token, action, resource, correlation_id, request_digest)
async def _authorize(self, token, action, resource, correlation_id, request_digest):
actor = await self._dependency("identity", lambda: self.identity.authenticate(token))
try: try:
return await self._authorize_actor(actor, action, resource, correlation_id, request_digest) return await self._authorize_actor(actor, action, resource, correlation_id, request_digest)
except Exception as exc: except Exception as exc:
@ -127,11 +171,7 @@ class AccessController:
or actor.tenant != "tenant:platform" or actor.assurance not in {"aal2", "aal3"} or actor.tenant != "tenant:platform" or actor.assurance not in {"aal2", "aal3"}
): ):
raise AccessFailure(403, "root_required") raise AccessFailure(403, "root_required")
facts = await self.facts.resolve(actor, resource) facts = await self._dependency("facts", lambda: self._resolve_facts(actor, resource))
if (facts.issuer, facts.subject, facts.actor_tenant) != (
actor.issuer, actor.subject, actor.tenant
) or not 0 <= time.time() - facts.checked_at <= 5 or not facts.evidence_id:
raise AccessFailure(503, "untrusted_or_stale_facts")
# v1 explicitly classifies Hub records as platform-owned. Other tenants # v1 explicitly classifies Hub records as platform-owned. Other tenants
# require the Phase 2 resource resolver/storage contract, not a header. # require the Phase 2 resource resolver/storage contract, not a header.
if facts.target_tenant != "tenant:platform": if facts.target_tenant != "tenant:platform":
@ -141,8 +181,8 @@ class AccessController:
if actor.principal_type == "human" and not facts.root_entitled: if actor.principal_type == "human" and not facts.root_entitled:
raise AccessFailure(403, "root_entitlement_required") raise AccessFailure(403, "root_entitlement_required")
context = Authorization(actor, action, resource, facts, correlation_id, request_digest) context = Authorization(actor, action, resource, facts, correlation_id, request_digest)
decision = await self.policy.evaluate(context) decision = await self._dependency("policy", lambda: self.policy.evaluate(context))
await self.audit.append({ await self.append_audit({
"profile": PROFILE, "correlation_id": correlation_id, "profile": PROFILE, "correlation_id": correlation_id,
"issuer": actor.issuer, "subject": actor.subject, "issuer": actor.issuer, "subject": actor.subject,
"principal_type": actor.principal_type, "actor_tenant": actor.tenant, "principal_type": actor.principal_type, "actor_tenant": actor.tenant,
@ -159,10 +199,19 @@ class AccessController:
if actor.expires_at <= time.time(): if actor.expires_at <= time.time():
raise AccessFailure(401, "expired_access_token") raise AccessFailure(401, "expired_access_token")
if time.time() - facts.checked_at > 5: if time.time() - facts.checked_at > 5:
self._observations["facts"] = ("unavailable", time.monotonic())
raise AccessFailure(503, "facts_expired_during_authorization") raise AccessFailure(503, "facts_expired_during_authorization")
return replace(context, decision_id=decision.decision_id, return replace(context, decision_id=decision.decision_id,
policy_version=decision.policy_version, policy_caller=decision.caller) policy_version=decision.policy_version, policy_caller=decision.caller)
async def _resolve_facts(self, actor, resource):
facts = await self.facts.resolve(actor, resource)
if (facts.issuer, facts.subject, facts.actor_tenant) != (
actor.issuer, actor.subject, actor.tenant
) or not 0 <= time.time() - facts.checked_at <= 5 or not facts.evidence_id:
raise AccessFailure(503, "untrusted_or_stale_facts")
return facts
def route_key(route, method: str) -> str: def route_key(route, method: str) -> str:
endpoint = route.endpoint endpoint = route.endpoint
@ -279,7 +328,7 @@ class AccessBoundary:
if self.controller is not None: if self.controller is not None:
try: try:
async with asyncio.timeout(3): async with asyncio.timeout(3):
await self.controller.audit.append({ await self.controller.append_audit({
"profile": PROFILE, "correlation_id": correlation, "profile": PROFILE, "correlation_id": correlation,
"outcome": "refused", "reason": failure.code, "outcome": "refused", "reason": failure.code,
"subject": actor.subject if actor else None, "subject": actor.subject if actor else None,

View file

@ -259,7 +259,7 @@ class BrowserSessions:
401 if isinstance(exc, jwt.PyJWTError) else 503, "browser_login_unavailable") 401 if isinstance(exc, jwt.PyJWTError) else 503, "browser_login_unavailable")
try: try:
async with asyncio.timeout(3): async with asyncio.timeout(3):
await self.controller.audit.append({ await self.controller.append_audit({
"profile": PROFILE, "correlation_id": correlation, "profile": PROFILE, "correlation_id": correlation,
"outcome": "refused", "reason": failure.code, "outcome": "refused", "reason": failure.code,
"action": SESSION_ACTION, "target_tenant": "tenant:platform", "action": SESSION_ACTION, "target_tenant": "tenant:platform",

View file

@ -0,0 +1,148 @@
import asyncio
from dataclasses import replace
from unittest.mock import patch
import pytest
from fastapi.testclient import TestClient
from hub_core.security.boundary import ACCESS_DEPENDENCIES
from hub_core.security.identity import AccessFailure
from test_access_boundary import Owners, HEADERS, runtime
async def authorize(controller):
return await controller.authorize('verified-root','hub.test','/docs','request:test','a'*64)
def test_missing_and_expired_observations_are_not_ready():
owners = Owners()
controller = owners.controller()
assert set(controller.readiness_checks().values()) == {'unavailable'}
with patch('hub_core.security.boundary.time.monotonic',return_value=100):
asyncio.run(authorize(controller))
assert set(controller.readiness_checks().values()) == {'ok'}
with patch('hub_core.security.boundary.time.monotonic',return_value=111):
checks = controller.readiness_checks()
assert checks['access_profile'] == 'unavailable'
assert all(checks['access_'+name] == 'stale' for name in ACCESS_DEPENDENCIES)
@pytest.mark.parametrize('dependency',ACCESS_DEPENDENCIES)
def test_owner_failure_is_named_and_recovers(dependency):
owners = Owners()
controller = owners.controller()
method = {'identity':'authenticate','facts':'resolve','policy':'evaluate','audit':'append'}[dependency]
original = getattr(owners,method)
async def failing(*args):
raise RuntimeError('private-credential-or-endpoint')
async def run():
await authorize(controller)
setattr(owners,method,failing)
with pytest.raises((RuntimeError,AccessFailure)):
await authorize(controller)
checks = controller.readiness_checks()
assert checks['access_'+dependency] == 'unavailable'
assert checks['access_profile'] == 'unavailable'
assert 'private' not in str(checks)
setattr(owners,method,original)
await authorize(controller)
assert set(controller.readiness_checks().values()) == {'ok'}
asyncio.run(run())
def test_bad_token_does_not_refresh_or_poison_owner_health():
owners = Owners()
controller = owners.controller()
async def run():
with patch('hub_core.security.boundary.time.monotonic',return_value=100):
await authorize(controller)
with patch('hub_core.security.boundary.time.monotonic',return_value=101):
with pytest.raises(AccessFailure):
await controller.authorize('invalid','hub.test','/docs','r','a'*64)
assert controller.readiness_checks()['access_identity'] == 'ok'
with patch('hub_core.security.boundary.time.monotonic',return_value=111):
assert controller.readiness_checks()['access_identity'] == 'stale'
asyncio.run(run())
def test_verified_denial_is_healthy_policy_but_never_a_grant():
owners = Owners()
owners.allow = False
controller = owners.controller()
async def run():
with pytest.raises(AccessFailure,match='policy_denied'):
await authorize(controller)
assert set(controller.readiness_checks().values()) == {'ok'}
with pytest.raises(AccessFailure,match='policy_denied'):
await authorize(controller) # Healthy observations never authorize.
asyncio.run(run())
def test_untrusted_facts_are_not_a_successful_health_observation():
owners = Owners()
owners.facts = replace(owners.facts,subject='wrong-principal')
controller = owners.controller()
with pytest.raises(AccessFailure,match='untrusted_or_stale_facts'):
asyncio.run(authorize(controller))
assert controller.readiness_checks()['access_facts'] == 'unavailable'
def test_cancelled_dependency_is_unavailable():
owners = Owners()
controller = owners.controller()
started = asyncio.Event()
async def wait_forever(*args):
started.set()
await asyncio.Event().wait()
owners.resolve = wait_forever
async def run():
task = asyncio.create_task(authorize(controller))
await started.wait()
task.cancel()
with pytest.raises(asyncio.CancelledError):
await task
assert controller.readiness_checks()['access_facts'] == 'unavailable'
asyncio.run(run())
def test_readyz_reports_verified_dependencies_without_extra_probes():
owners = Owners()
app = runtime(owners)
with TestClient(app) as client:
assert client.get('/readyz').status_code == 401
response = client.get('/readyz',headers=HEADERS)
assert response.status_code == 200
checks = response.json()['checks']
assert all(checks['access_'+name] == 'ok' for name in ACCESS_DEPENDENCIES)
assert len(owners.requests) == 1
owners.policy_down = True
assert client.get('/readyz',headers=HEADERS).status_code == 503
assert app.state.access_controller.readiness_checks()['access_policy'] == 'unavailable'
assert client.get('/healthz').json() == {'status':'ok'}
def test_controller_itself_bounds_a_hung_dependency(monkeypatch):
monkeypatch.setattr('hub_core.security.boundary.AUTHORIZATION_TIMEOUT', .01)
owners = Owners()
controller = owners.controller()
async def hang(*args):
await asyncio.Event().wait()
owners.evaluate = hang
async def run():
with pytest.raises(TimeoutError):
await authorize(controller)
assert controller.readiness_checks()['access_policy'] == 'unavailable'
assert not owners.records
asyncio.run(run())
def test_malformed_policy_result_is_not_reported_healthy():
owners = Owners()
controller = owners.controller()
async def invalid(*args):
return {'allowed': True}
owners.evaluate = invalid
with pytest.raises(AccessFailure):
asyncio.run(authorize(controller))
assert controller.readiness_checks()['access_policy'] == 'unavailable'
assert not owners.records

View file

@ -167,13 +167,11 @@ def test_all_native_mutation_families_keep_attributed_outcomes(target):
from hub_core.conformance import ConformanceHarness from hub_core.conformance import ConformanceHarness
client,store,owners,_ = target client,store,owners,_ = target
owners.facts = replace(owners.facts,producer_addresses=frozenset({'hub:ops-hub'})) owners.facts = replace(owners.facts,producer_addresses=frozenset({'hub:ops-hub'}))
# Composition is frozen when the app is created; run the twelve business # The harness must recognize the missing dispatcher as a degraded dependency.
# checks except dependency readiness, which separately reports no dispatcher.
harness = ConformanceHarness(client) harness = ConformanceHarness(client)
client.headers.update(HEADERS) client.headers.update(HEADERS)
report = harness.run() report = harness.run()
assert all(c.status == 'pass' for c in report.checks if c.check_id != 'C9'), report.to_dict() assert report.passed, report.to_dict()
assert next(c for c in report.checks if c.check_id == 'C9').status == 'fail'
assert client.get('/readyz').json()['checks']['outcome_delivery'] == 'unavailable' assert client.get('/readyz').json()['checks']['outcome_delivery'] == 'unavailable'
pending = rows(store,runtime_outcome_outbox) pending = rows(store,runtime_outcome_outbox)
operations = {r['envelope']['data']['operation'] for r in pending} operations = {r['envelope']['data']['operation'] for r in pending}

View file

@ -408,6 +408,24 @@ Production grants, deployment and live receiver acceptance remain open; no live
database was contacted or migrated. database was contacted or migrated.
See [test details](../docs/conformance.md#disposable-postgresql-gate). See [test details](../docs/conformance.md#disposable-postgresql-gate).
## Access readiness continuation — 2026-09-28
Replaced controller-presence readiness with named identity/facts/policy/audit
observations and an aggregate. Missing, failed or stale observations fail
readiness. Observations come from verified operations and never replace access
checks; invalid tokens cannot poison/refresh dependency health and verified
policy denials remain healthy evaluations. The controller itself now bounds
all callers to ten seconds. Readiness remains protected and liveness minimal.
Flex-auth source exposes only liveness; no admitted independent facts health API
exists. The [readiness contract](../docs/access-readiness.md) explicitly describes
recent usability rather than inventing an owner probe or claiming reachability.
Updated conformance to include access and outcome-delivery dependencies.
T04 remains `progress`; independent owner monitoring and live acceptance remain
open. Validation: **372 ordinary tests**, **five real PostgreSQL tests** and
**six owner-source Audit Core tests** pass; inventory, package build and isolated
wheel validation pass. No deployment or external owner contract changed.
## Acceptance checkpoints ## Acceptance checkpoints
- [x] Architecture/source/runtime review captured; new implementation owner is hub-core - [x] Architecture/source/runtime review captured; new implementation owner is hub-core