Implement bounded release runtime and durable native health sampler
Assistant: codex Assistant-Model: gpt-6.1-sol Assistant-Session: 01a126e1-76f8-7643-8cf3-e93336539ade
This commit is contained in:
parent
cc2bb9d0f7
commit
52dcc08f23
13 changed files with 937 additions and 10 deletions
|
|
@ -0,0 +1,10 @@
|
|||
{
|
||||
"schema_version": "activity-core-release-caller-metadata/v1",
|
||||
"serviceaccounts": [
|
||||
"activity-core-eso",
|
||||
"default"
|
||||
],
|
||||
"bindings": [],
|
||||
"secret_read": false,
|
||||
"native_mutation": false
|
||||
}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"schema_version": "activity-core-release-principal-observation/v1",
|
||||
"estate_cluster_uid_verified": true,
|
||||
"dedicated_serviceaccount_present": false,
|
||||
"secret_read": false,
|
||||
"native_mutation": false
|
||||
}
|
||||
36
docs/evidence/2026-10-11-release-runtime/validation.json
Normal file
36
docs/evidence/2026-10-11-release-runtime/validation.json
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
{
|
||||
"schema_version": "activity-core-release-runtime-preparation/v1",
|
||||
"task": "ACTIVITY-WP-0041-T03",
|
||||
"prepared_at": "2026-10-11T21:19:23.517339+00:00",
|
||||
"scope": "ACTIVITY-WP-0041-image-only-v1",
|
||||
"tests": {
|
||||
"command": "PYTHONPATH=src python -m pytest tests/test_gitops_release.py tests/test_release_broker.py tests/test_release_transport.py tests/test_release_dispatch.py tests/test_release_health.py tests/test_release_sampler.py tests/test_release_runtime.py -q",
|
||||
"passed": 113
|
||||
},
|
||||
"native_acceptance": false,
|
||||
"native_mutation": false,
|
||||
"secret_read": false,
|
||||
"fixture_keys_are_authority": false,
|
||||
"source_bootstrap_starts_soak": false,
|
||||
"native_required": [
|
||||
"dedicated authenticated principal",
|
||||
"exclusive Git/Kubernetes credential custody",
|
||||
"independent signed issuer/owner acceptance",
|
||||
"deployed continuous observer",
|
||||
"Temporal registration",
|
||||
"normal release and failed health rollback",
|
||||
"revocation and denials"
|
||||
],
|
||||
"coordination": {
|
||||
"case_id": "ACT41-release-custody-2026-10-11",
|
||||
"scenario_type": "cross_owner_wait",
|
||||
"state": "waiting",
|
||||
"beneficiary_workplan_id": "ACTIVITY-WP-0041",
|
||||
"beneficiary_task_id": "ACTIVITY-WP-0041-T03",
|
||||
"obligor_repo": "railiance-platform",
|
||||
"obligor_intake_id": "RPF-IN-0011",
|
||||
"blocked_on": "message-from:railiance-platform",
|
||||
"reason": "independent owner preparation; awaiting exact owner task binding; no circular RPF48 dependency",
|
||||
"central_projection": "not yet published"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,7 +1,8 @@
|
|||
# Image-only release broker core
|
||||
|
||||
ACTIVITY-WP-0041-T03 owns this implementation. **Not activated in production.**
|
||||
The modules are not wired to a production API, worker, schedule or credential source.
|
||||
The source now includes a read-only API invariant route, sampler and explicit dedicated
|
||||
worker entrypoint. They are not deployed or wired to production custody or trusted issuers.
|
||||
Construction requires `admitted=True` from trusted deployment configuration; that
|
||||
switch is a local guard, not proof that an identity has actually been admitted.
|
||||
The current deployed revision is unchanged, so this code does not restart soak.
|
||||
|
|
@ -50,8 +51,8 @@ production credentials and live Git/ArgoCD rollback are **not** proven by these
|
|||
## Still required before activation
|
||||
|
||||
1. Admit and exercise the implemented transport adapter against an isolated
|
||||
authenticated Git/Kubernetes environment. Supply the actual bounded report/schedule
|
||||
invariant probe, pinned cluster UID and immutable credential/context configuration.
|
||||
authenticated Git/Kubernetes environment. Deploy and verify the supplied bounded report/schedule invariant probe,
|
||||
pinned cluster UID and immutable credential/context configuration.
|
||||
Verify the real authorization denials and restart behavior; simulated Kubernetes
|
||||
responses do not prove those. A generic repository-write token is not path enforcement.
|
||||
2. Admit the dedicated principal and credential custody through the owner lane.
|
||||
|
|
@ -89,8 +90,8 @@ Kubernetes JSON patches compare resourceVersion and spec before setting the fixe
|
|||
sync operation. Existing foreign operations are never overwritten. Root sync selects
|
||||
only activity-core; child sync never prunes. Health checks require exact revision,
|
||||
Synced/Healthy and all three deployments ready at their observed generations, plus
|
||||
a mandatory report/schedule invariant probe. That trusted callback must itself use
|
||||
bounded I/O; no production probe is supplied or implied by the fixture implementation.
|
||||
a mandatory report/schedule invariant probe. The concrete callback is `release_sampler.Probe`, with bounded fixed reads and
|
||||
the source-defined API invariant route. Its deployment remains unproven.
|
||||
Polling windows are bounded, with individual command timeout overhead possible.
|
||||
|
||||
`release_observer.HealthObserver` records samples durably. It requires consecutive
|
||||
|
|
@ -102,8 +103,8 @@ not a claim that every instant between samples was observed. Two distant healthy
|
|||
snapshots cannot establish the interval. New observers start from their first actual
|
||||
sample; they do not backfill the earlier deployment timestamp.
|
||||
|
||||
Neither module is connected to a production schedule, endpoint or credential source.
|
||||
Trusted signer custody, real invariant probes, production Temporal registration and isolated
|
||||
The new source entrypoints are described in [release-runtime-owner-handoff.md](release-runtime-owner-handoff.md).
|
||||
Trusted signer custody, deployed invariant probes, production Temporal registration and isolated
|
||||
Kubernetes authorization/rollback tests remain required before activation. Production
|
||||
revision and the deployment observation clock are unchanged by this source-only work.
|
||||
|
||||
|
|
@ -138,5 +139,10 @@ of delivery acknowledgements. The sink can fan out to approved evidence destinat
|
|||
Tests exercise the real ledger through the activities and validate Temporal sandbox
|
||||
construction, with in-memory transports/sinks and orchestration stubs. This does not
|
||||
prove a deployed Temporal server, real Kubernetes authorization or production sinks.
|
||||
Continuous observer scheduling, actual sink adapters, admission credentials and
|
||||
authenticated deployment proof remain activation prerequisites in T03.
|
||||
`release_runtime` binds the actual transport, durable sanitized SQLite audit sink,
|
||||
immutable private custody files and independent public-key registry to this worker.
|
||||
It verifies the actual Kubernetes username and eleven allow/deny boundaries before
|
||||
starting. `release_sampler --watch` supplies 60-second durable observations, without
|
||||
backfill. Continuous deployment, native Git identity/custody, independent signed
|
||||
owner acceptance and authenticated deployment/rollback proof remain prerequisites.
|
||||
A trusted configuration file does not itself establish those facts.
|
||||
|
|
|
|||
80
docs/release-runtime-owner-handoff.md
Normal file
80
docs/release-runtime-owner-handoff.md
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
# Bounded release runtime owner handoff
|
||||
|
||||
ACTIVITY-WP-0041-T03 implements the existing D2-authorized image-only scope
|
||||
`ACTIVITY-WP-0041-image-only-v1`. This source preparation does not request another
|
||||
founder decision and does not activate release authority. Platform custody remains
|
||||
independent preparation, tracked by RPF-IN-0011 rather than the dependent RPF48 gate.
|
||||
|
||||
The fixed target is `system:serviceaccount:activity-core:activity-core-release-broker`
|
||||
on cluster UID `a553c742-0115-43d4-99a4-a5ca56fe0786`, estate IP `92.205.62.239`.
|
||||
The current native metadata observation found only `default` and `activity-core-eso`
|
||||
service accounts and no matching namespace/ArgoCD role bindings. The ESO account
|
||||
has unrelated secret projection custody and is not an interchangeable release actor.
|
||||
These observations do not attest ClusterRoleBinding absence or authenticate as a
|
||||
workload. No Secret read or credential value is part of this record.
|
||||
|
||||
## Executable source
|
||||
|
||||
`GET /release/health` checks the three source-defined frontend-patterns definitions,
|
||||
exact deterministic report sinks, cron/calendar schedule semantics, actual Temporal
|
||||
workflow/action arguments, queue, pause state, overlap and catch-up policy. It reads
|
||||
only the fixed definition IDs and schedule IDs; it never updates either subsystem.
|
||||
Its 15-second probe deadline and sanitized boolean result bound failures.
|
||||
|
||||
`python -m activity_core.release_sampler --database OWNER_PRIVATE_LEDGER
|
||||
--receipt OWNER_RECEIPT --watch` polls fixed estate/Application/deployment/service
|
||||
metadata and that API route every 60 seconds. Gaps beyond 90 seconds, unavailability,
|
||||
revision changes or backwards clocks invalidate continuous observation. A fresh
|
||||
ledger starts now. Prior sync timestamps and two snapshots cannot manufacture a
|
||||
24-hour receipt. Source-only preparation does not start this observer.
|
||||
|
||||
`python -m activity_core.release_runtime --configuration OWNER_PRIVATE_CONFIGURATION`
|
||||
starts the existing dedicated Temporal release dispatcher. The deployment-owned
|
||||
configuration requires schema `activity-core-release-runtime/v1`, `enabled: true`,
|
||||
principal `activity-core-release-broker`, exact cluster UID, private owned
|
||||
`state_directory`, `kubeconfig`, `git_config`, and `receipt_trust`. Every trust entry
|
||||
contains an independently admitted public key, principal and role set; all four
|
||||
build/review/health/retention roles are required. No private signing key or caller
|
||||
supplied credential is accepted. Config/custody drift or disabling stops subsequent
|
||||
calls. Inherited Git/Kubernetes overrides and SSH agent credentials are excluded.
|
||||
|
||||
At startup, the actual credential must report the exact Kubernetes username and
|
||||
pass four fixed Application allow checks plus seven negative boundary checks.
|
||||
The unchanged broker then verifies five-minute signed receipts, image-only source
|
||||
and CI bounds, retention and continuous health. Git publication changes only the
|
||||
fixed child targetRevision with compare-and-swap; root synchronization selects that
|
||||
one child and child pruning stays false. Temporal input is only an already-admitted
|
||||
release ID. Durable private SQLite audit delivery is idempotent and sanitized.
|
||||
A configuration flag or fixture key does not establish native owner acceptance.
|
||||
|
||||
## Platform preparation and acceptance
|
||||
|
||||
Under the existing accepted scope, Platform supplies the dedicated workload identity,
|
||||
credential route and independently operated issuer custody. Fixed allowed operations
|
||||
are read-only estate/runtime metadata plus the reviewed broker's Git one-field
|
||||
publication and exact two-Application operations. Repository write credentials alone
|
||||
cannot enforce a field boundary; keep them exclusively behind this broker and prove
|
||||
real wrong-path, wrong-application, direct deployment, secret-read, RBAC, exec and
|
||||
revocation denials. Do not reuse workstation admin or ESO credentials. The runtime
|
||||
checks Kubernetes identity, but native Git push identity and custody require owner
|
||||
acceptance evidence before deployment.
|
||||
|
||||
Acceptance must bind exact source/runtime digests, principal/credential custody,
|
||||
independent issuers, observer process and durable ledger, real Temporal registration,
|
||||
one normal release and one failed-health rollback, restart/concurrency/denial and
|
||||
revocation receipts. The separately admitted signed owner acceptance must use the
|
||||
existing ACTIVITY-WP-0041-T03 scope and expire within five minutes. Platform's
|
||||
`docs/activity-core-release-admission.md` and readiness gate define this contract.
|
||||
No evidence is self-signed by this worker. RPF-WP-0036-T03 covers backup freshness,
|
||||
so it is not a substitute release-custody dependency.
|
||||
|
||||
Abort owner is activity-core for a candidate/invariant/dispatch failure and
|
||||
railiance-platform for custody/authentication or native Git/Argo admission failure.
|
||||
Stop before publication when identity, source or admission is uncertain. After
|
||||
publication, preserve the durable release row; the admitted broker reverses the
|
||||
one-field pin to its retained rollback revision and performs the same targeted,
|
||||
non-pruning sync. Failed rollback retains its active slot for recovery. No direct
|
||||
Deployment or broad root operation is a recovery shortcut. Disabling configuration
|
||||
halts new work; actual credential revocation and worker shutdown remain Platform
|
||||
operations. Attended admin login is orange only when a reviewed native custody
|
||||
command requires it; none has been scheduled or executed in this preparation.
|
||||
|
|
@ -382,3 +382,10 @@ async def approve_event_type(type_id: str) -> dict[str, str]:
|
|||
{"tid": type_id},
|
||||
)
|
||||
return {"type_id": type_id, "status": "active", "message": "approved"}
|
||||
|
||||
|
||||
@app.get("/release/health")
|
||||
async def release_health() -> dict[str, Any]:
|
||||
"""Fixed read-only release invariants; no caller-defined operation or target."""
|
||||
from activity_core.release_health import probe
|
||||
return await probe(_get_db(), _get_temporal())
|
||||
|
|
|
|||
119
src/activity_core/release_health.py
Normal file
119
src/activity_core/release_health.py
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
"""Fixed read-only release invariants; no trigger, schedule mutation or credential API."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from uuid import NAMESPACE_URL, uuid5, UUID
|
||||
|
||||
from sqlalchemy import select
|
||||
from temporalio.client import ScheduleActionStartWorkflow, ScheduleOverlapPolicy
|
||||
|
||||
from .orm import ActivityDefinition
|
||||
|
||||
NAMESPACE = uuid5(NAMESPACE_URL, 'activity-core:activity-definition')
|
||||
FRONTEND = {
|
||||
str(uuid5(NAMESPACE, f'frontend-patterns-{cadence}')): (cadence, cron)
|
||||
for cadence, cron in [('daily', '0 2 * * *'), ('weekly', '0 3 * * 2'), ('monthly', '0 9 1 * *')]
|
||||
}
|
||||
SINK = {'type': 'state-hub-progress', 'event_type': 'fep_feedback_intake',
|
||||
'author': 'activity-core', 'topic_id': 'fe2aaa78-9c20-4feb-b3d2-4fe0529572a3'}
|
||||
|
||||
|
||||
def definition_matches(row):
|
||||
expected = FRONTEND.get(str(row.id))
|
||||
if expected is None:
|
||||
return False
|
||||
cadence, cron = expected
|
||||
instructions = row.instructions_json
|
||||
return (row.enabled is True and row.trigger_type == 'cron'
|
||||
and row.trigger_config.get('cron_expression') == cron
|
||||
and row.trigger_config.get('timezone') == 'Europe/Berlin'
|
||||
and row.trigger_config.get('misfire_policy') == 'catchup_latest'
|
||||
and row.trigger_config.get('catchup_window_seconds') == 86400
|
||||
and row.trigger_config.get('jitter_seconds', 0) == 0
|
||||
and isinstance(instructions, list) and len(instructions) == 1
|
||||
and instructions[0].get('id') == f'frontend-patterns-{cadence}-report'
|
||||
and instructions[0].get('model') == 'deterministic'
|
||||
and instructions[0].get('report_sinks') == [SINK])
|
||||
|
||||
|
||||
def expanded(ranges, minimum, maximum):
|
||||
values = set()
|
||||
for item in ranges:
|
||||
if not minimum <= item.start <= item.end <= maximum or item.step < 1:
|
||||
raise ValueError('invalid schedule range')
|
||||
values.update(range(item.start, item.end + 1, item.step))
|
||||
return values
|
||||
|
||||
|
||||
def schedule_times_match(spec, cadence, cron):
|
||||
if (spec.intervals or spec.skip or spec.start_at is not None or spec.end_at is not None
|
||||
or spec.jitter not in (None, timedelta(0)) or spec.time_zone_name != 'Europe/Berlin'):
|
||||
return False
|
||||
# Temporal converts cron strings to calendar specs on the server. Accept the
|
||||
# exact equivalent fixed calendar, never merely the source cron text.
|
||||
if list(spec.cron_expressions) == [cron] and not spec.calendars:
|
||||
return True
|
||||
if spec.cron_expressions or len(spec.calendars) != 1:
|
||||
return False
|
||||
calendar = spec.calendars[0]
|
||||
hour = {'daily': 2, 'weekly': 3, 'monthly': 9}[cadence]
|
||||
return (not calendar.year and expanded(calendar.second,0,59) == {0}
|
||||
and expanded(calendar.minute,0,59) == {0} and expanded(calendar.hour,0,23) == {hour}
|
||||
and expanded(calendar.day_of_month,1,31) == ({1} if cadence == 'monthly' else set(range(1,32)))
|
||||
and expanded(calendar.month,1,12) == set(range(1,13))
|
||||
and expanded(calendar.day_of_week,0,6) == ({2} if cadence == 'weekly' else set(range(7))))
|
||||
|
||||
|
||||
def schedule_matches(description, definition_id, decoded_args):
|
||||
schedule = description.schedule
|
||||
cadence, cron = FRONTEND[definition_id]
|
||||
action = schedule.action
|
||||
return (schedule.state.paused is False and not schedule.state.limited_actions
|
||||
and schedule_times_match(schedule.spec,cadence,cron)
|
||||
and schedule.policy.overlap == ScheduleOverlapPolicy.BUFFER_ONE
|
||||
and schedule.policy.catchup_window == timedelta(seconds=86400)
|
||||
and isinstance(action, ScheduleActionStartWorkflow)
|
||||
and action.workflow == 'RunActivityWorkflow' and action.task_queue == 'orchestrator-tq'
|
||||
and action.id == f'activity-{definition_id}:${{firstScheduledTime}}'
|
||||
and list(decoded_args) == [definition_id, 'scheduled', None])
|
||||
|
||||
|
||||
async def schedule_args(temporal, description):
|
||||
action = description.schedule.action
|
||||
if not isinstance(action, ScheduleActionStartWorkflow):
|
||||
raise ValueError('unknown schedule action')
|
||||
# Actual describe() results contain encoded Payload arguments, not Python
|
||||
# values. Decode using the already-bound client converter, never a request codec.
|
||||
return await temporal.data_converter.decode(action.args) if action._from_raw else list(action.args)
|
||||
|
||||
|
||||
async def probe(session_factory, temporal):
|
||||
"""Read the exact three definitions and their real Temporal schedules, bounded."""
|
||||
result = {'schema_version': 'activity-core-release-health/v1',
|
||||
'observed_at': datetime.now(timezone.utc).isoformat(),
|
||||
'healthy': False, 'definition_count': 0, 'schedule_count': 0,
|
||||
'definitions_and_report_sinks_match': False, 'schedules_match': False}
|
||||
try:
|
||||
async with asyncio.timeout(15):
|
||||
async with session_factory() as session:
|
||||
rows = list((await session.scalars(select(ActivityDefinition).where(
|
||||
ActivityDefinition.id.in_([UUID(identity) for identity in FRONTEND])))).all())
|
||||
result['definition_count'] = len(rows)
|
||||
if len(rows) != 3 or {str(r.id) for r in rows} != set(FRONTEND) or not all(definition_matches(r) for r in rows):
|
||||
return result
|
||||
result['definitions_and_report_sinks_match'] = True
|
||||
descriptions = await asyncio.gather(*[
|
||||
temporal.get_schedule_handle(f'activity-schedule-{identity}').describe()
|
||||
for identity in sorted(FRONTEND)
|
||||
])
|
||||
result['schedule_count'] = len(descriptions)
|
||||
decoded = await asyncio.gather(*[schedule_args(temporal,description) for description in descriptions])
|
||||
result['schedules_match'] = all(schedule_matches(description, identity, args)
|
||||
for identity, description, args in zip(sorted(FRONTEND), descriptions, decoded, strict=True))
|
||||
result['healthy'] = result['schedules_match']
|
||||
except Exception:
|
||||
# Database contents, report payloads and Temporal exceptions never escape.
|
||||
result['error'] = 'release invariant observation unavailable'
|
||||
result['observed_at'] = datetime.now(timezone.utc).isoformat()
|
||||
return result
|
||||
201
src/activity_core/release_runtime.py
Normal file
201
src/activity_core/release_runtime.py
Normal file
|
|
@ -0,0 +1,201 @@
|
|||
"""Owner-configured entrypoint for the existing dedicated Temporal release worker.
|
||||
|
||||
No caller-supplied transport factory, signing key, command or admission endpoint.
|
||||
A trusted deployment file is configuration, not proof of native authority/custody.
|
||||
The actual principal, key issuers and isolated denial/rollback acceptance remain
|
||||
owner admission requirements before this process can be deployed.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import stat
|
||||
|
||||
from temporalio.client import Client
|
||||
from temporalio.exceptions import ApplicationError
|
||||
|
||||
from .release_broker import Broker, Receipts, ROLES, canonical
|
||||
from .release_dispatch import ReleaseActivities, release_worker
|
||||
from .release_sampler import CLUSTER_UID, Probe
|
||||
from .release_transport import GitArgoBackend
|
||||
|
||||
TEMPORAL_HOST = 'actcore-temporal.activity-core.svc.cluster.local:7233'
|
||||
FIELDS = {'schema_version','enabled','principal','cluster_uid','state_directory',
|
||||
'kubeconfig','git_config','receipt_trust'}
|
||||
|
||||
|
||||
def trusted_file(path):
|
||||
path = Path(path)
|
||||
status = path.lstat()
|
||||
if (not stat.S_ISREG(status.st_mode) or status.st_uid != os.geteuid()
|
||||
or stat.S_IMODE(status.st_mode) & 0o077):
|
||||
raise ValueError('trusted deployment file must be owned and private')
|
||||
return path
|
||||
|
||||
|
||||
class Admission:
|
||||
def __init__(self, path):
|
||||
self.path=trusted_file(path)
|
||||
self.value=json.loads(self.path.read_text())
|
||||
value=self.value
|
||||
if (set(value)!=FIELDS or value['schema_version']!='activity-core-release-runtime/v1'
|
||||
or value['enabled'] is not True or value['principal']!='activity-core-release-broker'
|
||||
or value['cluster_uid']!=CLUSTER_UID):
|
||||
raise ValueError('runtime configuration is disabled or outside admitted scope')
|
||||
self.kubeconfig=trusted_file(value['kubeconfig'])
|
||||
self.git_config=trusted_file(value['git_config'])
|
||||
self.configuration=canonical({k:v for k,v in value.items() if k!='enabled'})
|
||||
# Retain only digests for drift/revocation checks, never credential copies.
|
||||
import hashlib
|
||||
self.custody={str(path):hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
for path in (self.kubeconfig,self.git_config)}
|
||||
self.root=Path(value['state_directory'])
|
||||
self.root.mkdir(parents=True,exist_ok=True,mode=0o700)
|
||||
status=self.root.lstat()
|
||||
if not stat.S_ISDIR(status.st_mode) or status.st_uid!=os.geteuid() or stat.S_IMODE(status.st_mode)&0o077:
|
||||
raise ValueError('runtime state directory must be owned and private')
|
||||
self.trust={}
|
||||
for key_id,entry in value['receipt_trust'].items():
|
||||
if (set(entry)!={'public_key','principal','roles'} or not entry['principal']
|
||||
or not entry['roles'] or not set(entry['roles'])<=ROLES):
|
||||
raise ValueError('invalid deployment-owned receipt trust')
|
||||
public=base64.b64decode(entry['public_key'],validate=True)
|
||||
if len(public)!=32:raise ValueError('invalid Ed25519 public key')
|
||||
self.trust[key_id]={'public_key':public,'principal':entry['principal'],'roles':set(entry['roles'])}
|
||||
if set().union(*(entry['roles'] for entry in self.trust.values()))!=ROLES:
|
||||
raise ValueError('all independent receipt roles require trusted issuers')
|
||||
|
||||
def require(self):
|
||||
import hashlib
|
||||
current=json.loads(trusted_file(self.path).read_text())
|
||||
if (current.get('enabled') is not True
|
||||
or canonical({k:v for k,v in current.items() if k!='enabled'})!=self.configuration):
|
||||
raise ValueError('runtime admission disabled or deployment configuration changed')
|
||||
if any(hashlib.sha256(trusted_file(path).read_bytes()).hexdigest()!=digest for path,digest in self.custody.items()):
|
||||
raise ValueError('credential custody files changed; owner re-admission required')
|
||||
|
||||
def environment(self):
|
||||
# Prevent fallback to an unrelated global Git credential helper/kubeconfig.
|
||||
return {'KUBECONFIG':str(self.kubeconfig),'GIT_CONFIG_GLOBAL':str(self.git_config),
|
||||
'GIT_CONFIG_NOSYSTEM':'1','GIT_TERMINAL_PROMPT':'0'}
|
||||
|
||||
|
||||
class DurableAudit:
|
||||
"""Private durable idempotent local receipt sink; no raw transport payloads."""
|
||||
KEYS={'event_id','event_type','release_id','application','phase','observed_at','candidate','rollback'}
|
||||
def __init__(self,path):
|
||||
self.path=Path(path)
|
||||
with self.connect() as db:
|
||||
db.execute('CREATE TABLE IF NOT EXISTS events (event_id TEXT PRIMARY KEY, payload TEXT NOT NULL)')
|
||||
|
||||
def connect(self):
|
||||
db=sqlite3.connect(self.path,timeout=5)
|
||||
db.execute('PRAGMA synchronous=FULL')
|
||||
return db
|
||||
|
||||
def __call__(self,event):
|
||||
if set(event)!=self.KEYS or event['application']!='activity-core' or event['event_type']!='release_transition':
|
||||
raise ValueError('audit event outside release scope')
|
||||
payload=canonical(event).decode()
|
||||
with self.connect() as db:
|
||||
prior=db.execute('SELECT payload FROM events WHERE event_id=?',(event['event_id'],)).fetchone()
|
||||
if prior and prior[0]!=payload:raise ValueError('audit event identity conflict')
|
||||
db.execute('INSERT OR IGNORE INTO events VALUES (?,?)',(event['event_id'],payload))
|
||||
|
||||
|
||||
def verify_runtime_identity(runner):
|
||||
"""Check the real configured Kubernetes subject and exact allow/deny boundary."""
|
||||
namespace=json.loads(runner(['kubectl','--request-timeout=20s','get','namespace','kube-system','-o','json'],timeout=25))
|
||||
if namespace.get('metadata',{}).get('uid')!=CLUSTER_UID:
|
||||
raise ValueError('runtime cluster UID unverified')
|
||||
identity=json.loads(runner(['kubectl','--request-timeout=20s','auth','whoami','-o','json'],timeout=25))
|
||||
user=identity.get('status',{}).get('userInfo',{})
|
||||
if (user.get('username')!='system:serviceaccount:activity-core:activity-core-release-broker'
|
||||
or 'system:masters' in user.get('groups',[])):
|
||||
raise ValueError('actual release principal differs from dedicated service account')
|
||||
allowed=[('get','argoproj.io','applications','argocd','activity-core'),
|
||||
('patch','argoproj.io','applications','argocd','activity-core'),
|
||||
('get','argoproj.io','applications','argocd','railiance-apps-root'),
|
||||
('patch','argoproj.io','applications','argocd','railiance-apps-root')]
|
||||
denied=[('get','','secrets','activity-core',''),('get','','secrets','argocd',''),
|
||||
('patch','argoproj.io','applications','argocd','target-revenue'),
|
||||
('patch','apps','deployments','activity-core','actcore-worker'),
|
||||
('create','','pods/exec','activity-core',''),
|
||||
('create','rbac.authorization.k8s.io','rolebindings','activity-core',''),
|
||||
('create','batch','jobs','activity-core','')]
|
||||
for expected,checks in [(True,allowed),(False,denied)]:
|
||||
for verb,group,resource,namespace,name in checks:
|
||||
attrs={'verb':verb,'group':group,'resource':resource,'namespace':namespace}
|
||||
if '/' in resource:attrs['resource'],attrs['subresource']=resource.split('/',1)
|
||||
if name:attrs['name']=name
|
||||
body={'apiVersion':'authorization.k8s.io/v1','kind':'SelfSubjectAccessReview',
|
||||
'spec':{'resourceAttributes':attrs}}
|
||||
result=json.loads(runner(['kubectl','--request-timeout=20s','create',
|
||||
'--raw=/apis/authorization.k8s.io/v1/selfsubjectaccessreviews','-f','-'],
|
||||
input=json.dumps(body),timeout=25))
|
||||
status=result.get('status',{})
|
||||
if status.get('allowed') is not expected or status.get('evaluationError'):
|
||||
raise ValueError('release principal allow/deny boundary unverified')
|
||||
return {'principal_verified':True,'allowed_checks':len(allowed),'denied_checks':len(denied)}
|
||||
|
||||
|
||||
class RuntimeActivities(ReleaseActivities):
|
||||
def __init__(self,admission):
|
||||
self.admission=admission
|
||||
broker=Broker(admission.root/'release.sqlite',Receipts(admission.trust),admitted=True)
|
||||
env=admission.environment()
|
||||
from .release_transport import run
|
||||
def bounded_run(argv,**kwargs):
|
||||
admission.require()
|
||||
inherited=kwargs.get('env',os.environ)
|
||||
clean={key:value for key,value in inherited.items()
|
||||
if not key.startswith(('GIT_','KUBE')) and key!='SSH_AUTH_SOCK'}
|
||||
kwargs['env']={**clean,**env}
|
||||
return run(argv,**kwargs)
|
||||
self.identity_probe=lambda:verify_runtime_identity(bounded_run)
|
||||
self.identity_probe()
|
||||
self.probe=Probe(runner=bounded_run)
|
||||
def factory(plan):
|
||||
admission.require()
|
||||
self.identity_probe()
|
||||
def invariants(target):
|
||||
observed,healthy=self.probe()
|
||||
return observed==target and healthy is True
|
||||
backend=GitArgoBackend(plan,admission.root/'transport',invariants,
|
||||
cluster_uid=CLUSTER_UID,admitted=True,runner=bounded_run)
|
||||
return backend
|
||||
super().__init__(broker,factory,DurableAudit(admission.root/'audit.sqlite'))
|
||||
|
||||
def _plan(self,release_id):
|
||||
try:self.admission.require()
|
||||
except Exception:
|
||||
self.broker.admitted=False
|
||||
raise ApplicationError('release runtime admission unavailable',non_retryable=True) from None
|
||||
return super()._plan(release_id)
|
||||
|
||||
|
||||
async def serve(path):
|
||||
admission=Admission(path)
|
||||
activities=RuntimeActivities(admission)
|
||||
client=await asyncio.wait_for(Client.connect(TEMPORAL_HOST,namespace='default'),timeout=30)
|
||||
await release_worker(client,activities).run()
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--configuration',type=Path,required=True)
|
||||
args=parser.parse_args()
|
||||
try:asyncio.run(serve(args.configuration))
|
||||
except Exception:
|
||||
print(json.dumps({'schema_version':'activity-core-release-runtime/v1','status':'unverified',
|
||||
'error':'runtime unavailable; private configuration and transport details suppressed'}))
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__=='__main__':raise SystemExit(main())
|
||||
136
src/activity_core/release_sampler.py
Normal file
136
src/activity_core/release_sampler.py
Normal file
|
|
@ -0,0 +1,136 @@
|
|||
"""Executable read-only sampler for the existing durable release HealthObserver.
|
||||
|
||||
No signer, identity admission or mutation is provided. Each failed/unavailable
|
||||
probe records False, missed intervals reset, and a new ledger starts now.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import ipaddress
|
||||
import json
|
||||
from pathlib import Path
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
import yaml
|
||||
|
||||
from .release_observer import HealthObserver
|
||||
from .release_operations import revision, update_child
|
||||
from .release_transport import run
|
||||
|
||||
CLUSTER_UID = 'a553c742-0115-43d4-99a4-a5ca56fe0786'
|
||||
BASELINE = 'a86a14332d76d92a7ed176ea34dca4429902c396'
|
||||
DEPLOYMENTS = {'actcore-api', 'actcore-worker', 'actcore-event-router'}
|
||||
RESOURCE_SET = {
|
||||
('ConfigMap', name) for name in ('actcore-runtime-config','actcore-external-activity-definitions','actcore-report-schemas','actcore-ops-service-inventory')
|
||||
} | {('Service',name) for name in ('actcore-api','actcore-worker-metrics')} | {('Deployment',name) for name in DEPLOYMENTS}
|
||||
|
||||
|
||||
def http_health(url):
|
||||
with urllib.request.urlopen(url, timeout=20) as response:
|
||||
if response.status != 200:
|
||||
raise ValueError('invariant service unavailable')
|
||||
body = response.read(16385)
|
||||
if len(body) > 16384:
|
||||
raise ValueError('invariant response exceeds fixed bound')
|
||||
return json.loads(body)
|
||||
|
||||
|
||||
def invariant_matches(value, now):
|
||||
if (not isinstance(value, dict) or value.get('schema_version') != 'activity-core-release-health/v1'
|
||||
or value.get('healthy') is not True or value.get('definitions_and_report_sinks_match') is not True
|
||||
or value.get('schedules_match') is not True or value.get('definition_count') != 3
|
||||
or value.get('schedule_count') != 3 or 'error' in value):
|
||||
return False
|
||||
at = datetime.fromisoformat(value['observed_at'].replace('Z', '+00:00'))
|
||||
# Same estate observer should use synchronized clocks; unknown/future time
|
||||
# is a failed sample, never a manufactured healthy interval.
|
||||
return at.tzinfo is not None and 0 <= (now-at).total_seconds() <= 90
|
||||
|
||||
|
||||
class Probe:
|
||||
"""Only fixed GETs, a cluster-pinned service origin and public boolean health."""
|
||||
def __init__(self, runner=run, reader=http_health):
|
||||
self.runner, self.reader = runner, reader
|
||||
|
||||
def get(self, *args):
|
||||
return json.loads(self.runner(['kubectl','--request-timeout=20s',*args,'-o','json'],timeout=25))
|
||||
|
||||
def __call__(self):
|
||||
nodes = self.get('get','nodes')
|
||||
if '92.205.62.239' not in [a['address'] for n in nodes['items'] for a in n['status']['addresses']]:
|
||||
raise ValueError('estate identity mismatch')
|
||||
if self.get('get','namespace','kube-system')['metadata']['uid'] != CLUSTER_UID:
|
||||
raise ValueError('cluster UID mismatch')
|
||||
app = self.get('-n','argocd','get','application','activity-core')
|
||||
target = revision(app['spec']['source']['targetRevision'])
|
||||
update_child(yaml.safe_dump(app),target,target)
|
||||
status = app.get('status',{})
|
||||
healthy = (not app.get('operation') and status.get('sync',{}).get('status') == 'Synced'
|
||||
and status.get('sync',{}).get('revision') == target
|
||||
and status.get('health',{}).get('status') == 'Healthy' and not status.get('conditions'))
|
||||
resources = status.get('resources',[])
|
||||
healthy &= (len(resources) == 9 and {(r['kind'],r['name']) for r in resources} == RESOURCE_SET
|
||||
and all(r.get('namespace') == 'activity-core' and r.get('status') == 'Synced'
|
||||
and (r.get('group') or '') == ('apps' if r['kind']=='Deployment' else '') for r in resources))
|
||||
for name in sorted(DEPLOYMENTS):
|
||||
deployment = self.get('-n','activity-core','get','deployment',name)
|
||||
desired = deployment['spec'].get('replicas',1);ds=deployment.get('status',{})
|
||||
healthy &= (type(desired) is int and desired>0 and ds.get('observedGeneration') == deployment['metadata']['generation']
|
||||
and all(ds.get(k,0) == desired for k in ('readyReplicas','updatedReplicas','availableReplicas')))
|
||||
service = self.get('-n','activity-core','get','service','actcore-api')
|
||||
address = ipaddress.ip_address(service['spec']['clusterIP'])
|
||||
if not any(p.get('port') == 8010 for p in service['spec'].get('ports',[])):
|
||||
raise ValueError('fixed invariant API port unavailable')
|
||||
host = f'[{address}]' if address.version == 6 else str(address)
|
||||
value = self.reader(f'http://{host}:8010/release/health')
|
||||
healthy &= invariant_matches(value,datetime.now(timezone.utc))
|
||||
return target,bool(healthy)
|
||||
|
||||
|
||||
class Sampler:
|
||||
def __init__(self, database, probe, *, anchor_revision=BASELINE):
|
||||
self.observer=HealthObserver(database,max_gap_seconds=90)
|
||||
self.probe=probe;self.anchor=revision(anchor_revision)
|
||||
|
||||
def sample(self,now=None):
|
||||
explicit_now=now
|
||||
try:
|
||||
target,healthy=self.probe()
|
||||
revision(target)
|
||||
if type(healthy) is not bool:raise ValueError('explicit health required')
|
||||
except Exception:
|
||||
# Use the last durably seen identity only for a failed sample. No
|
||||
# raw error, fabricated timestamp or successful identity is emitted.
|
||||
with self.observer.connect() as db:
|
||||
row=db.execute('SELECT revision FROM observations ORDER BY rowid DESC LIMIT 1').fetchone()
|
||||
target=row[0] if row else self.anchor;healthy=False
|
||||
# Measure after the probe, not before a slow/failed network operation.
|
||||
measured=explicit_now if explicit_now is not None else datetime.now(timezone.utc)
|
||||
since=self.observer.observe(target,healthy,measured)
|
||||
try:attestation=self.observer.attestation(target,measured)
|
||||
except ValueError:attestation=None
|
||||
return {'schema_version':'activity-core-release-sample/v1','revision':target,
|
||||
'observed_at':measured.isoformat(),'healthy':healthy,'healthy_since':since,
|
||||
'continuous_24h_ready':attestation is not None,'attestation':attestation,
|
||||
'native_mutation':False}
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--database',type=Path,required=True)
|
||||
parser.add_argument('--receipt',type=Path,required=True)
|
||||
parser.add_argument('--watch',action='store_true',help='read-only 60s sampler; requires an owner-deployed process')
|
||||
args=parser.parse_args()
|
||||
sampler=Sampler(args.database,Probe())
|
||||
while True:
|
||||
try:result=sampler.sample()
|
||||
except Exception:result={'schema_version':'activity-core-release-sample/v1','healthy':False,'continuous_24h_ready':False,'error':'durable observation unavailable','native_mutation':False}
|
||||
args.receipt.write_text(json.dumps(result,sort_keys=True)+'\n')
|
||||
if not args.watch:
|
||||
print(json.dumps(result,sort_keys=True));return 0 if result.get('healthy') else 1
|
||||
time.sleep(60)
|
||||
|
||||
|
||||
if __name__=='__main__':raise SystemExit(main())
|
||||
124
tests/test_release_health.py
Normal file
124
tests/test_release_health.py
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
import copy
|
||||
from datetime import timedelta
|
||||
from types import SimpleNamespace
|
||||
from uuid import UUID
|
||||
|
||||
import pytest
|
||||
from temporalio.client import (Schedule, ScheduleActionStartWorkflow, ScheduleCalendarSpec,
|
||||
SchedulePolicy, ScheduleRange, ScheduleSpec, ScheduleState, ScheduleOverlapPolicy)
|
||||
from temporalio.converter import DataConverter
|
||||
from temporalio.api.common.v1 import Payloads, WorkflowType
|
||||
from temporalio.api.taskqueue.v1 import TaskQueue
|
||||
from temporalio.api.workflow.v1 import NewWorkflowExecutionInfo
|
||||
|
||||
from activity_core.release_health import FRONTEND, SINK, definition_matches, probe, schedule_matches
|
||||
|
||||
|
||||
def rows():
|
||||
return [SimpleNamespace(id=UUID(identity),enabled=True,trigger_type='cron',
|
||||
trigger_config={'cron_expression':cron,'timezone':'Europe/Berlin','misfire_policy':'catchup_latest','catchup_window_seconds':86400},
|
||||
instructions_json=[{'id':f'frontend-patterns-{cadence}-report','model':'deterministic','report_sinks':[dict(SINK)]}])
|
||||
for identity,(cadence,cron) in FRONTEND.items()]
|
||||
|
||||
|
||||
def description(identity,calendar=False):
|
||||
cadence,cron=FRONTEND[identity]
|
||||
action=ScheduleActionStartWorkflow('RunActivityWorkflow',args=[identity,'scheduled',None],
|
||||
id=f'activity-{identity}:${{firstScheduledTime}}',task_queue='orchestrator-tq')
|
||||
if calendar:
|
||||
cal=ScheduleCalendarSpec(hour=[ScheduleRange({'daily':2,'weekly':3,'monthly':9}[cadence])],
|
||||
day_of_week=[ScheduleRange(2)] if cadence=='weekly' else [ScheduleRange(0,6)],
|
||||
day_of_month=[ScheduleRange(1)] if cadence=='monthly' else [ScheduleRange(1,31)])
|
||||
spec=ScheduleSpec(calendars=[cal],time_zone_name='Europe/Berlin')
|
||||
else:spec=ScheduleSpec(cron_expressions=[cron],time_zone_name='Europe/Berlin')
|
||||
schedule=Schedule(action=action,spec=spec,policy=SchedulePolicy(overlap=ScheduleOverlapPolicy.BUFFER_ONE,
|
||||
catchup_window=timedelta(days=1)),state=ScheduleState(paused=False))
|
||||
return SimpleNamespace(schedule=schedule)
|
||||
|
||||
|
||||
def sources(items=None,descriptions=None):
|
||||
data=items if items is not None else rows()
|
||||
class Session:
|
||||
async def __aenter__(self):return self
|
||||
async def __aexit__(self,*_):pass
|
||||
async def scalars(self,statement):
|
||||
assert len(statement.compile().params['id_1'])==3
|
||||
return SimpleNamespace(all=lambda:data)
|
||||
descriptions=descriptions or {identity:description(identity,calendar=True) for identity in FRONTEND}
|
||||
class Temporal:
|
||||
data_converter=DataConverter.default
|
||||
def get_schedule_handle(self,sid):
|
||||
assert sid.startswith('activity-schedule-')
|
||||
async def describe():return descriptions[sid.removeprefix('activity-schedule-')]
|
||||
return SimpleNamespace(describe=describe)
|
||||
return Session,Temporal()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('calendar',[False,True])
|
||||
async def test_real_invariants_accept_exact_cron_and_server_normalized_calendar(calendar):
|
||||
session,temporal=sources(descriptions={identity:description(identity,calendar) for identity in FRONTEND})
|
||||
result=await probe(session,temporal)
|
||||
assert result['healthy'] is True and result['definition_count']==result['schedule_count']==3
|
||||
assert set(result)=={'schema_version','observed_at','healthy','definition_count','schedule_count','definitions_and_report_sinks_match','schedules_match'}
|
||||
|
||||
|
||||
async def test_actual_describe_payload_arguments_are_decoded_with_bound_client_converter():
|
||||
descriptions={identity:description(identity,True) for identity in FRONTEND}
|
||||
for identity,value in descriptions.items():
|
||||
encoded=await DataConverter.default.encode([identity,'scheduled',None])
|
||||
raw=NewWorkflowExecutionInfo(workflow_id=f'activity-{identity}:${{firstScheduledTime}}',
|
||||
workflow_type=WorkflowType(name='RunActivityWorkflow'),task_queue=TaskQueue(name='orchestrator-tq'),input=Payloads(payloads=encoded))
|
||||
value.schedule.action=ScheduleActionStartWorkflow('<unused>',raw_info=raw)
|
||||
session,temporal=sources(descriptions=descriptions)
|
||||
assert (await probe(session,temporal))['healthy'] is True
|
||||
|
||||
|
||||
@pytest.mark.parametrize('field,value',[('enabled',False),('trigger_type','event'),
|
||||
('instructions_json',[{'model':'llm','report_sinks':[SINK]}])])
|
||||
async def test_definition_behavior_drift_refused(field,value):
|
||||
data=rows();setattr(data[0],field,value)
|
||||
assert (await probe(*sources(items=data)))['healthy'] is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize('field,value',[('cron_expression','* * * * *'),('timezone','UTC'),
|
||||
('misfire_policy','catchup_all'),('catchup_window_seconds',3600),('jitter_seconds',5)])
|
||||
def test_trigger_scope_drift_refused(field,value):
|
||||
row=rows()[0];row.trigger_config[field]=value
|
||||
assert not definition_matches(row)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('change',['pause','limit','extra_calendar','interval','skip','cron','queue','args','sink','policy','timezone'])
|
||||
async def test_schedule_and_report_drift_fail_closed(change):
|
||||
data=rows();descriptions={identity:description(identity,True) for identity in FRONTEND}
|
||||
identity=next(iter(FRONTEND));s=descriptions[identity].schedule
|
||||
if change=='pause':s.state.paused=True
|
||||
elif change=='limit':s.state.limited_actions=True
|
||||
elif change=='extra_calendar':s.spec.calendars.append(ScheduleCalendarSpec())
|
||||
elif change=='interval':s.spec.intervals=[object()]
|
||||
elif change=='skip':s.spec.skip=[ScheduleCalendarSpec()]
|
||||
elif change=='cron':s.spec.calendars[0].hour=[ScheduleRange(5)]
|
||||
elif change=='queue':s.action.task_queue='privileged-other-queue'
|
||||
elif change=='args':s.action.args=['other-identity','scheduled',None]
|
||||
elif change=='sink':data[0].instructions_json[0]['report_sinks'][0]['topic_id']='foreign'
|
||||
elif change=='policy':s.policy.overlap=ScheduleOverlapPolicy.ALLOW_ALL
|
||||
elif change=='timezone':s.spec.time_zone_name='UTC'
|
||||
assert (await probe(*sources(items=data,descriptions=descriptions)))['healthy'] is False
|
||||
|
||||
|
||||
async def test_missing_duplicate_rows_and_private_exceptions_are_not_healthy_or_leaked():
|
||||
for data in [[],rows()[:2],[rows()[0]]*3]:assert not (await probe(*sources(items=data)))['healthy']
|
||||
def unavailable():raise RuntimeError('PRIVATE-DATABASE-PASSWORD-CANARY')
|
||||
result=await probe(unavailable,sources()[1])
|
||||
assert not result['healthy'] and 'CANARY' not in str(result)
|
||||
|
||||
|
||||
def test_release_health_endpoint_is_read_only_and_uses_bound_dependencies(monkeypatch):
|
||||
from fastapi.testclient import TestClient
|
||||
from activity_core import api
|
||||
session,temporal=sources()
|
||||
monkeypatch.setattr(api,'_session_factory',session)
|
||||
monkeypatch.setattr(api,'_temporal_client',temporal)
|
||||
client=TestClient(api.app)
|
||||
result=client.get('/release/health')
|
||||
assert result.status_code==200 and result.json()['healthy']
|
||||
assert client.post('/release/health',json={'operation':'trigger'}).status_code==405
|
||||
84
tests/test_release_runtime.py
Normal file
84
tests/test_release_runtime.py
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
import base64
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
|
||||
from cryptography.hazmat.primitives.serialization import Encoding,PublicFormat
|
||||
|
||||
from activity_core.release_runtime import Admission, CLUSTER_UID, DurableAudit, verify_runtime_identity
|
||||
|
||||
|
||||
def configuration(root):
|
||||
kube=root/'kubeconfig';kube.write_text('fixture private kube configuration');kube.chmod(0o600)
|
||||
git=root/'gitconfig';git.write_text('fixture private git configuration');git.chmod(0o600)
|
||||
trust={}
|
||||
for role in ('build','review','health','retention'):
|
||||
public=Ed25519PrivateKey.generate().public_key().public_bytes(Encoding.Raw,PublicFormat.Raw)
|
||||
trust[role]={'public_key':base64.b64encode(public).decode(),'principal':role,'roles':[role]}
|
||||
value={'schema_version':'activity-core-release-runtime/v1','enabled':True,'principal':'activity-core-release-broker',
|
||||
'cluster_uid':CLUSTER_UID,'state_directory':str(root/'state'),'kubeconfig':str(kube),'git_config':str(git),'receipt_trust':trust}
|
||||
path=root/'runtime.json';path.write_text(json.dumps(value));path.chmod(0o600)
|
||||
return path,value
|
||||
|
||||
|
||||
def test_operator_private_config_drift_disable_and_custody_replacement_refused(tmp_path):
|
||||
path,value=configuration(tmp_path);admission=Admission(path);admission.require()
|
||||
assert admission.environment()['GIT_CONFIG_NOSYSTEM']=='1'
|
||||
value['enabled']=False;path.write_text(json.dumps(value))
|
||||
with pytest.raises(ValueError,match='disabled'):admission.require()
|
||||
value['enabled']=True;value['principal']='borrowed-admin';path.write_text(json.dumps(value))
|
||||
with pytest.raises(ValueError):admission.require()
|
||||
value['principal']='activity-core-release-broker';path.write_text(json.dumps(value))
|
||||
Path(value['kubeconfig']).write_text('replaced fixture custody')
|
||||
with pytest.raises(ValueError,match='custody'):admission.require()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('delta',[{'enabled':False},{'principal':'operator'},{'cluster_uid':'other'},
|
||||
{'schema_version':'other'},{'untrusted_transport_factory':'arbitrary-import'}])
|
||||
def test_disabled_widened_configuration_cannot_construct_runtime(tmp_path,delta):
|
||||
path,value=configuration(tmp_path);path.write_text(json.dumps(value|delta))
|
||||
with pytest.raises(ValueError):Admission(path)
|
||||
|
||||
|
||||
def test_public_permissions_symlink_or_incomplete_role_registry_refused(tmp_path):
|
||||
path,value=configuration(tmp_path);path.chmod(0o644)
|
||||
with pytest.raises(ValueError,match='private'):Admission(path)
|
||||
path.chmod(0o600);alias=tmp_path/'symlink.json';alias.symlink_to(path)
|
||||
with pytest.raises(ValueError,match='private'):Admission(alias)
|
||||
del value['receipt_trust']['health'];path.write_text(json.dumps(value))
|
||||
with pytest.raises(ValueError,match='independent'):Admission(path)
|
||||
|
||||
|
||||
def identity_runner(*,admin=False,extra_access=False,wrong_cluster=False):
|
||||
calls=[]
|
||||
def run(argv,**kwargs):
|
||||
calls.append(argv)
|
||||
if 'namespace' in argv:return json.dumps({'metadata':{'uid':'wrong' if wrong_cluster else CLUSTER_UID}})
|
||||
if 'whoami' in argv:return json.dumps({'status':{'userInfo':{'username':'root' if admin else 'system:serviceaccount:activity-core:activity-core-release-broker','groups':['system:masters'] if admin else ['system:serviceaccounts']}}})
|
||||
assert '--raw=/apis/authorization.k8s.io/v1/selfsubjectaccessreviews' in argv
|
||||
attrs=json.loads(kwargs['input'])['spec']['resourceAttributes']
|
||||
allowed=attrs['resource']=='applications' and attrs.get('name') in {'activity-core','railiance-apps-root'}
|
||||
if extra_access and attrs['resource']=='secrets':allowed=True
|
||||
return json.dumps({'status':{'allowed':allowed}})
|
||||
return run,calls
|
||||
|
||||
|
||||
def test_actual_subject_scope_and_denial_checks_are_required_before_dispatch():
|
||||
run,calls=identity_runner();result=verify_runtime_identity(run)
|
||||
assert result=={'principal_verified':True,'allowed_checks':4,'denied_checks':7}
|
||||
assert len(calls)==13
|
||||
for kwargs in [{'admin':True},{'extra_access':True},{'wrong_cluster':True}]:
|
||||
run,calls=identity_runner(**kwargs)
|
||||
with pytest.raises(ValueError):verify_runtime_identity(run)
|
||||
if kwargs.get('wrong_cluster'):assert len(calls)==1
|
||||
|
||||
|
||||
def test_audit_sink_durably_deduplicates_and_refuses_collision_or_private_fields(tmp_path):
|
||||
sink=DurableAudit(tmp_path/'audit.sqlite')
|
||||
event={'event_id':'release:id:1','event_type':'release_transition','release_id':'id','application':'activity-core',
|
||||
'phase':'complete','observed_at':'2026-10-11T20:00:00Z','candidate':'a'*40,'rollback':'b'*40}
|
||||
sink(event);DurableAudit(tmp_path/'audit.sqlite')(event)
|
||||
with sink.connect() as db:assert db.execute('SELECT count(*) FROM events').fetchone()[0]==1
|
||||
with pytest.raises(ValueError,match='conflict'):sink(event|{'phase':'failed'})
|
||||
with pytest.raises(ValueError):sink(event|{'credential':'secret fixture canary'})
|
||||
101
tests/test_release_sampler.py
Normal file
101
tests/test_release_sampler.py
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
from datetime import datetime,timezone,timedelta
|
||||
import pytest
|
||||
from activity_core.release_sampler import BASELINE, Sampler, invariant_matches
|
||||
|
||||
NOW=datetime(2026,10,11,20,tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def test_gap_sensitive_actual_sample_interval_survives_restart_without_backfill(tmp_path):
|
||||
database=tmp_path/'health.sqlite'
|
||||
sampler=Sampler(database,lambda:(BASELINE,True))
|
||||
assert sampler.sample(NOW)['continuous_24h_ready'] is False
|
||||
later=NOW+timedelta(days=1)
|
||||
assert sampler.sample(later)['healthy_since']==later.isoformat()
|
||||
for minute in range(1,1441):sampler.sample(later+timedelta(minutes=minute))
|
||||
restarted=Sampler(database,lambda:(BASELINE,True))
|
||||
result=restarted.sample(later+timedelta(days=1,seconds=60))
|
||||
assert result['continuous_24h_ready'] and result['attestation']['continuous']
|
||||
|
||||
|
||||
def test_unavailable_probe_records_failure_and_resets_interval(tmp_path):
|
||||
sampler=Sampler(tmp_path/'health.sqlite',lambda:(BASELINE,True))
|
||||
sampler.sample(NOW)
|
||||
def unavailable():raise RuntimeError('PRIVATE-TOKEN-CANARY')
|
||||
sampler.probe=unavailable
|
||||
result=sampler.sample(NOW+timedelta(minutes=1))
|
||||
assert result['healthy'] is False and result['healthy_since'] is None
|
||||
assert 'CANARY' not in str(result)
|
||||
sampler.probe=lambda:(BASELINE,True)
|
||||
result=sampler.sample(NOW+timedelta(minutes=2))
|
||||
assert result['healthy_since']==(NOW+timedelta(minutes=2)).isoformat()
|
||||
|
||||
|
||||
def test_revision_and_clock_change_refuse_old_observation(tmp_path):
|
||||
sampler=Sampler(tmp_path/'health.sqlite',lambda:(BASELINE,True));sampler.sample(NOW)
|
||||
with pytest.raises(ValueError):sampler.sample(NOW)
|
||||
sampler.probe=lambda:('b'*40,True)
|
||||
assert sampler.sample(NOW+timedelta(minutes=1))['healthy_since']==(NOW+timedelta(minutes=1)).isoformat()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('delta',[{'healthy':False},{'schedules_match':False},{'definition_count':2},
|
||||
{'observed_at':(NOW-timedelta(seconds=91)).isoformat()},{'observed_at':(NOW+timedelta(seconds=1)).isoformat()},
|
||||
{'error':'private canary'},{'schema_version':'other'}])
|
||||
def test_invariant_response_cannot_manufacture_health(delta):
|
||||
value={'schema_version':'activity-core-release-health/v1','healthy':True,'schedules_match':True,
|
||||
'definitions_and_report_sinks_match':True,'definition_count':3,'schedule_count':3,'observed_at':NOW.isoformat()}
|
||||
assert not invariant_matches(value|delta,NOW)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def native_probe():
|
||||
import json
|
||||
from activity_core.release_sampler import Probe,CLUSTER_UID,RESOURCE_SET
|
||||
from activity_core.release_transport import ACTIVITY_URL
|
||||
app={'apiVersion':'argoproj.io/v1alpha1','kind':'Application',
|
||||
'metadata':{'name':'activity-core','namespace':'argocd'},
|
||||
'spec':{'project':'activity-core','source':{'repoURL':ACTIVITY_URL,'path':'k8s/gitops','targetRevision':BASELINE},
|
||||
'destination':{'namespace':'activity-core','server':'https://kubernetes.default.svc'},
|
||||
'syncPolicy':{'syncOptions':['CreateNamespace=false','ApplyOutOfSyncOnly=true','PruneLast=true','FailOnSharedResource=true']}},
|
||||
'status':{'sync':{'status':'Synced','revision':BASELINE},'health':{'status':'Healthy'},
|
||||
'resources':[{'kind':kind,'name':name,'namespace':'activity-core','status':'Synced','group':'apps' if kind=='Deployment' else ''} for kind,name in RESOURCE_SET]}}
|
||||
deployment={'metadata':{'generation':1},'spec':{'replicas':1},'status':{'observedGeneration':1,'readyReplicas':1,'updatedReplicas':1,'availableReplicas':1}}
|
||||
objects={'nodes':{'items':[{'status':{'addresses':[{'address':'92.205.62.239'}]}}]},
|
||||
'namespace':{'metadata':{'uid':CLUSTER_UID}},'application':app,'deployment':deployment,
|
||||
'service':{'spec':{'clusterIP':'10.43.1.2','ports':[{'port':8010}]}}}
|
||||
calls=[]
|
||||
def runner(argv,**kwargs):
|
||||
calls.append(argv)
|
||||
assert 'get' in argv and not any(verb in argv for verb in ['patch','apply','create','secrets'])
|
||||
return json.dumps(objects[argv[argv.index('get')+1]])
|
||||
def reader(url):
|
||||
assert url=='http://10.43.1.2:8010/release/health'
|
||||
return {'schema_version':'activity-core-release-health/v1','healthy':True,'schedules_match':True,
|
||||
'definitions_and_report_sinks_match':True,'definition_count':3,'schedule_count':3,
|
||||
'observed_at':datetime.now(timezone.utc).isoformat()}
|
||||
return Probe(runner,reader),objects,calls
|
||||
|
||||
|
||||
def test_native_probe_fixed_reads_and_invariants(native_probe):
|
||||
probe,_,calls=native_probe
|
||||
assert probe()==(BASELINE,True)
|
||||
assert len(calls)==7
|
||||
|
||||
|
||||
@pytest.mark.parametrize('change',['estate','uid','automated','missing_resource','extra_resource','wrong_group','wrong_namespace','stale_generation','unready','revision','operation'])
|
||||
def test_native_probe_drift_fails_closed(native_probe,change):
|
||||
probe,objects,_=native_probe
|
||||
app=objects['application']
|
||||
if change=='estate':objects['nodes']['items'][0]['status']['addresses'][0]['address']='192.0.2.1'
|
||||
elif change=='uid':objects['namespace']['metadata']['uid']='foreign-cluster'
|
||||
elif change=='automated':app['spec']['syncPolicy']['automated']={}
|
||||
elif change=='missing_resource':app['status']['resources'].pop()
|
||||
elif change=='extra_resource':app['status']['resources'].append(dict(app['status']['resources'][0]))
|
||||
elif change=='wrong_group':app['status']['resources'][0]['group']='foreign'
|
||||
elif change=='wrong_namespace':app['status']['resources'][0]['namespace']='foreign'
|
||||
elif change=='stale_generation':objects['deployment']['status']['observedGeneration']=0
|
||||
elif change=='unready':objects['deployment']['status']['readyReplicas']=0
|
||||
elif change=='revision':app['status']['sync']['revision']='b'*40
|
||||
elif change=='operation':app['operation']={'sync':{}}
|
||||
try:_,healthy=probe()
|
||||
except (ValueError,KeyError):return
|
||||
assert healthy is False
|
||||
|
|
@ -84,7 +84,9 @@ on the WP-0042 points; see that plan.
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0041-T03
|
||||
status: todo
|
||||
status: wait
|
||||
blocked_on: message-from:railiance-platform
|
||||
blocking_reason: "RPF-IN-0011: return dedicated release principal/custody owner task and native acceptance; D2 scope already accepted"
|
||||
decision_id: 89f1058f-33bc-47a8-8a49-0b44d426c67e
|
||||
priority: high
|
||||
state_hub_task_id: "8e2b4ed9-d455-5367-85d9-8e4222ce75f3"
|
||||
|
|
@ -107,6 +109,20 @@ and produce visible evidence rather than asking the founder to approve each run.
|
|||
Prove one normal release and one failed health-check rollback through Git/ArgoCD.
|
||||
|
||||
2026-10-03: human gate routed to hub decision 89f1058f-33bc-47a8-8a49-0b44d426c67e (D2: ArgoCD owns production: go-ahead sequence for railiance-apps-root, issue-core, policy-nexus and activity-core automation) by CUST-WP-0075; resolving that decision raises C-40 here.
|
||||
2026-10-11 owner implementation: concrete read-only definition/sink/Temporal
|
||||
invariant API, bounded native metadata probe, durable 60-second health sampler and
|
||||
private dedicated Temporal runtime entrypoint implemented; 113 scoped tests pass.
|
||||
Actual username plus fixed allow/deny checks reject borrowed admin credentials and
|
||||
config/custody drift stops dispatch. Native metadata confirms the dedicated release
|
||||
account is absent; only default and ESO identities exist. No activation or borrowed
|
||||
credential was attempted. Independent Platform preparation intake RPF-IN-0011
|
||||
requests native Git/Kubernetes custody, issuer and rollback acceptance within the
|
||||
existing D2 image-only scope. Await its exact owner task binding, then upgrade
|
||||
blocked_on to depends_on; do not depend on the RPF48 beneficiary. Cross-owner case
|
||||
ACT41-release-custody-2026-10-11 retained with source validation receipt. No new
|
||||
founder approval or unqualified needs_human gate is introduced. See
|
||||
docs/release-runtime-owner-handoff.md and docs/evidence/2026-10-11-release-runtime/.
|
||||
|
||||
|
||||
## Acceptance
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue