feat(identity): add PostgreSQL provenance and verify sealed SOPS custody
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Identity provider journey acceptance / provider (push) Failing after 13s
Build and Publish identity-provisioner / build-and-push (push) Successful in 26s

Assistant: codex
Assistant-Model: gpt-6.1-sol
Assistant-Session: 01a1468b-63fa-7710-85dc-8f2e16b3ebfa
This commit is contained in:
tegwick 2026-10-17 01:36:38 +02:00
parent 6cb43c0cc5
commit 70f408ca23
12 changed files with 810 additions and 27 deletions

View file

@ -80,6 +80,64 @@ unresolved intents; losing a journal never authorizes replacement initialization
and replay. No arbitrary JSON import, receipt expiry or automatic uncertainty
clearing is introduced. Owner reconciliation needs independent evidence.
## PostgreSQL source seam (not native admission)
`PostgresGroupProvenance` implements the same reserve/get/claim-revoke/save/check
journal contract. It is injection-only: the ordinary HTTP server has no
PostgreSQL environment switch and remains without a group store unless the
existing local SQLite configuration is explicitly supplied. `GROUP_POSTGRES_DSN`
or a caller JSON object cannot enable the PostgreSQL path. Production group
operations remain unadmitted and disabled pending the owner callbacks and gates
above; a backend constructor is not an access decision.
A trusted owner integration must provide an exact `GroupJournalBinding` with
opaque authoritative tenant scope, directory instance, accepted case digest,
source digest, represented principal and journal epoch. There is no guessed
platform-root scope or request-derived tenant fallback. The owner also injects a
renewable private connection-info callback using the existing `rapp-postgres`
consumer/lease contract, and independently durable `JournalContinuity` callbacks.
These callbacks retain/check/advance exact epoch+directory+head+commit digest outside the
PostgreSQL state that might be restored. Neither callback authenticates a caller
or grants a protected side effect. Their native custody, source binding,
revocation and execution bounds still require independent admission. Only the
disposable fixture supplies a test checkpoint implementation here.
The v1 migration under `identity-provisioner/migrations/` creates tables within
the existing consumer schema; it creates no database/schema/roles, inserts no
epoch and clears no quarantine. Startup does not run migrations or initialize
missing state. The owner must separately admit an epoch/checkpoint after
independent continuity/recovery review; absent or quarantined state refuses.
A restored database with an older head than the independently retained checkpoint
also refuses rather than treating missing later intents as authority to replay.
PostgreSQL transactions lock the single journal metadata row, commit bounded
receipts/intents and advance the head and exact transition digest with an exactly-one conditional
update. The digest binds the prior marker, canonical body, context, request key
and receipt version; the independent checkpoint must compare/CAS both content
and sequence, never accept a counter-only callback. Divergent restored clones
that commit different transitions at the same counter cannot both be accepted.
Every receipt update also requires its exact context/key/request/version and
increments its version. The active slot spans all contexts in the journal;
it serializes cooperating producer operations, not external LDAP writers.
SQL commit precedes independently durable checkpoint acknowledgment. A lost
commit/acknowledgment, stale version, missing row or continuity mismatch refuses
and poisons that backend object; exception handling cannot automatically rewrite
its journal into success. Known request-binding mismatches remain domain
refusals. Definite before-mutation refusal records can release the active slot
while the original key remains terminal.
The container source includes a hash-pinned Linux amd64 CPython3.12 psycopg3
closure selected from User Engine's existing lockfile. No image build or package
installation is part of this source proof; build/publication, artifact scanning
and native runtime/dependency admission remain owner work. SQLite is still
local/disposable only. The separate PostgreSQL fixture proposal uses a cached
pinned PostgreSQL image and a cached pinned User Engine Python image, real SQL,
separate-process reopen, concurrency, stale-CAS and actual pg_dump/pg_restore
behind an independently retained test checkpoint. That fixture needs exact
review before Docker execution and does not prove live CNPG HA, TLS/lease
custody, tenant authority, external writer fencing, production retention or
protected session/evidence integration.
## Safety and remaining native acceptance
Before each directory mutation, durable intent is committed. A successful

View file

@ -3,7 +3,10 @@ FROM python:3.12-slim@sha256:d764629ce0ddd8c71fd371e9901efb324a95789d2315a47db7e
RUN useradd --create-home --uid 10001 provisioner
WORKDIR /app
COPY --from=lldap /app/lldap_set_password /app/lldap_set_password
COPY provisioner.py group_provenance.py password_setup.py server.py /app/
COPY requirements-postgres.txt /app/
RUN python -m pip install --no-cache-dir --require-hashes -r /app/requirements-postgres.txt
COPY provisioner.py group_provenance.py group_provenance_postgres.py password_setup.py server.py /app/
COPY migrations/ /app/migrations/
USER 10001:10001
EXPOSE 8080
CMD ["python", "server.py"]

View file

@ -11,10 +11,19 @@ import os
from pathlib import Path
import sqlite3
import stat
from typing import Any
from typing import Any, Protocol
from urllib.parse import quote
class GroupJournalStore(Protocol):
"""Owner-injected persistence only; grants no caller or native authority."""
def check(self) -> None: ...
def reserve(self, key: str, binding: str) -> tuple[dict[str, Any], bool]: ...
def get(self, key: str, binding: str) -> dict[str, Any]: ...
def claim_revoke(self, key: str, binding: str, delete_if_empty: bool) -> dict[str, Any]: ...
def save(self, receipt: dict[str, Any]) -> None: ...
class GroupProvenanceUnavailable(RuntimeError):
def __init__(self):
super().__init__("group_provenance_unavailable")

View file

@ -0,0 +1,253 @@
"""Injection-only PostgreSQL implementation of the existing owner journal seam.
Binding and continuity callbacks are supplied by a trusted owner integration,
never constructed from an HTTP payload. Neither is an authorization decision.
No server environment switch, automatic schema creation or recovery admission.
"""
from __future__ import annotations
from contextlib import contextmanager
from dataclasses import asdict, dataclass
import hashlib
import json
import re
from typing import Any, Callable, Protocol
from uuid import UUID
from group_provenance import GroupProvenance, GroupProvenanceUnavailable
@dataclass(frozen=True)
class GroupJournalBinding:
tenant_scope: str
directory: str
case_digest: str
source_digest: str
principal: str
epoch: str
def __post_init__(self):
for value in (self.tenant_scope, self.directory, self.principal):
if not isinstance(value, str) or not 1 <= len(value) <= 256 or any(ord(c) < 32 for c in value):
raise ValueError("explicit owner journal binding required")
for value in (self.case_digest, self.source_digest):
if not isinstance(value, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", value):
raise ValueError("exact owner source and case digests required")
if not isinstance(self.epoch, str) or str(UUID(self.epoch)) != self.epoch:
raise ValueError("explicit canonical journal epoch required")
@property
def digest(self):
return hashlib.sha256(json.dumps(asdict(self), sort_keys=True, separators=(",", ":")).encode()).hexdigest()
def journal_genesis(epoch: str, directory: str) -> str:
"""Deterministic initial marker for explicit owner migration/admission only."""
return hashlib.sha256(json.dumps({"schema": "nk-group-journal-genesis.v1", "epoch": epoch,
"directory": directory}, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
class JournalContinuity(Protocol):
"""Existing independently retained owner checkpoint, outside restored SQL state.
Implementations compare/advance exact epoch+directory+head+commit digest atomically and
durably. An in-memory implementation is a fixture only. Missing/stale or
ambiguous checkpoints must refuse; no callback grants mutation authority.
"""
def check(self, epoch: str, directory: str, head: int, digest: str) -> bool: ...
def advance(self, epoch: str, directory: str, previous: int, previous_digest: str, current: int, current_digest: str) -> bool: ...
class PostgresGroupProvenance:
def __init__(self, *, connect: Callable, binding: GroupJournalBinding,
continuity: JournalContinuity, capacity: int = 256):
if not callable(connect) or not isinstance(binding, GroupJournalBinding) or not 1 <= capacity <= 4096:
raise ValueError("explicit owner connector, binding and bounded capacity required")
if not callable(getattr(continuity, "check", None)) or not callable(getattr(continuity, "advance", None)):
raise ValueError("independent owner continuity callbacks required")
self.connect, self.binding, self.continuity, self.capacity = connect, binding, continuity, capacity
self._poisoned = False
def _continuity(self, method, *args):
try:
if getattr(self.continuity, method)(self.binding.epoch, self.binding.directory, *args) is not True:
raise GroupProvenanceUnavailable()
except Exception:
raise GroupProvenanceUnavailable() from None
@contextmanager
def _transaction(self):
if self._poisoned:
raise GroupProvenanceUnavailable()
connection = None
tx = None
try:
connection = self.connect()
with connection:
with connection.cursor() as cursor:
cursor.execute("SET LOCAL statement_timeout = '5s'")
cursor.execute("SET LOCAL idle_in_transaction_session_timeout = '15s'")
cursor.execute("SELECT epoch::text, directory, head, head_digest, quarantined FROM identity_provisioner.group_journal_meta WHERE singleton = true FOR UPDATE")
meta = cursor.fetchone()
if not meta or meta[0] != self.binding.epoch or meta[1] != self.binding.directory or meta[4] is not False or type(meta[2]) is not int or meta[2] < 0 or not isinstance(meta[3], str) or not re.fullmatch(r"[0-9a-f]{64}", meta[3]):
raise GroupProvenanceUnavailable()
self._continuity("check", meta[2], meta[3])
tx = {"cursor": cursor, "head": meta[2], "digest": meta[3], "dirty": False}
yield tx
if tx["dirty"]:
cursor.execute("UPDATE identity_provisioner.group_journal_meta SET head = head + 1, head_digest = %s WHERE singleton = true AND epoch = %s AND directory = %s AND head = %s AND head_digest = %s AND quarantined = false",
(tx["next_digest"], self.binding.epoch, self.binding.directory, tx["head"], tx["digest"]))
if cursor.rowcount != 1:
raise GroupProvenanceUnavailable()
# SQL commit precedes the independent checkpoint acknowledgment.
# A lost acknowledgment refuses before the next directory effect.
if tx and tx["dirty"]:
self._continuity("advance", tx["head"], tx["digest"], tx["head"] + 1, tx["next_digest"])
except ValueError:
raise
except GroupProvenanceUnavailable:
# Caller mismatches are domain refusals. Storage/continuity failures
# poison this object; durable pending records are never rewritten
# automatically by an exception handler after ambiguous commit.
self._poisoned = True
raise
except Exception:
self._poisoned = True
raise GroupProvenanceUnavailable() from None
finally:
if connection is not None:
try:
connection.close()
except Exception:
self._poisoned = True
raise GroupProvenanceUnavailable() from None
def check(self):
with self._transaction():
pass
def _read(self, cursor, key, request_binding):
cursor.execute("SELECT binding, version, body, context_json FROM identity_provisioner.group_receipts WHERE context_digest = %s AND key = %s",
(self.binding.digest, key))
row = cursor.fetchone()
if row is None:
return None
if row[0] != request_binding:
raise ValueError("ensure receipt request binding mismatch")
if row[3] != json.dumps(asdict(self.binding), sort_keys=True):
raise GroupProvenanceUnavailable()
receipt = GroupProvenance._load(row[2], key, request_binding)
if type(row[1]) is not int or row[1] < 1 or type(receipt.get("_pg_version")) is not int or receipt.get("_pg_version") != row[1] or receipt.get("_pg_context") != self.binding.digest:
raise GroupProvenanceUnavailable()
return receipt
@staticmethod
def _inputs(key, binding):
if not isinstance(key, str) or not 1 <= len(key) <= 256 or not isinstance(binding, str) or not re.fullmatch(r"[0-9a-f]{64}", binding):
raise ValueError("bounded exact journal request binding required")
def _active(self, cursor):
cursor.execute("SELECT 1 FROM identity_provisioner.group_receipts WHERE active = true LIMIT 1")
if cursor.fetchone():
raise GroupProvenanceUnavailable()
def _transition(self, tx, receipt, operation):
# Bind the exact transition, not merely a counter. Two restored/forked
# copies that commit different bodies at the same next head cannot both
# match an independently retained checkpoint after reopening.
transition = {"schema": "nk-group-journal-transition.v1", "epoch": self.binding.epoch,
"directory": self.binding.directory, "prior_head": tx["head"], "prior_digest": tx["digest"],
"head": tx["head"] + 1, "operation": operation, "context": asdict(self.binding),
"key": receipt["key"], "request_binding": receipt["binding"],
"version": receipt["_pg_version"], "body": receipt}
tx["next_digest"] = hashlib.sha256(json.dumps(transition, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
tx["dirty"] = True
def reserve(self, key, binding):
self._inputs(key, binding)
with self._transaction() as tx:
cursor = tx["cursor"]
receipt = self._read(cursor, key, binding)
if receipt is not None:
return receipt, False
self._active(cursor)
cursor.execute("SELECT COUNT(*) FROM identity_provisioner.group_receipts")
if cursor.fetchone()[0] >= self.capacity:
raise GroupProvenanceUnavailable()
receipt = {"key": key, "binding": binding, "state": "pending", "intent": None, "groups": {},
"created_groups": [], "created_memberships": [], "_pg_version": 1, "_pg_context": self.binding.digest}
cursor.execute("INSERT INTO identity_provisioner.group_receipts (context_digest, context_json, key, binding, version, state, active, body) VALUES (%s, %s, %s, %s, 1, 'pending', true, %s)",
(self.binding.digest, json.dumps(asdict(self.binding), sort_keys=True), key, binding, json.dumps(receipt, sort_keys=True)))
if cursor.rowcount != 1:
raise GroupProvenanceUnavailable()
self._transition(tx, receipt, "reserve")
return receipt, True
def get(self, key, binding):
self._inputs(key, binding)
with self._transaction() as tx:
receipt = self._read(tx["cursor"], key, binding)
if receipt is None:
raise ValueError("unknown ensure receipt or request binding mismatch")
return receipt
def _write(self, tx, receipt, operation="save"):
self._inputs(receipt.get("key"), receipt.get("binding"))
if receipt.get("_pg_context") != self.binding.digest or type(receipt.get("_pg_version")) is not int or receipt["_pg_version"] < 1:
raise GroupProvenanceUnavailable()
encoded = json.dumps(receipt, sort_keys=True)
GroupProvenance._load(encoded, receipt["key"], receipt["binding"])
updated = json.loads(encoded)
updated["_pg_version"] += 1
active = True if receipt["state"] in {"pending", "revoking", "uncertain"} else None
tx["cursor"].execute("UPDATE identity_provisioner.group_receipts SET state = %s, active = %s, body = %s, version = version + 1 WHERE context_digest = %s AND key = %s AND binding = %s AND version = %s",
(receipt["state"], active, json.dumps(updated, sort_keys=True), self.binding.digest, receipt["key"], receipt["binding"], receipt["_pg_version"]))
if tx["cursor"].rowcount != 1:
raise GroupProvenanceUnavailable()
self._transition(tx, updated, operation)
return updated
def save(self, receipt):
with self._transaction() as tx:
updated = self._write(tx, receipt)
# The provisioner keeps references to nested group pre/poststate across
# intent persistence. Replace only this backend's acknowledged version;
# _write serializes without changing the caller's mutation body.
receipt["_pg_version"] = updated["_pg_version"]
def claim_revoke(self, key, binding, delete_if_empty):
self._inputs(key, binding)
if not isinstance(delete_if_empty, bool):
raise ValueError("delete_if_empty must be boolean")
with self._transaction() as tx:
receipt = self._read(tx["cursor"], key, binding)
if receipt is None:
raise ValueError("unknown ensure receipt or request binding mismatch")
if receipt["state"] == "revoked":
if receipt["delete_if_empty"] != delete_if_empty:
raise ValueError("completed rollback options cannot change")
return receipt
if receipt["state"] != "ensured":
raise GroupProvenanceUnavailable()
self._active(tx["cursor"])
receipt.update(state="revoking", delete_if_empty=delete_if_empty)
updated = self._write(tx, receipt, "claim_revoke")
return updated
def psycopg_connector(*, connection_info: Callable[[], dict[str, Any]]):
"""Owner-provided renewable lease values stay in memory; never env/argv/logs.
No connection happens here. Lazy import keeps disabled group endpoints free
of a runtime dependency. The admitted owner supplies TLS/host/db/role facts.
"""
if not callable(connection_info):
raise ValueError("owner connection-info callback required")
def connect():
try:
import psycopg
values = dict(connection_info())
return psycopg.connect(**values, connect_timeout=5)
except Exception:
raise GroupProvenanceUnavailable() from None
return connect

View file

@ -0,0 +1,23 @@
-- Run only by the admitted migration role in its existing owner schema.
-- Does not create a database/schema/credential, admit an epoch or clear quarantine.
CREATE TABLE identity_provisioner.group_journal_meta (
singleton boolean PRIMARY KEY CHECK (singleton = true),
epoch uuid NOT NULL,
directory text NOT NULL CHECK (octet_length(directory) BETWEEN 1 AND 1024),
head bigint NOT NULL CHECK (head >= 0),
head_digest text NOT NULL CHECK (head_digest ~ '^[0-9a-f]{64}$'),
quarantined boolean NOT NULL DEFAULT true
);
CREATE TABLE identity_provisioner.group_receipts (
context_digest text NOT NULL CHECK (context_digest ~ '^[0-9a-f]{64}$'),
context_json text NOT NULL CHECK (octet_length(context_json) <= 16384),
key text NOT NULL CHECK (char_length(key) BETWEEN 1 AND 256),
binding text NOT NULL CHECK (binding ~ '^[0-9a-f]{64}$'),
version bigint NOT NULL CHECK (version >= 1),
state text NOT NULL CHECK (state IN ('pending','ensured','revoking','revoked','uncertain','refused')),
active boolean UNIQUE CHECK (active IS NULL OR active = true),
body text NOT NULL CHECK (octet_length(body) <= 1048576),
PRIMARY KEY (context_digest, key)
);
-- Epoch insertion, initial continuity checkpoint and admission are separate
-- owner actions; this migration leaves all operations refused without them.

View file

@ -11,7 +11,7 @@ from typing import Any, Callable
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
from group_provenance import GroupProvenance, GroupProvenanceUnavailable
from group_provenance import GroupJournalStore, GroupProvenanceUnavailable
@dataclass(frozen=True)
@ -64,7 +64,7 @@ class LLDAPProvisioner:
opener: Callable = urlopen,
auth_rejected_cooldown: float = 30.0,
clock: Callable[[], float] = time.monotonic,
group_provenance: GroupProvenance | None = None,
group_provenance: GroupJournalStore | None = None,
) -> None:
self.group_provenance = group_provenance
self.base_url = base_url.rstrip("/")
@ -243,7 +243,7 @@ mutation Remove($userId: String!, $groupId: Int!) {
timeout=timeout,
)
def _group_store(self) -> GroupProvenance:
def _group_store(self) -> GroupJournalStore:
store = self.group_provenance
if store is None:
raise GroupProvenanceUnavailable()

View file

@ -0,0 +1,5 @@
# Linux amd64 CPython3.12 runtime closure from user-engine/uv.lock.
# Build-only dependency resolution; no packages are installed at service startup.
psycopg==3.3.5 --hash=sha256:ce5aa5cdb4f9379f00f487590e5890bfa7df9a164648c969ffa628505e21af4e
psycopg-binary==3.3.5 --hash=sha256:682a17a57415c3ca1731eec018ed031f012ffcb81ba74806eb219cb396065672
typing-extensions==4.16.0 --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8

View file

@ -0,0 +1,231 @@
"""Binding, CAS and ambiguous-commit refusal tests; native SQL proof is separate."""
from dataclasses import replace
import json
from unittest.mock import Mock
import unittest
from group_provenance import GroupProvenanceUnavailable
from group_provenance_postgres import GroupJournalBinding, PostgresGroupProvenance
from server import configured_group_provenance
from unittest.mock import patch
import os
EPOCH = '41fbb0ed-ea3a-4682-996b-e9c94a6aa9c6'
BINDING = GroupJournalBinding('tenant:fixture:one', 'synthetic-directory', 'sha256:' + '1' * 64,
'sha256:' + '2' * 64, 'synthetic-owner', EPOCH)
REQUEST = '3' * 64
HEAD_DIGEST = 'a' * 64
META = (EPOCH, 'synthetic-directory', 7, HEAD_DIGEST, False)
def fixture(rows=(), rowcount=1, commit_error=None):
cursor = Mock()
cursor.__enter__ = Mock(return_value=cursor)
cursor.__exit__ = Mock(return_value=False)
cursor.fetchone.side_effect = rows
cursor.rowcount = rowcount
connection = Mock()
connection.__enter__ = Mock(return_value=connection)
connection.__exit__ = Mock(return_value=False, side_effect=commit_error)
connection.cursor.return_value = cursor
connector = Mock(return_value=connection)
continuity = Mock()
continuity.check.return_value = True
continuity.advance.return_value = True
store = PostgresGroupProvenance(connect=connector, binding=BINDING, continuity=continuity)
return store, connector, connection, cursor, continuity
def receipt(version=4):
return {'key': 'synthetic-operation-key', 'binding': REQUEST, 'state': 'pending', 'intent': None,
'groups': {}, 'created_groups': [], 'created_memberships': [],
'_pg_version': version, '_pg_context': BINDING.digest}
class PostgresSourceTests(unittest.TestCase):
def test_explicit_binding_and_continuity_required_without_connecting(self):
connector = Mock()
for binding, continuity in ((None, Mock()), (BINDING, None), ({'tenant_scope': 'caller-json'}, Mock())):
with self.assertRaises(ValueError):
PostgresGroupProvenance(connect=connector, binding=binding, continuity=continuity)
connector.assert_not_called()
for change in ({'tenant_scope': ''}, {'principal': ''}, {'case_digest': 'unsigned'}, {'epoch': 'default'}):
with self.assertRaises(ValueError):
replace(BINDING, **change)
self.assertNotEqual(BINDING.digest, replace(BINDING, principal='another-owner').digest)
self.assertNotEqual(BINDING.digest, replace(BINDING, tenant_scope='another-explicit-scope').digest)
def test_missing_environment_callback_path_keeps_production_disabled(self):
with patch.dict(os.environ, {'GROUP_POSTGRES_DSN': 'synthetic-only', 'GROUP_PROVENANCE_POSTGRES': 'true'}, clear=True):
self.assertIsNone(configured_group_provenance())
def test_checkpoint_mismatch_poisoned_without_receipt_query_or_effect(self):
store, connect, _, cursor, checkpoint = fixture([META])
checkpoint.check.return_value = False
with self.assertRaises(GroupProvenanceUnavailable):
store.reserve('synthetic-operation-key', REQUEST)
self.assertFalse(any('group_receipts' in c.args[0] for c in cursor.execute.call_args_list))
with self.assertRaises(GroupProvenanceUnavailable):
store.check()
self.assertEqual(1, connect.call_count)
checkpoint.advance.assert_not_called()
def test_restored_epoch_or_quarantine_refused(self):
for meta in ((EPOCH, 'other-directory', 7, HEAD_DIGEST, False), (EPOCH, 'synthetic-directory', 7, HEAD_DIGEST, True),
('98aa2b6d-a32a-44a5-a89f-b2480befc00e', 'synthetic-directory', 7, HEAD_DIGEST, False), None):
store, _, _, _, checkpoint = fixture([meta])
with self.assertRaises(GroupProvenanceUnavailable):
store.check()
checkpoint.check.assert_not_called()
def test_exactly_one_version_update_required_before_checkpoint_advance(self):
store, _, _, cursor, checkpoint = fixture([META], rowcount=0)
original = receipt()
with self.assertRaises(GroupProvenanceUnavailable):
store.save(original)
query, params = next(c.args for c in cursor.execute.call_args_list if c.args[0].startswith('UPDATE identity_provisioner.group_receipts'))
self.assertIn('AND version = %s', query)
self.assertEqual(4, params[-1])
self.assertEqual(4, original['_pg_version'])
checkpoint.advance.assert_not_called()
def test_success_acknowledges_commit_and_updates_receipt_version(self):
store, _, connection, _, checkpoint = fixture([META])
original = receipt()
store.save(original)
self.assertEqual(5, original['_pg_version'])
connection.__exit__.assert_called_once()
args = checkpoint.advance.call_args.args
self.assertEqual((EPOCH, 'synthetic-directory', 7, HEAD_DIGEST, 8), args[:5])
self.assertRegex(args[5], r'^[0-9a-f]{64}$')
self.assertNotEqual(HEAD_DIGEST, args[5])
connection.close.assert_called_once()
def test_save_preserves_nested_receipt_references_held_by_provisioner(self):
store, *_ = fixture([META])
original = receipt()
original['groups']['forge-coulomb-owners'] = {'id': None, 'before': [], 'after': ['tegwick']}
state = original['groups']['forge-coulomb-owners']
groups, created, memberships = original['groups'], original['created_groups'], original['created_memberships']
original['intent'] = {'operation': 'create_group', 'name': 'forge-coulomb-owners'}
store.save(original)
self.assertIs(groups, original['groups'])
self.assertIs(state, original['groups']['forge-coulomb-owners'])
self.assertIs(created, original['created_groups'])
self.assertIs(memberships, original['created_memberships'])
state['id'] = 4
self.assertEqual(4, original['groups']['forge-coulomb-owners']['id'])
self.assertEqual(5, original['_pg_version'])
def test_actual_provisioner_ensure_with_postgres_save_retains_created_ids(self):
from test_service_groups import Directory, req
directory = Directory()
directory.members[3].add('tegwick')
store, _, _, cursor, _ = fixture([META] * 32)
original = receipt(version=1)
payload = req()
original['key'] = payload['idempotency_key']
original['binding'] = directory._group_binding(payload['groups'], payload['members'], payload['correlation_id'])
# Reservation input is already covered by SQL unit/native oracles. Keep
# real save/_write/transaction and the real provisioner creation path.
store.reserve = Mock(return_value=(original, True))
directory.group_provenance = store
result = directory.ensure_service_groups(payload)
self.assertEqual('groups_ensured', result.status)
self.assertEqual({'forge-coulomb-owners', 'forge-coulomb-readers'}, {g['name'] for g in result.created_groups})
self.assertEqual(1, len(result.created_memberships))
self.assertEqual({'tegwick'}, directory.members[3])
bodies = [json.loads(c.args[1][2]) for c in cursor.execute.call_args_list
if c.args[0].startswith('UPDATE identity_provisioner.group_receipts')]
self.assertTrue(all(g['id'] is not None for g in bodies[-1]['groups'].values()))
self.assertEqual('ensured', bodies[-1]['state'])
def test_connection_close_failure_is_sanitized_and_stops_continuation(self):
store, _, connection, _, _ = fixture([META])
connection.close.side_effect = OSError('synthetic-private-diagnostic')
with self.assertRaisesRegex(GroupProvenanceUnavailable, '^group_provenance_unavailable$'):
store.check()
self.assertTrue(store._poisoned)
def test_commit_failure_never_advances_owner_checkpoint_or_receipt(self):
store, _, _, _, checkpoint = fixture([META], commit_error=OSError('synthetic-secret-diagnostic'))
original = receipt()
with self.assertRaisesRegex(GroupProvenanceUnavailable, '^group_provenance_unavailable$'):
store.save(original)
self.assertEqual(4, original['_pg_version'])
checkpoint.advance.assert_not_called()
def test_lost_checkpoint_acknowledgment_refuses_then_poisoned(self):
store, connect, _, _, checkpoint = fixture([META])
checkpoint.advance.side_effect = OSError('synthetic-secret-diagnostic')
original = receipt()
with self.assertRaisesRegex(GroupProvenanceUnavailable, '^group_provenance_unavailable$'):
store.save(original)
self.assertEqual(4, original['_pg_version'])
with self.assertRaises(GroupProvenanceUnavailable):
store.save(original)
self.assertEqual(1, connect.call_count)
def test_cross_case_body_or_version_cannot_be_imported(self):
for body in (dict(receipt(), _pg_context='0' * 64), dict(receipt(), _pg_version=6)):
store, _, _, _, _ = fixture([META, (REQUEST, 4, json.dumps(body), json.dumps(BINDING.__dict__, sort_keys=True))])
with self.assertRaises(GroupProvenanceUnavailable):
store.get('synthetic-operation-key', REQUEST)
def test_same_counter_divergent_body_or_context_yields_distinct_marker(self):
store, *_ = fixture()
markers = []
for change in ({}, {'intent': {'operation': 'different'}}, {'key': 'another-exact-key'}, {'_pg_version': 5}):
tx = {'head': 7, 'digest': HEAD_DIGEST}
store._transition(tx, dict(receipt(), **change), 'save')
markers.append(tx['next_digest'])
self.assertEqual(4, len(set(markers)))
tx = {'head': 7, 'digest': 'b' * 64}
store._transition(tx, receipt(), 'save')
self.assertNotIn(tx['next_digest'], markers)
foreign, *_ = fixture()
foreign.binding = replace(BINDING, principal='another-owner')
tx = {'head': 7, 'digest': HEAD_DIGEST}
foreign._transition(tx, receipt(), 'save')
self.assertNotIn(tx['next_digest'], markers)
def test_reopened_loser_clone_at_same_head_refuses_different_marker(self):
# Both DBs report head8. The independent callback accepts only winner
# digest; loser must be refused before loading any receipt or effect.
loser_digest = 'b' * 64
store, _, _, cursor, checkpoint = fixture([(EPOCH, 'synthetic-directory', 8, loser_digest, False)])
checkpoint.check.side_effect = lambda epoch, directory, head, digest: (head, digest) == (8, 'c' * 64)
with self.assertRaises(GroupProvenanceUnavailable):
store.check()
checkpoint.check.assert_called_once_with(EPOCH, 'synthetic-directory', 8, loser_digest)
self.assertFalse(any('group_receipts' in c.args[0] for c in cursor.execute.call_args_list))
def test_existing_request_mismatch_is_refused_not_rebound(self):
store, _, _, _, checkpoint = fixture([META, ('0' * 64, 4, '{}', '{}')])
with self.assertRaisesRegex(ValueError, 'binding mismatch'):
store.get('synthetic-operation-key', REQUEST)
checkpoint.advance.assert_not_called()
self.assertFalse(store._poisoned)
def test_active_fence_capacity_refusal_and_no_insert(self):
for rows in ([META, None, (1,)], [META, None, None, (256,)]):
store, _, _, cursor, checkpoint = fixture(rows)
with self.assertRaises(GroupProvenanceUnavailable):
store.reserve('synthetic-operation-key', REQUEST)
self.assertFalse(any('INSERT' in c.args[0] for c in cursor.execute.call_args_list))
checkpoint.advance.assert_not_called()
def test_new_reservation_retains_bound_context_and_write_intent(self):
store, _, _, cursor, checkpoint = fixture([META, None, None, (0,)])
created, fresh = store.reserve('synthetic-operation-key', REQUEST)
self.assertTrue(fresh)
self.assertEqual('pending', created['state'])
self.assertEqual(BINDING.digest, created['_pg_context'])
insert = next(c for c in cursor.execute.call_args_list if c.args[0].startswith('INSERT'))
self.assertIn('context_json', insert.args[0])
self.assertEqual(BINDING.tenant_scope, json.loads(insert.args[1][1])['tenant_scope'])
checkpoint.advance.assert_called_once()
if __name__ == '__main__':
unittest.main()

View file

@ -12,6 +12,40 @@ SOPS = shutil.which('sops')
pytestmark = pytest.mark.skipif(SOPS is None, reason='SOPS required for real crypto regression')
@pytest.mark.parametrize('wrong_recipient', [False, True])
def test_real_sops_decrypt_with_only_sealed_descriptor(case, wrong_recipient):
import fcntl
original = case[1].read_bytes()
material = (subprocess.run(['age-keygen'], capture_output=True, check=True).stdout
if wrong_recipient else original)
fd = os.memfd_create('generated-sops-custody', os.MFD_ALLOW_SEALING)
try:
os.fchmod(fd, 0o600); os.write(fd, material); os.lseek(fd, 0, os.SEEK_SET)
fcntl.fcntl(fd, fcntl.F_ADD_SEALS,
fcntl.F_SEAL_WRITE | fcntl.F_SEAL_GROW | fcntl.F_SEAL_SHRINK | fcntl.F_SEAL_SEAL)
case[1].unlink()
env = {**case[2], 'NETKINGDOM_CUSTODY_AGE_KEY_FD': str(fd)}
env.pop('NETKINGDOM_CUSTODY_AGE_KEY_FILE')
if wrong_recipient:
# A correct ambient key must never rescue the wrong explicit FD.
env['SOPS_AGE_KEY'] = original.decode()
result = subprocess.run(['make', '-s', 'sops-decrypt', 'FILE=' + str(case[3]),
'OUTPUT=' + str(case[4])], cwd=case[0], env=env,
pass_fds=(fd,), capture_output=True, text=True, timeout=15)
assert not case[1].exists()
assert 'synthetic-secret-payload' not in result.stdout + result.stderr
if wrong_recipient:
assert result.returncode != 0
assert not case[4].exists()
else:
assert result.returncode == 0
assert json.loads(case[4].read_text()) == {'credential': 'synthetic-secret-payload'}
assert case[4].stat().st_mode & 0o077 == 0
assert not list(case[4].parent.glob('.netkingdom-sops-*'))
finally:
os.close(fd)
@pytest.fixture
def case(tmp_path):
repo = tmp_path / 'repo'; (repo / 'sso-mfa/bootstrap').mkdir(parents=True)

View file

@ -13,6 +13,36 @@ pytestmark=pytest.mark.skipif(SOPS is None,reason='real SOPS required')
PLAIN='synthetic-sops-write-value'
@pytest.mark.parametrize('action', ['rotate', 'edit'])
def test_real_sops_write_with_only_sealed_descriptor(case, action):
import fcntl
fd = os.memfd_create('generated-sops-write-custody', os.MFD_ALLOW_SEALING)
try:
os.fchmod(fd, 0o600); os.write(fd, case['key'].read_bytes()); os.lseek(fd, 0, os.SEEK_SET)
fcntl.fcntl(fd, fcntl.F_ADD_SEALS,
fcntl.F_SEAL_WRITE | fcntl.F_SEAL_GROW | fcntl.F_SEAL_SHRINK | fcntl.F_SEAL_SEAL)
case['key'].unlink()
env = {**case['env'], 'NETKINGDOM_CUSTODY_AGE_KEY_FD': str(fd)}
env.pop('NETKINGDOM_CUSTODY_AGE_KEY_FILE'); env.pop('SOPS_AGE_KEY_FILE')
before = case['cipher'].read_bytes()
result = subprocess.run(['make', '-s', 'sops-' + action,
'FILE=' + str(case['cipher']), 'INPUT=' + str(case['input'])],
cwd=case['repo'], env=env, pass_fds=(fd,),
capture_output=True, text=True, timeout=15)
assert result.returncode == 0
assert PLAIN not in result.stdout + result.stderr
assert not case['key'].exists()
assert before != case['cipher'].read_bytes()
clear = subprocess.run([SOPS, '--decrypt', str(case['cipher'])],
env={**env, 'SOPS_AGE_KEY_FILE': '/proc/self/fd/' + str(fd)},
pass_fds=(fd,), capture_output=True, check=True)
assert json.loads(clear.stdout) == {'credential': PLAIN + ('-edited' if action == 'edit' else '')}
assert case['cipher'].stat().st_mode & 0o077 == 0
assert not list(case['cipher'].parent.glob('.netkingdom-sops-write-*'))
finally:
os.close(fd)
@pytest.fixture
def case(tmp_path):
repo=tmp_path/'repo';boot=repo/'sso-mfa/bootstrap';boot.mkdir(parents=True)

View file

@ -691,31 +691,30 @@ Native admission and all other-owner acceptance remain open.
```task
id: NK-WP-0048-T14
status: progress
depends_on: [NK-WP-0048-T20]
depends_on: [NK-WP-0048-T20, NK-WP-0048-T21]
priority: high
state_hub_task_id: "fce95440-a25b-5f27-8ef3-a3d182099113"
```
Owner: NetKingdom. Next action: specify a CNPG-backed journal through the
existing `rapp-postgres` database-per-consumer contract, or obtain an explicit
scoped governed exception to accepted ADR-0015 before any production SQLite
proposal. The implemented SQLite backend supplies local/disposable proof only;
a new PVC is not an admitted production path. Prepare the native writer-isolation
contract for the existing identity-provisioner, and bind the
existing protected owner callbacks to the accepted case/source/principal,
per-attempt evidence and session revocation. Identify the actual deployment and
custody owner through the existing ADR-0015 routes before proposing a rollout;
an existing service token, caller-supplied JSON or provenance receipt establishes
no new authorization.
Owner: NetKingdom. Next action: bind the completed T21 PostgreSQL journal to
an admitted dedicated CNPG consumer through the existing `rapp-postgres`
database-per-consumer contract. Identify the actual deployment/package and
custody owner, admitted placement/capacity and role/renewable lease/TLS target;
implement the independently retained exact content checkpoint and authoritative
tenant/case/source/principal callbacks before enabling governed group operations.
The injection-only backend and fixture callbacks do not supply production scope
or authorization. Existing service tokens, caller-supplied JSON and provenance
receipts confer no new grant. The default production endpoint remains disabled
without the trusted integration. SQLite remains local/disposable only; any
production exception requires explicit scoped governance under ADR-0015.
2026-10-17 refinement: T20 extracts the completed durable provenance source
candidate and disposable LLDAP compatibility proof from this task. The source
returns exact creation inventories, retains prestate/intents across process
restart and refuses detected shared-member/group-identity drift. Its publication
and projection are still pending the coordinator's separate integration receipt.
This narrows T14 without closing its original externally referenced native gate.
The task remains `todo` because preparation of the native contracts can proceed;
no missing owner assent is represented as received.
2026-10-17 refinement: T20 retains the historical durable SQLite source and
actual disposable LLDAP proof. T21 extracts the completed PostgreSQL source and
bounded real disposable SQL compatibility proof. T14 remains `progress` for
native CNPG admission/recovery, protected owner integration and exclusion of all
external LDAP writers. Separate source publication/projection and independent
cleanup receipts belong to coordinator follow-through; no missing native assent
is represented as received.
Completion still requires independently reviewed/admitted ADR-0015-conforming
CNPG storage (or an explicit governed exception), backup/lost-store recovery and
@ -994,3 +993,74 @@ helper or lifecycle writers. Capacity placement and EUR cost/risk remain owner
decisions. No package, database, credential, rollout or owner assent was created.
Custodian evidence: group-provenance-r1/native-storage-proposal-r1; final source,
projection and cleanup review: resume-integration-r1/publication-verification.json.
## T21 — Implement the PostgreSQL journal and prove disposable SQL compatibility
```task
id: NK-WP-0048-T21
status: done
depends_on: [NK-WP-0048-T20]
priority: high
state_hub_task_id: "afe78e4d-408a-516a-ab92-e212e8a2d11d"
```
Completed 2026-10-17: bounded source preparation and actual disposable PostgreSQL
compatibility only. T14 retains native storage, custody, callbacks, authorization,
all-writer isolation and production admission. Coordinator source publication and
projection remain separate follow-through; no source revision or task UUID is
invented here. T20 remains the historical local SQLite/actual LLDAP proof.
The eight-path source candidate adds an injection-only PostgreSQL implementation
behind the existing owner journal seam, with explicit trusted tenant/directory/
case/source/principal/epoch binding and independent continuity callbacks. It
persists bounded intent/created-only receipts, exact-one version updates, durable
active fences, and a content-bound journal chain. SQL commits precede independent
exact-marker acknowledgment; lost replies/checkpoints poison continuation and
retain uncertainty. A counter-only checkpoint is explicitly insufficient for
divergent clones. Migration does not create database/schema/credential, admit an
epoch or clear quarantine; production HTTP activation is absent pending trusted
callbacks. Pinned psycopg package/image-copy closure is included, without claiming
an image build or native dependency admission.
Candidate base `6cb43c0cc55212363a5406b02db046e2384a7c40`; final eight-path patch
`sha256:5549d2982b61ff70aba4b3928fecd5639db0f1a732ab2f8630f678e7acf4d34d`.
All 82 source tests pass, alongside 13 fixture plan/diagnostic checks and two
source-backed snapshot regressions. Counts overlap the same source and fixture;
they are not additive native acceptance or completion percentages.
The independently reviewed R5 fixture ran the actual SQL migration/backend and
existing provisioner path against cached pinned disposable PostgreSQL. Directory
operations deliberately used the existing synthetic LLDAP test double. All
manifest-declared phase oracles passed: creation-only inventory and DDL refusal;
separate-process replay/revoke with complete preexisting/shared visible-state
preservation; foreign-context refusal; concurrent exactly-once reservation and
stale version refusal; actual database fork with divergent bodies at the same
counter and fresh-object rejection of the losing exact checkpoint; retained
uncertainty after checkpoint/directory/SQL response loss; and actual pg_dump/
pg_restore rollback refusal against the independently retained fixture checkpoint.
The checkpoint is fixture-only, not an admitted native owner evidence holder.
Only generated credentials/state were used, with two nonroot/read-only containers
on private loopback, bounded tmpfs/CPU/memory/PIDs/lifetimes, no host ports/mounts,
no pulls and no production contact. The safe receipt reports both exact owned
containers absent, with no daemon uncertainty. Independent final pinned-daemon
cleanup verification remains coordinator follow-through.
Failed R2/R3/R4 receipts and source revisions remain retained. R2's helper failure
subphase is unknown; R3 exposed a real backend bug where replacing nested receipt
objects lost the created ID held by ensure, fixed by acknowledging only the backend
version after commit/checkpoint. R4 verified that fix through actual SQL bootstrap,
then exposed fixture snapshot aliasing and empty deleted-group backing entries;
source-backed regressions corrected the snapshot without weakening complete
visible-prestate equality, and reject nonempty orphan membership. These refinements
belong to one trial, not extra native acceptance successes.
Custodian evidence: `docs/changes/netkingdom-target-2026-10-16/group-postgres-r1/`;
exact source/pre/postimages and retained predecessors; native R5 manifest
`sha256:d51f90816616848b93bcdc482e10a98d07061deb01eeed9e50700fc6e68b6051`;
`final-integration-r1/typed-execution-evidence.json` and byte-exact safe receipt
(original `/tmp/nk48-postgres-native-review-sguj7t8p/result.json`). No CNPG native
storage/HA/backup/lease/TLS/capacity/restore admission, authoritative protected
callbacks, real LDAP writer fencing/CAS/ABA exclusion, owner assent or production
endpoint activation is claimed. Those remain live T14 work.

View file

@ -28,7 +28,7 @@ These are retrospective task records of evidenced work, not additional delivery
or newly granted authority. The earlier dated narrative remains in Git at
`cc2dd01`; its evidence references are carried into the tasks below.
**Current position, source refinement 2026-10-17:** 24 tasks done; 4 in
**Current position, source refinement 2026-10-17:** 25 tasks done; 4 in
progress; 1 ready to start (`todo`); 7 waiting on explicit prerequisites. These are task counts, not an effort-based
completion percentage. Assignees are also written in task descriptions because
the current Forge projector does not populate the dedicated Hub assignee field. The outage-retrieval tool and restore-packet draft are now prepared;
@ -1071,6 +1071,19 @@ Record any unavailable recovery capability as a live owner work record and bind
its dependency/unblock condition; a prose-only gap or untested archive is not
completion. This delivers preparation/disposition, not actual TLS recovery.
2026-10-17 owner handoff progress: the four reviewed Railiance Cluster source
additions are present, including `RCLUSTER-WP-0018` and public-only TLS metadata
preparation. They are not published. The normal publication path stopped at the
existing historical identifier guard (`ensure_missing_work_record_identifiers`,
execute=False): 15 bare task-label findings and five collisions. A separately
retained read-only allocator preview records those findings; the original failed
publication stderr was not retained and is not reconstructed. No Cluster Hub
workplan/task existence or projection, actual certificate/key custody, reissue
assent or native TLS acceptance is claimed. The source record and next owner
publication action remain live; do not bypass the historical identity guard.
Evidence: Custodian October17 `cluster-integration.json`, `coordinator-checkpoint.json`
and October16 `cluster-historical-publication-r1/actual-source-allocator-preview.json`.
## Prepare selected original service registration and private admission bindings
```task
@ -1103,6 +1116,60 @@ combine with T35's original TLS/trust disposition under T33. This task may prepa
registration metadata while T35 proceeds independently; it does not finalize the
whole admission, perform actual recovery, redirect a consumer or confer assent.
2026-10-17 owner handoff progress: Secrets Engine source workplan
`SECRETS-WP-0015` and bounded public selector/preflight source were published as
`c59bbf568c0be8a72ac593508f48c9b1f61ec4c1`; the central primary/railiance01
reconciliation receipt reports the same expected/derived commit, four owner task
bindings and two T04 dependency edges, with no unresolved edges. T01/T02 source
preparation is done, T03 exact outage-compatible consumer proof/custody planning
is todo, and T04 actual admitted consumer result waits on T03 and NK-WP-0049-T33.
This publishes an owner handoff, not exact archived registration validation,
current native assent, an OpenBao read, a new grant or consumer continuity.
T36 therefore remains progress; T18/T19 and final native gates remain open.
Evidence: Custodian October17 `secrets-engine-projection.json` and October16
`recovery-owner-handoffs-r1` source review; independent ID/UUID/status/remote
readback is recorded separately when performed.
## Verify actual SOPS compatibility and sealed-descriptor read/write custody
```task
id: NK-WP-0049-T37
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T34]
state_hub_task_id: "3403e266-d63b-5ce1-b891-3bfeac93d28c"
```
**Accountable worker:** NetKingdom codex; independent Custodian fixture review.
Completed 2026-10-17 as generated-fixture compatibility proof. The coordinator
ran the actual reviewed SOPS 3.13.3 binary in a network/host-custody-isolated
namespace: all 65 tests passed in 14.85 seconds, including the 61 real-SOPS cases
previously skipped in T34 and four new actual sealed-FD cases. The latter cover
correct-key descriptor-only decryption, wrong-recipient refusal and descriptor-only
edit/rotation through the real Make entry points. Generated custody key files
were removed before these FD-only invocations; explicit descriptor passing,
private output modes, declared encryption-policy preservation, suppressed synthetic plaintext
and operation cleanup were checked.
This is a dated successor to T34's historical 94-pass/61-skip record, which stays
unchanged. The prior skip is not rewritten as an earlier pass. Binary digest
`sha256:e5bec3346a873ae91d871550f3e698c1aad962aff462a080e40f25fde17fef6b`
matched the official release asset/checksum; Sigstore signature verification was
not performed. SOPS was not installed systemwide or admitted for production
runtime. Host custody was not mounted, network routes were absent, diagnostics
were suppressed and namespace teardown completed. All material was generated;
no original owner key/archive, private manifest, production credential, native
service continuity or owner assent was exercised.
Evidence: Custodian
`docs/changes/netkingdom-target-2026-10-17/sops-real-regressions.json`,
`sops-source-tool.json` and `sops-fd-r2/` (exact test patch/manifest). The focused
custody/source checks and frozen R6/R7 native fixture evidence remain separate.
T33 attempt readiness, T18 actual recovery/consumer acceptance, T19 retained MFA,
T35 TLS/trust disposition and T36 original registration/private admission remain
open under their existing owner boundaries.
## Accept remaining recovery and caller evidence and close the G08 handoff
```task
@ -1122,7 +1189,7 @@ Owner boundary: NetKingdom, Platform and affected service owners, with Custodian
G08 receiving the final evidence. The completed archive/job milestones remain
accepted independently of this final gate.
**Done when:** T02 and T15–T36 have their stated completed or explicitly accepted
**Done when:** T02 and T15–T37 have their stated completed or explicitly accepted
handoff outcomes; service authentication/token/audit proof and current/legacy
caller dispositions are accepted; the source workplan and projected tasks agree;
and G08 has an exact evidence/handoff record. Record actionable residuals as live