state-hub/workplans/STATE-WP-0084-forge-read-for-private-repositories.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	quality_dod	quality_dod_at	quality_dod_by	quality_dod_note	related	state_hub_workstream_id
STATE-WP-0084	workplan	Derive private repositories, and report unreadable as unreadable	infotech	state-hub	finished	codex	infotech	2026-08-26	2026-08-31	DoD-Ok	2026-08-31	codex	Credential delivery, safe consumer behavior, nine-repository reviewed apply, and nine-noop repeat verification are evidenced without storing a secret.	CUST-ADR-012 STATE-WP-0083 MASON-WP-0003	9c4b8c79-edba-5d2d-ad37-a45707e89304

Derive private repositories, and report unreadable as unreadable

Goal

Consume the forge read credential that ops-mason/MASON-WP-0003 builds, so derivation covers private repositories — and make a repository central cannot read report as unreadable rather than as an error or, worse, as a repository whose records no longer derive.

Why this exists

STATE-WP-0083-T04 reset 121 repositories on 2026-08-26. Nine failed identically:

fatal: could not read Username for 'https://forgejo.coulomb.social'

rapp-core-hub, rapp-issue-core, rapp-openbao, rapp-policy-nexus and five others are invisible to derivation entirely. ADR-012's premise — the forge is the projection source — holds only for repositories central can read, and nothing in the current design says so.

Split from MASON-WP-0003

The credential lane itself (OpenBao policy, AppRole, KV path, and the warden route find catalog entry) is ops-mason's to build: that repository exists to build "AppRoles, policies, and KV secret paths so ops-warden always has something real to route to". Deployment plumbing and derivation code are this repository's.

T01 does not depend on the credential and should land first. T02 and T03 wait on MASON-WP-0003-T02.

Distinguish "cannot read" from "does not exist"

id: STATE-WP-0084-T01
status: done
priority: high
state_hub_task_id: "e955067d-2152-59f7-b672-12755ceb26b0"

A repository the hub is not permitted to read must be reported as unreadable — a named condition — not as a generic clone error and never as an empty derivation.

Today the reset treats a failed clone as an error, which is correct only by accident: if a clone ever returned empty instead of failing, every record in that repository would be proposed for retirement. That is the failure that nearly retired vergabe-teilnahme, and it must be impossible here by construction rather than by luck.

An empty clone result and an unreadable repository must both be incapable of producing retirement work. Retirement proposals require a successful read that positively lacks the record.

This also carries the decision in MASON-WP-0003-T01: a per-repository grant is only safe to choose if a missing grant surfaces as a named condition. Until this lands, the narrow option fails silently.

Acceptance: an unreadable repository is reported as unreadable; a clone that succeeds but returns nothing produces no retirement proposals; both cases are covered by tests that fail if the retirement path is reachable from either.

Done 2026-08-26. api/services/forge_projection.py:

• ForgeUnreadableError subclasses ForgeDeriveError, so existing callers keep catching it while new ones can tell the cases apart. A permission-shaped git failure is classified into it; a genuine fault stays a plain error. Forgejo's 404 for an unauthenticated private repository classifies as unreadable, because the two are indistinguishable at this layer and the safe reading of an ambiguous answer is the one that cannot destroy a record.
• _run_git runs with GIT_TERMINAL_PROMPT=0. Without it an unattended pass blocks on a username prompt instead of failing, and an unreadable repository is only observable if it fails.
• DerivedProjection.records_source_present / .retirement_eligible separate "no records found" from "no records exist". A checkout with no workplans/ directory can no longer evidence an absence.
• diff_against_hub withholds stale rather than computing and filtering it, and says so in stale_withheld. would_remove is 0 for such a source.
• reset_repository_projection returns status="unreadable" for a repository it may not read, and refuses retirement from an ineligible source even when acknowledge_retirements=True — consenting to a conclusion is not the same as the evidence for it existing.
• reset_fleet_projection keeps unreadable repositories out of errors, in their own bucket with its own count. Nine repositories in an error bucket read as nine broken repositories; they were nine we were not allowed to read.

13 tests in TestUnreadableIsNotMissing (tests/test_forge_projection.py); 58 pass across the forge/projection/backfill suites.

Deliver the credential to the pod

id: STATE-WP-0084-T02
status: done
priority: medium
state_hub_task_id: "d8d41a89-1ebc-5118-bb73-cf8d8c114e16"

Blocked on MASON-WP-0003-T02.

Make the credential available to the state-hub deployment as a Kubernetes Secret, mounted or injected, referenced by the chart the same way state-hub-env already is.

It must not be baked into the image and must not be committed to the chart. Rotation must not require a chart change or a redeploy.

Acceptance: the pod can read the credential; nothing in the repository contains it; rotating the token does not require a redeploy.

Chart landed 2026-08-27; awaits the minted token. The cluster has no OpenBao agent injector and no secrets-store CSI driver — checked, not assumed — so there is nothing to inject with. The pod authenticates to OpenBao itself, which is what MASON-WP-0003-T02's Kubernetes auth role was built for.

• templates/serviceaccount.yaml creates ServiceAccount state-hub, and the Deployment now sets serviceAccountName. The auth role binds to this name and deliberately not to default, so until this ships the pod cannot authenticate at all. This changes the identity the running pod uses — expect a pod restart on upgrade.
• The OpenBao token is a projected ServiceAccount token with audience openbao, not the legacy auto-mounted one. The auto-mounted token's audience is the API server, so a copy of it is a credential for the cluster; this one is only accepted by OpenBao, and the kubelet rotates it in place.
• forgeRead.* carries coordinates only — address, role, KV path, key. No credential is a chart value, an image layer, or a Kubernetes Secret in this release. Rotating the token in OpenBao needs no chart change and no redeploy: the value is re-read every 5 minutes.
• Default forgeRead.enabled: false. A hub without the credential still derives every public repository, so this is added capability, not a prerequisite.

Remaining before this is done: the minted read-only Forgejo token in the KV path (paste_once_provision, outside both repositories), and a deploy with forgeRead.enabled=true plus the real address and path in deploy/railiance/apps/helm/state-hub-values.yaml.

Done 2026-08-31. The live railliance01 release has forgeRead.enabled: true; the API pod runs as ServiceAccount state-hub with the projected OpenBao token and coordinate-only configuration. A non-secret in-pod resolution check returned available. The Forge token remains outside Git and is re-read through the five-minute cache, so rotation needs no chart change or pod redeploy.

Teach the derivation to use it

id: STATE-WP-0084-T03
status: done
priority: medium
state_hub_task_id: "b22b24d9-7ed3-533c-bda7-3130693cf4d2"

Blocked on T02.

derive_from_forge() clones anonymously. It should use the credential when present and continue working without it — a hub with no token must still derive public repositories rather than failing wholesale.

Credentials must never reach the clone URL in a way that lands in logs or process listings: use a credential helper or an HTTP header, not an embedded userinfo URL.

Acceptance: private repositories derive; a hub without the credential still derives public ones; no credential appears in logs, process listings, or recorded clone URLs.

Code landed 2026-08-27, ahead of T02 — writing the consumer does not need the secret to exist. api/services/forge_projection.py:

• forge_read_token() reads FORGE_READ_TOKEN_FILE in preference to FORGE_READ_TOKEN. A mounted file is what lets T02 satisfy "rotation must not require a redeploy"; the environment cannot be rotated in place. An unreadable token file returns None rather than falling back to the environment — falling back would let a broken mount look like success while quietly using a stale value.
• Absent is a supported state, not a degraded one: with no credential the git environment is untouched and public derivation runs exactly as before.
• The credential is passed as GIT_CONFIG_COUNT/GIT_CONFIG_KEY_0/ GIT_CONFIG_VALUE_0 setting http.extraHeader, not as -c on the command line and not as userinfo in the URL. Both alternatives put the token in the process listing, where anything that can run ps reads it.
• _run_git redacts the token from ForgeDeriveError before raising, because that message is logged, stored in a reset outcome, and returned over the API.
• api/services/forge_credential.py resolves the credential from a mounted file, an environment variable, or OpenBao via Kubernetes auth, in that order. Production uses the third; the first two make the code runnable and testable outside the cluster. Configuring a file means that file and nothing else — no fall-through, so a broken mount cannot silently resolve to a stale value.
• Every failure resolves to None, never an exception: no configuration, no network, OpenBao down, permission denied. Raising would convert "nine repositories are unreadable" into "the whole pass failed", which is the outcome T01 exists to prevent.
• The resolved value is cached for 5 minutes — a fleet reset of 121 repositories must not authenticate 121 times, and a rotated token must still be picked up without a redeploy.

Nine tests in TestForgeCredential cover each of those, including the two that are silent when wrong: token-in-argv and token-in-exception. 43 pass in tests/test_forge_projection.py; 717 pass across the suite.

Remains progress because the acceptance clause "private repositories derive" cannot be observed until T02 supplies a credential; T04 confirms it.

Done 2026-08-31. A transaction-rolled-back review successfully derived all nine private rapp-* repositories at exact Forge commits. No repository was unreadable or errored, and no retirement was proposed. The anonymous/public path remains covered by the existing absent-credential tests.

Confirm the nine

id: STATE-WP-0084-T04
status: done
priority: medium
state_hub_task_id: "33c09a78-5974-57c6-bb7f-ad73082046bb"

Blocked on T03.

Re-run the fleet reset and confirm the nine repositories derive rather than error. Expect a real diff for each — quite possibly a large one, since none has ever been reconciled against its files.

Treat the first diff for each as review material, not as work to apply. Every repository examined this way for the first time has produced findings.

Acceptance: fleet errors fall from nine to zero; each newly readable repository has its first diff reviewed before anything is applied.

Done 2026-08-31. The first review returned seven bounded source/status updates and two no-ops, with zero creations, retirements, refusals, or errors. Those exact commits were applied through the guarded per-repository API with retirement disabled. The repeated review returned nine no-ops and zero errors. Evidence: docs/evidence/STATE-WP-0084-private-forge-cutover-2026-08-31.md.