state-hub/workplans/STATE-WP-0084-forge-read-for-private-repositories.md

---
id: STATE-WP-0084
type: workplan
title: "Derive private repositories, and report unreadable as unreadable"
domain: infotech
repo: state-hub
status: finished
owner: codex
topic_slug: infotech
created: "2026-08-26"
updated: "2026-08-31"
quality_dod: DoD-Ok
quality_dod_at: "2026-08-31"
quality_dod_by: codex
quality_dod_note: >-
  Credential delivery, safe consumer behavior, nine-repository reviewed apply,
  and nine-noop repeat verification are evidenced without storing a secret.
related:
  - CUST-ADR-012
  - STATE-WP-0083
  - MASON-WP-0003
state_hub_workstream_id: "9c4b8c79-edba-5d2d-ad37-a45707e89304"
---

# Derive private repositories, and report unreadable as unreadable

## Goal

Consume the forge read credential that `ops-mason/MASON-WP-0003` builds, so
derivation covers private repositories — and make a repository central cannot
read report as *unreadable* rather than as an error or, worse, as a repository
whose records no longer derive.

## Why this exists

`STATE-WP-0083-T04` reset 121 repositories on 2026-08-26. Nine failed
identically:

```text
fatal: could not read Username for 'https://forgejo.coulomb.social'
```

`rapp-core-hub`, `rapp-issue-core`, `rapp-openbao`, `rapp-policy-nexus` and five
others are invisible to derivation entirely. `ADR-012`'s premise — the forge is
the projection source — holds only for repositories central can read, and
nothing in the current design says so.

## Split from MASON-WP-0003

The credential lane itself (OpenBao policy, AppRole, KV path, and the
`warden route find` catalog entry) is `ops-mason`'s to build: that repository
exists to build "AppRoles, policies, and KV secret paths so ops-warden always
has something real to route to". Deployment plumbing and derivation code are
this repository's.

T01 does **not** depend on the credential and should land first. T02 and T03
wait on `MASON-WP-0003-T02`.

## Distinguish "cannot read" from "does not exist"

```task
id: STATE-WP-0084-T01
status: done
priority: high
state_hub_task_id: "e955067d-2152-59f7-b672-12755ceb26b0"
```

A repository the hub is not permitted to read must be reported as unreadable —
a named condition — not as a generic clone error and never as an empty
derivation.

Today the reset treats a failed clone as an error, which is correct only by
accident: if a clone ever returned empty instead of failing, every record in
that repository would be proposed for retirement. That is the failure that
nearly retired `vergabe-teilnahme`, and it must be impossible here by
construction rather than by luck.

An empty clone result and an unreadable repository must both be incapable of
producing retirement work. Retirement proposals require a *successful* read
that positively lacks the record.

This also carries the decision in `MASON-WP-0003-T01`: a per-repository grant
is only safe to choose if a missing grant surfaces as a named condition. Until
this lands, the narrow option fails silently.

Acceptance: an unreadable repository is reported as unreadable; a clone that
succeeds but returns nothing produces no retirement proposals; both cases are
covered by tests that fail if the retirement path is reachable from either.

**Done 2026-08-26.** `api/services/forge_projection.py`:

- `ForgeUnreadableError` subclasses `ForgeDeriveError`, so existing callers
  keep catching it while new ones can tell the cases apart. A permission-shaped
  git failure is classified into it; a genuine fault stays a plain error.
  Forgejo's 404 for an unauthenticated private repository classifies as
  unreadable, because the two are indistinguishable at this layer and the safe
  reading of an ambiguous answer is the one that cannot destroy a record.
- `_run_git` runs with `GIT_TERMINAL_PROMPT=0`. Without it an unattended pass
  blocks on a username prompt instead of failing, and an unreadable repository
  is only observable if it fails.
- `DerivedProjection.records_source_present` / `.retirement_eligible` separate
  "no records found" from "no records exist". A checkout with no `workplans/`
  directory can no longer evidence an absence.
- `diff_against_hub` withholds `stale` rather than computing and filtering it,
  and says so in `stale_withheld`. `would_remove` is 0 for such a source.
- `reset_repository_projection` returns `status="unreadable"` for a repository
  it may not read, and refuses retirement from an ineligible source **even when
  `acknowledge_retirements=True`** — consenting to a conclusion is not the same
  as the evidence for it existing.
- `reset_fleet_projection` keeps unreadable repositories out of `errors`, in
  their own bucket with its own count. Nine repositories in an error bucket
  read as nine broken repositories; they were nine we were not allowed to read.

13 tests in `TestUnreadableIsNotMissing` (`tests/test_forge_projection.py`);
58 pass across the forge/projection/backfill suites.

## Deliver the credential to the pod

```task
id: STATE-WP-0084-T02
status: done
priority: medium
state_hub_task_id: "d8d41a89-1ebc-5118-bb73-cf8d8c114e16"
```

Blocked on `MASON-WP-0003-T02`.

Make the credential available to the `state-hub` deployment as a Kubernetes
Secret, mounted or injected, referenced by the chart the same way
`state-hub-env` already is.

It must not be baked into the image and must not be committed to the chart.
Rotation must not require a chart change or a redeploy.

Acceptance: the pod can read the credential; nothing in the repository contains
it; rotating the token does not require a redeploy.

**Chart landed 2026-08-27; awaits the minted token.** The cluster has no OpenBao
agent injector and no secrets-store CSI driver — checked, not assumed — so
there is nothing to inject with. The pod authenticates to OpenBao itself, which
is what `MASON-WP-0003-T02`'s Kubernetes auth role was built for.

- `templates/serviceaccount.yaml` creates ServiceAccount `state-hub`, and the
  Deployment now sets `serviceAccountName`. The auth role binds to this name and
  deliberately not to `default`, so until this ships the pod cannot authenticate
  at all. **This changes the identity the running pod uses** — expect a pod
  restart on upgrade.
- The OpenBao token is a *projected* ServiceAccount token with audience
  `openbao`, not the legacy auto-mounted one. The auto-mounted token's audience
  is the API server, so a copy of it is a credential for the cluster; this one
  is only accepted by OpenBao, and the kubelet rotates it in place.
- `forgeRead.*` carries coordinates only — address, role, KV path, key. No
  credential is a chart value, an image layer, or a Kubernetes Secret in this
  release. Rotating the token in OpenBao needs no chart change and no redeploy:
  the value is re-read every 5 minutes.
- Default `forgeRead.enabled: false`. A hub without the credential still derives
  every public repository, so this is added capability, not a prerequisite.

Remaining before this is `done`: the minted read-only Forgejo token in the KV
path (`paste_once_provision`, outside both repositories), and a deploy with
`forgeRead.enabled=true` plus the real address and path in
`deploy/railiance/apps/helm/state-hub-values.yaml`.

**Done 2026-08-31.** The live `railliance01` release has
`forgeRead.enabled: true`; the API pod runs as ServiceAccount `state-hub` with
the projected OpenBao token and coordinate-only configuration. A non-secret
in-pod resolution check returned `available`. The Forge token remains outside
Git and is re-read through the five-minute cache, so rotation needs no chart
change or pod redeploy.

## Teach the derivation to use it

```task
id: STATE-WP-0084-T03
status: done
priority: medium
state_hub_task_id: "b22b24d9-7ed3-533c-bda7-3130693cf4d2"
```

Blocked on T02.

`derive_from_forge()` clones anonymously. It should use the credential when
present and continue working without it — a hub with no token must still derive
public repositories rather than failing wholesale.

Credentials must never reach the clone URL in a way that lands in logs or
process listings: use a credential helper or an HTTP header, not an embedded
userinfo URL.

Acceptance: private repositories derive; a hub without the credential still
derives public ones; no credential appears in logs, process listings, or
recorded clone URLs.

**Code landed 2026-08-27**, ahead of T02 — writing the consumer does not need
the secret to exist. `api/services/forge_projection.py`:

- `forge_read_token()` reads `FORGE_READ_TOKEN_FILE` in preference to
  `FORGE_READ_TOKEN`. A mounted file is what lets T02 satisfy "rotation must
  not require a redeploy"; the environment cannot be rotated in place. An
  unreadable token *file* returns `None` rather than falling back to the
  environment — falling back would let a broken mount look like success while
  quietly using a stale value.
- Absent is a supported state, not a degraded one: with no credential the git
  environment is untouched and public derivation runs exactly as before.
- The credential is passed as `GIT_CONFIG_COUNT`/`GIT_CONFIG_KEY_0`/
  `GIT_CONFIG_VALUE_0` setting `http.extraHeader`, not as `-c` on the command
  line and not as userinfo in the URL. Both alternatives put the token in the
  process listing, where anything that can run `ps` reads it.
- `_run_git` redacts the token from `ForgeDeriveError` before raising, because
  that message is logged, stored in a reset outcome, and returned over the API.
- `api/services/forge_credential.py` resolves the credential from a mounted
  file, an environment variable, or OpenBao via Kubernetes auth, in that order.
  Production uses the third; the first two make the code runnable and testable
  outside the cluster. Configuring a file means *that* file and nothing else —
  no fall-through, so a broken mount cannot silently resolve to a stale value.
- Every failure resolves to `None`, never an exception: no configuration, no
  network, OpenBao down, permission denied. Raising would convert "nine
  repositories are unreadable" into "the whole pass failed", which is the
  outcome T01 exists to prevent.
- The resolved value is cached for 5 minutes — a fleet reset of 121
  repositories must not authenticate 121 times, and a rotated token must still
  be picked up without a redeploy.

Nine tests in `TestForgeCredential` cover each of those, including the two that
are silent when wrong: token-in-argv and token-in-exception. 43 pass in
`tests/test_forge_projection.py`; 717 pass across the suite.

Remains `progress` because the acceptance clause "private repositories derive"
cannot be observed until T02 supplies a credential; T04 confirms it.

**Done 2026-08-31.** A transaction-rolled-back review successfully derived all
nine private `rapp-*` repositories at exact Forge commits. No repository was
unreadable or errored, and no retirement was proposed. The anonymous/public
path remains covered by the existing absent-credential tests.

## Confirm the nine

```task
id: STATE-WP-0084-T04
status: done
priority: medium
state_hub_task_id: "33c09a78-5974-57c6-bb7f-ad73082046bb"
```

Blocked on T03.

Re-run the fleet reset and confirm the nine repositories derive rather than
error. Expect a real diff for each — quite possibly a large one, since none has
ever been reconciled against its files.

Treat the first diff for each as review material, not as work to apply. Every
repository examined this way for the first time has produced findings.

Acceptance: fleet errors fall from nine to zero; each newly readable repository
has its first diff reviewed before anything is applied.

**Done 2026-08-31.** The first review returned seven bounded source/status
updates and two no-ops, with zero creations, retirements, refusals, or errors.
Those exact commits were applied through the guarded per-repository API with
retirement disabled. The repeated review returned nine no-ops and zero errors.
Evidence: `docs/evidence/STATE-WP-0084-private-forge-cutover-2026-08-31.md`.