state-hub/docs/rollout-dependency-edge-receiving.md

# Rollout package: dependency-edge receiving (STATE-WP-0094-T05)

> **REQUIRES FOUNDER PER-CHANGE AUTHORIZATION.** This package is prepared, not
> executed. Nothing below has been run against railiance01. No image was
> promoted, no Helm command, migration, schedule or production write was
> performed while preparing it. The 2026-10-04 "Do the rollout" instruction
> covered the qualifier receiving change (revision 71) only and does not
> authorize this change.

Prepared 2026-10-04 from `main` at `c287ed8`. Shape mirrors the previous
authorized rollout (`docs/evidence/2026-10-04-receiving-rollout.json`,
`the-custodian/docs/changes/state-hub-qualifier-rollout-2026-10-04.json`).

## What ships

| Commit | Change | Runtime effect |
| --- | --- | --- |
| `fceb5f2` | Forge receiving derives `depends_on` edges and reconciles source-owned `workplan_dependencies` rows | New `created_edges`/`removed_edges`/`unresolved_edges`/`external_edges` in receipt outcomes; receiving writes dependency rows |
| `3b9a5c5` | `GET /messages/` accepts `offset` (stable `created_at,id` order); C-25 pages replies | Additive query parameter |
| `c287ed8` | Dry-run edge preview in `diff_against_hub`; classifier no longer collapses duplicate edge keys | Pure preview; reconcile uses the shared planner |
| `57f2751` | Historical projection / owner-return planner source (already in `5dd3df9`) | Ships only if the live image predates `5dd3df9` |

Migration: `b3c4d5e6f7a8` (revises `a0b1c2d3e4f5`, the schema recorded as live
in revision 71). Additive only: two nullable columns
`workplan_dependencies.source_owner` (varchar 160) and `source_commit`
(varchar 64), plus index `ix_workplan_dependencies_source_owner`. Existing rows
stay NULL, i.e. externally authored, so receiving never removes them.

Local verification (2026-10-04, scratch database on the local PostgreSQL):
`a0b1c2d3e4f5 -> b3c4d5e6f7a8 -> a0b1c2d3e4f5 -> b3c4d5e6f7a8` passed with a
pre-existing dependency row preserved through every step (owner stayed NULL).
`alembic check` reports long-standing model/migration drift on unrelated tables
only; nothing on `workplan_dependencies`. Full test suite: 1018 passed.

## 0. Preconditions (read-only)

1. Founder authorization for this exact change and target commit, recorded as a
   change proposal in `the-custodian/docs/changes/` (copy the shape of
   `state-hub-qualifier-rollout-2026-10-04.json`; `outcome: pending`).
2. Identify the live revision and image. The historical-receiving proposal
   (`main-5dd3df9`) has no recorded outcome, so do not assume revision 71:
   ```bash
   helm -n state-hub history state-hub --max 5
   kubectl -n state-hub get deploy -o jsonpath='{range .items[*]}{.metadata.name}={.spec.template.spec.containers[0].image}{"\n"}{end}'
   kubectl -n state-hub exec deploy/state-hub -- /app/.venv/bin/python -m alembic current
   ```
   Record `PREV_REV`, `PREV_IMAGE` and the live schema. Expected schema:
   `a0b1c2d3e4f5`. Stop if it is anything else.
3. Fresh CPU headroom for the surge (API 250m, MCP 10m, migrate job 50m) as in
   the previous rollout; stop on unrelated pending pods.
4. The proof repository is `state-hub` itself. Read its current hub edges so
   the post-check can tell receiving's rows from earlier ones:
   ```bash
   curl -s "http://10.43.68.154:8000/workplans/?repo=state-hub" > /tmp/edge-pre-workplans.json
   ```
   and, inside the cluster DB (read-only):
   `select count(*), source_owner from workplan_dependencies group by 2;`
   (column does not exist yet; before migration expect an error, after it all
   NULL).

## 1. Image

1. Push `main` (operator; agents in this session did not push). The Forgejo
   `image.yaml` workflow builds on pushes touching `api/**`, `migrations/**`,
   `scripts/**` etc. and tags `main-<short sha of the pushed head>`.
2. Confirm the build-and-push run succeeded and that its source commit contains
   `c287ed8`:
   `git merge-base --is-ancestor c287ed8 <image source sha>`.
3. Record `TAG=main-<sha7>` and the image digest. Do not use a mutable tag.
4. Verify the receiving source digest in the image equals the tested source:
   `sha256sum api/services/forge_projection.py` at the image source commit vs.
   `kubectl exec ... sha256sum /app/api/services/forge_projection.py` after
   rollout.

## 2. Server-side dry run

```bash
helm pull / copy chart: deploy/railiance/apps/charts/state-hub -> /tmp/cust-edge-rollout/chart
helm -n state-hub upgrade state-hub /tmp/cust-edge-rollout/chart \
  --reuse-values --set image.tag=$TAG --dry-run=server
```

Expected diff vs. live: image tag, version labels and the consequent config
checksum only. Any other change stops the rollout.

## 3. Upgrade (the authorized action)

```bash
helm -n state-hub upgrade state-hub /tmp/cust-edge-rollout/chart \
  --reuse-values --set image.tag=$TAG --atomic --timeout 3m
```

The chart's `pre-upgrade` hook job (`migrate-job.yaml`) runs
`alembic upgrade head`, applying `b3c4d5e6f7a8`. `--atomic` restores the prior
release if the hook or rollout fails.

## 4. Post-checks

- `helm -n state-hub status state-hub` = deployed, new revision `NEW_REV`.
- API and MCP `1/1` Ready; container image IDs match the recorded digest.
- `alembic current` = `b3c4d5e6f7a8`.
- `\d workplan_dependencies` shows `source_owner`, `source_commit`; index
  present; every pre-existing row has `source_owner IS NULL`.
- `GET /state/health` OK; `GET /messages/?to_agent=state-hub&limit=1&offset=0`
  answers 200.
- Receiving source digest in the container matches the tested file.

## 5. Post-deploy proof (normal immutable Forge receive)

Use the normal receiving endpoint only. No task PATCH, no direct consistency
repair, no C-20 `--fix` between the two receipts.

1. `SHA=$(git ls-remote <forge state-hub> refs/heads/main | cut -f1)` - the
   exact forge head at receipt time (it must contain `c287ed8`).
2. Receipt:
   ```bash
   curl -s -X POST http://10.43.68.154:8000/repos/state-hub/work-record-projection/reconcile \
     -H 'Content-Type: application/json' -d "{\"expected_commit\":\"$SHA\"}"
   ```
   Expected at the time of writing (from the live C-20 report):
   - `created_edges` includes `STATE-WP-0095-T04 starts_after STATE-WP-0095-T03`
     and, if `RMGR-WP-0019-T03` resolves uniquely fleet-wide,
     `STATE-WP-0095-T04 starts_after RMGR-WP-0019-T03`, each with
     `source_owner = forge:state-hub`, `source_commit = $SHA`;
   - `unresolved_edges` lists `STATE-WP-0078 -> ACTIVITY-WP-0026` (`missing`)
     unless that workplan has been received by then;
   - `external_edges` lists declarations already present as C-20/API rows;
   - `removed_edges` is empty (no owned rows exist before the first receipt);
   - `status: applied`.
3. Repeat the identical request (fresh idempotency, same `$SHA`): expected
   `status: noop`, zero created/removed edges; `unresolved_edges` is reported
   again by design.
4. DB proof: `select source_owner, count(*) from workplan_dependencies group by 1`
   - new rows only under `forge:state-hub`; NULL-owner count unchanged.
5. Record evidence as `docs/evidence/<date>-dependency-edge-receiving-rollout.json`
   in the shape of `2026-10-04-receiving-rollout.json` (image, digest,
   `helm_revision`, schema, both receipt outcomes, DB counts).

A failed receipt (e.g. a historical identifier collision as seen for
`the-custodian`) rolls back transactionally; record it as a failure, not as
edge-receiving proof.

## 6. Rollback

- Upgrade failure: `--atomic` restores `PREV_REV` automatically.
- Post-upgrade verification failure:
  `helm -n state-hub rollback state-hub $PREV_REV --wait --timeout 3m`.
- **Schema: do not downgrade.** `b3c4d5e6f7a8` is additive and nullable; the
  previous image ignores the columns (its ORM model does not map them, and
  inserts leave them NULL). Downgrade would drop `source_owner` and so erase
  the ownership marker of every receiving-created row, turning those rows into
  "external" edges that later receipts could never remove. Retain schema and
  data on rollback.
- Receiving-created rows under the previous image: they remain as ordinary
  dependency edges. If they must go, delete only rows with
  `source_owner = 'forge:<repo>'` - a separate authorized data change, not
  part of rollback.

## Open risks

- Live revision/image is unconfirmed (historical-receiving outcome not
  recorded); step 0.2 is mandatory.
- Pre-existing C-20/API edges are never adopted; a removed declaration does not
  remove them.
- A checkout without `workplans/` is refused before edges are touched while
  the hub still holds that repository's workplans, so owned edges are not
  removed on an empty source; the dry-run preview does not model that refusal
  and would list those removals (preview-only discrepancy).
- No scheduled freshness is part of this change; receiving remains manual.