the-custodian/docs/weekly-allocation-report.md

# Retained weekly allocation assessment

`scripts/weekly_allocation_report.py` implements the offline evidence join for
CUST-WP-0071-T05. It reads the existing cluster collector, Custodian allocation
reconcile, Prometheus query-response capture and resource-control inventory.
It performs no collection, deployment, network calls, task writes or delivery.
It is suitable for an accepted durable executor; it does not install one.

## Inputs and retained evidence

The required inputs are:

- `--cluster`: `railiance.cluster-resource-observation.v1`, produced by
  railiance-cluster's `make cluster-observe REMOTE=railiance01`.
- `--allocation`: `custodian.allocation-reconcile.v1` from
  `scripts/reconcile_railiance_allocation.py`, matching the same capture and
  workload inventory. Run both against the same retained observation.
- `--telemetry`: the existing September 28 query-capture shape: `captured_at`,
  `window: 7d`, `step: 5m`, and `queries` mapping each metric name to its exact
  `promql` and Prometheus `response`. Preserve original sample evaluation times;
  do not restamp historical query results. These describe the query window,
  not independently verified raw scrape completeness.
- `--inventory`: resource-control's `data/resources/reef-railiance-k3s.json`.
  Dated portfolio capacity is retained as portfolio context, never summed with
  the live cluster or treated as current scheduler evidence.
- `--provenance`: exact source collector revisions and `source_observation`
  matching the cluster capture, as in the September 28 allocation provenance.
- `--as-of`: explicit evaluation timestamp; identical retries use the same value.
- `--output`: a new retained JSON path. Exact identical retries succeed; changed
  content at that path fails and requires an explicit revision path. Writes use
  a complete temporary file and atomic create-without-replacement publication.

Optional `--previous` selects the previous retained report. Deltas require the
same resource, exact query texts, sample step, seven-day window and consecutive
capture endpoints within one sample step. Other comparisons stay unknown; the
report still records disappeared workloads. Recommendation identity depends on
resource/namespace/workload/review purpose, not the date or recommendation text.
Existing receiving-task links survive retries and subsequent reports.

Metric names are `cpu_p95_m`, `cpu_peak_m`, `memory_p95_bytes`,
`memory_peak_bytes`, `cpu_throttle_ratio`, `latency_p95_seconds`, `errors`,
`request_count`, `restarts`, `oom_events`, `cpu_sample_count` and
`memory_sample_count`. Metrics are nonnegative, finite instant-vector aggregates.
Workload queries must carry exactly `namespace` and `workload` identity labels
(an optional Prometheus `__name__` is allowed). Extra labels or duplicate series
are not silently aggregated. Namespace-only results are retained separately;
they never become per-workload values, even for a currently single-workload
namespace. Namespace peaks and percentiles are not additive.

Missing series, failed queries and non-finite values remain null with explicit
reasons. Observed zero remains zero. Evaluation counts never prove raw scrape
coverage. CPU/memory limits and instantaneous observations come from the cluster
snapshot; historical latency, request/errors, OOM or throttling are not inferred
from that snapshot. Unknown operating margin remains null; reservation residual
does not establish processing capacity or release acceptance.

## Owner decisions

Without a complete current owner review the result is `investigate`, with low
confidence and explicit evidence gaps. `--reviews` accepts a JSON mapping keyed
by the stable recommendation ID. A review provides `decision` (keep, increase,
decrease or investigate), `owner`, `period_end` matching telemetry capture,
`reviewed_at`, `evidence_refs`, `receiving_task`, `configuration`, `rollback_ref`,
`coverage_ref`, `service_acceptance_ref`, and `margin_ref`. Optional `outcome`
records what happened to the preceding recommendation. These references are
owner assertions to be checked during acceptance, not proof generated by this
adapter. Wrong-owner/window reviews, missing workload metrics, zero CPU requests,
unknown owners and stale/future evidence keep the result at investigate and
retain the proposal for review. No idle sample automatically recommends a cut.

Only new observation evidence and verified owner disposition can complete T02–T04.
In particular, the historical Vergabe sample does not establish the attended
two-user p95/error acceptance target or incremental database attribution.

## Disabled execution contract

The owner-source candidate is
`activity-core/activity-definitions/weekly-railiance-allocation-review.md`.
Its fixed command is:

```sh
python3 scripts/weekly_allocation_report.py --request config/weekly-allocation-request.json
```

The request is intentionally unbound and the command fails until the accepted
executor supplies real input paths, a fixed `as_of`, and a durable output path.
Paths resolve relative to the request file. The executor must snapshot this
request per run, keep it stable for retries, bind a real previous report, and
retain reports beyond the seven-day raw-metric retention. The committed null
template is not a production storage location or collection implementation.

Activity Core source `4ae00fc` selects its existing ADR-007 bounded-operation
executor with fixed owner commands; both definitions remain disabled. The prior
coding profile cannot run Python or reach the Hub. Runtime owners must still
bind accepted source/fleet mounts, separate private retained state and scoped
network access. Activity Core owns the Monday 08:00 Europe/Berlin schedule and
bounded-activity/context/progress-sink evidence, with one Temporal attempt.
The full binding contract is `activity-core/docs/custodian-bounded-operations.md`.
The adapter's matching recommendation IDs are the downstream upsert keys for
existing owner work records; it does not itself create those records. Delivery,
explicit owner acknowledgment, independent missed/failed-run detection and
recovery remain unbound. A report file or completed bounded activity alone proves none
of them. T05 needs the first unattended schedule receipt and exercised recovery.

## Verification

```sh
python3 -m unittest discover -s tests -p test_weekly_allocation_report.py -v
```

The tests cover historical namespace-only input, missing/zero/NaN, invalid
aggregation, query failure, compatible/incompatible periods, stable identities
and owner-task links, stale/future evidence, owner review, resource/capture
mismatch and retained retry/collision behavior.

## In-cluster runner (CUST-WP-0071-T05, founder decision 2026-10-21)

`docs/decisions/2026-10-21-cust-wp-0071-delivery-interview.md` selects an
in-cluster Kubernetes CronJob for T05 and supersedes the Activity Core
placement above. `scripts/weekly_allocation_incluster.py` is the job entry
point. It is standard-library only and reuses the existing owner code
unchanged: railiance-cluster `build_observation`, `reconcile.join` (PyYAML is
now imported only by the reconcile CLI) and this adapter's `build_report` and
`retain`.

- Reads: the Kubernetes API through the pod ServiceAccount, with get/list on
  nodes, pods, persistentvolumeclaims, persistentvolumes, storageclasses and
  metrics.k8s.io nodes/pods only. Prometheus is reached through
  `PROMETHEUS_URL` with `config/weekly-allocation-queries.json`.
- Week identity: the ISO week of (run time - 6h). The telemetry window ends at
  Monday 06:00 UTC, so consecutive windows are exactly seven days apart in CET
  and CEST. Prometheus keeps seven days of data, so the run must happen on the Monday. A
  2026-10-21 (Wednesday) read-only dry run against railiance01 retained the
  report, but per-namespace CPU sample counts reached at most 1512 of 2016,
  because the window start had already expired.
- Retention: `<data>/reports/<YYYY>-W<ww>.json` (create-without-replacement),
  per-attempt inputs under `<data>/runs/<week>/<attempt>/`, and an append-only
  `<data>/receipts.jsonl` with status `retained`, `already-retained` or
  `failed`, previous week, `missed_weeks`, comparability and failed queries.
- Idempotency: a retained week is a no-op success. A failed attempt leaves no
  report, so the Job retry or a manual run collects again.
- Delivery (founder decision 2026-10-21, "Hub message + repo tasks"): with
  `STATE_HUB_URL` set, each newly retained week produces one State Hub message
  from `custodian-allocation-review` to `the-custodian`. The message carries a
  summary, the report path and its sha256, and its id is recorded in
  `receipts.jsonl`. A week that already has a message id is never posted again.
  If a post fails, the receipt says `delivery: failed`, the Job exits 1, and
  its retry posts for the already-retained week. The next Custodian session
  reviews the report and records recommendations as tasks in the owner
  repositories (ADR-001). The job itself creates no tasks.
- Missed/failed-run check: `scripts/weekly_allocation_check.py`, which
  `scripts/security_flow_daily.py` calls. It flags `failed` when the latest
  receipt failed, and `missing` when the due ISO week has no retained report
  (the current week falls due on Monday 12:00 Europe/Berlin). The agent is
  notified once per distinct flag set. It is inactive until
  `CUSTODIAN_ALLOCATION_RECEIPTS` (or `--allocation-receipts`) is set to `hub`
  (delivery messages, where a failure shows as missing) or to a path holding
  a copy of `receipts.jsonl`. Set it in `infra/security-flow/` only when the
  CronJob is activated.
- Not done: owner acknowledgment. Workload rows stay `investigate` because the queries
  are namespace-level. kube-state-metrics does not export `kube_pod_labels`, so
  no workload-labelled series exist yet.

Executor manifests (suspended, not registered with ArgoCD) live in
railiance-platform `argocd/platform-addons/custodian-allocation-review/`, with
the draft Application in `argocd/railiance01/drafts/`.

```sh
python3 -m pytest tests/test_weekly_allocation_incluster.py tests/test_weekly_allocation_report.py -q
```