tegwick
fc38ca0fdf
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a06bfe-2a55-7ed3-bacd-879977b099bf
162 lines
6.9 KiB
Markdown
162 lines
6.9 KiB
Markdown
---
|
||
id: ACTIVITY-WP-0036
|
||
type: workplan
|
||
title: "Bind queue mutations to worker identity and active leases"
|
||
domain: infotech
|
||
repo: activity-core
|
||
status: active
|
||
owner: codex
|
||
topic_slug: activity-core
|
||
priority: high
|
||
created: "2026-08-23"
|
||
updated: "2026-09-04"
|
||
related:
|
||
- ACT-ADR-005
|
||
- ACTIVITY-WP-0026
|
||
- ACTIVITY-WP-0032
|
||
- HARNESS-WP-0003
|
||
state_hub_workstream_id: "01e6d5d4-6e1a-5f0d-81f0-ded97e0f71cd"
|
||
---
|
||
|
||
# Bind Queue Mutations to Worker Identity and Active Leases
|
||
|
||
## Origin
|
||
|
||
Activity Core's review of rein-aharness ADR-002 found two enforcement gaps at
|
||
the `ops_run` boundary:
|
||
|
||
1. `ACTIVITY_CORE_WORKER_TOKEN` authenticates a shared caller class, but the
|
||
authenticated principal is discarded and the caller supplies any
|
||
`worker_id` in the request body.
|
||
2. heartbeat, completion, and failure accept an owned `claimed` row even when
|
||
its lease has expired, until another claim or explicit expiry request happens
|
||
to reopen it.
|
||
|
||
These gaps make queue ownership advisory at precisely the point where a
|
||
repository executor needs a durable acceptance decision. Rein-side
|
||
cancellation remains necessary, but cannot replace server-side enforcement.
|
||
|
||
## Boundary and rollout rules
|
||
|
||
- Activity Core authenticates queue callers and owns lease acceptance.
|
||
- The queue `worker_id` is separate from the governed actor (`agt`) used by
|
||
Glas and sand-boxer.
|
||
- Operator SSO and break-glass credentials are not worker identities and must
|
||
not claim, heartbeat, complete, or fail runs through the normal worker API.
|
||
- Existing result normalization and retry ceilings remain unchanged.
|
||
- Production must receive the configured worker identity in the same rollout
|
||
that activates strict binding; do not strand the live consumer between API
|
||
and configuration revisions.
|
||
|
||
## Confirm the cross-repo responsibility contract
|
||
|
||
```task
|
||
id: ACTIVITY-WP-0036-T01
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "fe4013f0-83ba-5697-8a3c-3fea1b0daedc"
|
||
```
|
||
|
||
Review rein-aharness ADR-002 against the implemented Activity Core boundary.
|
||
Report any distinction between current state and target guarantees, and require
|
||
the responsibility matrix to assign both worker authentication and lease
|
||
acceptance to an enforceable owner.
|
||
|
||
Activity Core sent required edits in State Hub message
|
||
`428abb02-75dc-450d-a7f6-56fc715409ab`: the ADR must disclose the current
|
||
self-asserted identity, require credential-to-worker binding, require
|
||
server-side rejection after lease expiry, and keep operator identity out of the
|
||
normal worker mutation path.
|
||
|
||
## Reject mutations without an active lease
|
||
|
||
```task
|
||
id: ACTIVITY-WP-0036-T02
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "57da10c2-db6f-5a41-b306-3629901751da"
|
||
```
|
||
|
||
Lock the target row while deciding heartbeat, completion, or failure. Require
|
||
the row to be claimed by the caller and `lease_until` to be strictly later than
|
||
the server's current UTC time. A missing lease, or a deadline equal to or before
|
||
the decision time, is a conflict and must not update result or state. Stale
|
||
claim reopening must use the same deadline boundary.
|
||
|
||
Done when unit/API tests cover missing, equal, expired, active, and wrong-owner
|
||
leases and prove no late completion or heartbeat can revive an expired claim.
|
||
|
||
Implemented with row-level locking and one strict server-time predicate shared
|
||
by heartbeat, completion, and failure. Stale reopening now uses the same
|
||
inclusive expiry boundary (`lease_until <= now`). Tests prove missing, equal,
|
||
expired, active, and wrong-owner behavior without mutating late rows.
|
||
|
||
## Bind authenticated worker credentials to the claim owner
|
||
|
||
```task
|
||
id: ACTIVITY-WP-0036-T03
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "de090738-01bd-58c7-80fa-6d36db222fc5"
|
||
```
|
||
|
||
Add an explicit configured queue worker identity and bind a valid worker token
|
||
to that identity. Compare the body `worker_id` to the authenticated identity
|
||
before every worker mutation and persist only the authenticated identity as
|
||
`claim_owner`. Refuse production worker mutations when token or identity
|
||
configuration is incomplete. Preserve an explicitly enabled local-development
|
||
path without presenting it as authenticated production behavior.
|
||
|
||
Done when tests prove spoofed worker ids, operator/SSO credentials on worker
|
||
mutations, missing production identity configuration, and invalid worker
|
||
tokens fail closed.
|
||
|
||
Implemented `ACTIVITY_CORE_WORKER_ID` as the non-secret identity bound to the
|
||
worker token. Claim and close paths validate the compatibility body field and
|
||
persist only the authenticated identity. Operator/SSO credentials are removed
|
||
from normal worker mutations; unauthenticated development requires the
|
||
existing explicit opt-in, and the former implicit open mode is gone.
|
||
|
||
## Document, deploy, and prove the boundary
|
||
|
||
```task
|
||
id: ACTIVITY-WP-0036-T04
|
||
status: progress
|
||
priority: high
|
||
state_hub_task_id: "1228e10a-067b-54db-baa3-a8fc9549492b"
|
||
```
|
||
|
||
Update the queue contract, runbook, and Railiance manifest with the non-secret
|
||
configured worker identity. Run focused and full tests, deploy API and consumer
|
||
configuration without a compatibility gap, and prove one authenticated claim
|
||
plus active heartbeat and a rejected mismatched/expired mutation. Return the
|
||
revision and bounded evidence to rein-aharness for its ADR acknowledgement
|
||
cycle.
|
||
|
||
Source verification: 468 tests passed with one live integration test skipped;
|
||
Python compilation, whitespace checks, and parsing all 16 Railiance Kubernetes
|
||
documents passed. The manifest binds the existing production worker token to
|
||
`rein-aharness@railiance01`. Production rollout and live rejection evidence
|
||
remain before this task and workplan can finish.
|
||
|
||
Production rollout completed on 2026-08-23 at revision `f0a897e`. The API pod
|
||
is Ready on the new image/configuration, the existing rein-aharness claim loop
|
||
received HTTP 200 on its first two authenticated polls, and a valid credential
|
||
paired with a spoofed body identity was rejected HTTP 403. Bounded evidence is
|
||
in `docs/evidence/ACTIVITY-WP-0036-queue-boundary-rollout-2026-08-23.md`.
|
||
|
||
No run was claimed during the observation window, so a natural active
|
||
heartbeat and rejected late close remain before T04 can be marked done. The
|
||
sole open row was an intentionally non-automated retention record and was not
|
||
modified to manufacture evidence.
|
||
|
||
Follow-up 2026-09-04: 24 natural runs have been claimed since rollout and all
|
||
terminated in 5–34 seconds. The rein consumer schedules its first heartbeat at
|
||
one third of the 900-second lease, capped at 300 seconds, so production API
|
||
logs contain accepted claim/fail traffic but zero heartbeat requests. This is
|
||
an observability gap in the consumer timing, not evidence that Activity Core's
|
||
lease enforcement is absent. State Hub message
|
||
`1250c36a-9fe8-45dc-bb5a-d5fa85e49fd0` asks rein-aharness to heartbeat once
|
||
immediately after claim, retain periodic heartbeats, and return a natural
|
||
claim/heartbeat/close trace. It also requests an isolated labeled late-close
|
||
smoke; no real workload will be delayed or expired to manufacture evidence.
|