coulomb/rapp-canned-prompts
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Lease-watch verdict must check what it claims

Browse source 
Version 3 reported 'survived past the lease TTL' after a clean 12-minute window
on a 13-minute-old pod. Every measurement in it was accurate; the conclusion
was not, because nothing checked that the observation window had actually
exceeded the 30-minute lease it claimed to have outlasted.

Fifth defect in this instrument, and the first to err toward reassurance.
Versions 1 and 2 cried wolf, which provokes investigation. This one would have
been believed, and readiness_state: verified recorded on it — the same way
live-image-digest-match would have been believed. A check reporting success it
has not established is indistinguishable from one that works, until it matters.

The verdict now requires uptime > lease TTL and reports INCONCLUSIVE when a
window is clean but too short. 'Clean' and 'proven' are different claims and
only one of them was being measured.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
This commit is contained in:
tegwick 2026-09-08 10:48:35 +02:00
parent bd5ee3dfbb
commit bdbef58259
8 changed files with 55 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

21
tools/lease-watch.sh
View file

@ -6,7 +6,7 @@
# transient API hiccup was recorded as a service failure and produced a FAILED
# verdict for a service that never returned a single 503. A check that cannot
# tell its own failure from the failure it watches for is worse than no check.
OUT="$1"; MINUTES="${2:-20}"
OUT="$1"; MINUTES="${2:-20}"; LEASE_TTL_MIN="${3:-30}"
DEADLINE=$(( $(date +%s) + MINUTES * 60 ))
: > "$OUT"
echo "started $(date -Is) — ${MINUTES}m watch (runtime lease TTL 30m)" >> "$OUT"
@ -41,14 +41,27 @@ else
P503="unavailable"
fi
UP=$(kubectl -n canned-prompts get pod -l app.kubernetes.io/name=canned-prompts -o jsonpath='{.items[0].status.startTime}' 2>/dev/null)
UPTIME_MIN=$(python3 -c "
import datetime as dt, sys
try:
s = dt.datetime.fromisoformat('$UP'.replace('Z','+00:00'))
print(int((dt.datetime.now(dt.timezone.utc)-s).total_seconds()//60))
except Exception:
print(-1)
" 2>/dev/null || echo -1)
R=$(kubectl -n canned-prompts get pod -l app.kubernetes.io/name=canned-prompts -o jsonpath='{.items[0].status.containerStatuses[0].restartCount}' 2>/dev/null)
{
echo "finished $(date -Is)"
echo "samples=$SAMPLES not_ready=$NOTREADY query_failed=$QUERYFAIL"
echo "kubelet readiness 503s in window: $P503 (probe every 5s)"
echo "pod started $UP, restarts=$R"
if [ "$NOTREADY" -eq 0 ] && [ "$P503" = "0" ]; then
echo "RESULT: survived — continuously ready past the lease TTL"
echo "pod started $UP, uptime=${UPTIME_MIN}m, restarts=$R"
if [ "$NOTREADY" -eq 0 ] && [ "$P503" = "0" ] && [ "$UPTIME_MIN" -gt "$LEASE_TTL_MIN" ]; then
echo "RESULT: survived — continuously ready, uptime ${UPTIME_MIN}m > lease TTL ${LEASE_TTL_MIN}m"
elif [ "$NOTREADY" -eq 0 ] && [ "$P503" = "0" ]; then
# The claim is "survived a lease rotation". A clean window shorter than the
# lease does not establish it, and saying so anyway is the failure mode that
# errs toward reassurance — the one that ships.
echo "RESULT: INCONCLUSIVE — clean, but uptime ${UPTIME_MIN}m has not yet passed the ${LEASE_TTL_MIN}m lease"
else
echo "RESULT: FAILED"
fi

13
tools/smoke.sh
View file

@ -55,8 +55,19 @@ esac
SERVICE_SMOKE=${SERVICE_SMOKE:-$HOME/canned-prompts/service/tools/smoke.py}
if [ -f "$SERVICE_SMOKE" ]; then
echo "--- service-level (${SERVICE_SMOKE}) ---"
# Wait for a ready endpoint before forwarding. Port-forwarding into a pod
# that is still rolling reports "connection closed" for every service-level
# check — a false failure that looks exactly like a broken service.
for _ in $(seq 1 30); do
[ "$(kubectl -n "$NS" get deploy canned-prompts -o jsonpath='{.status.readyReplicas}' 2>/dev/null)" = "1" ] && break
sleep 2
done
kubectl -n "$NS" port-forward svc/canned-prompts 18000:8000 >/dev/null 2>&1 &
PF=$!; trap 'kill $PF 2>/dev/null || true' EXIT; sleep 3
PF=$!; trap 'kill $PF 2>/dev/null || true' EXIT
for _ in $(seq 1 15); do
curl -s -m 2 http://127.0.0.1:18000/healthz >/dev/null 2>&1 && break
sleep 1
done
python3 "$SERVICE_SMOKE" --base http://127.0.0.1:18000 --expect-migration "${EXPECT_MIGRATION:-0002}" || FAILED=$((FAILED+1))
else
check "service-level-checks" "canned-prompts checkout not found at $SERVICE_SMOKE"