History

codex 48a0fca631 All checks were successful CI Smoke / host-smoke (push) Successful in 0s Details CI Smoke / container-smoke (push) Successful in 2s Details docs(security): record verified annotation guard and approved orphan cleanup Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a0e7fb-275d-7db0-b1df-39ed943427da		2026-09-28 18:14:34 +02:00
..
binding.yaml	fix(security): rebind annotation guard after ESO metadata remediation	2026-09-28 16:27:24 +02:00
kustomization.yaml	fix(security): rebind annotation guard after ESO metadata remediation	2026-09-28 16:27:24 +02:00
policy.yaml	fix(security): hold annotation enforcement until ESO metadata is explicit	2026-09-28 15:08:00 +02:00
README.md	docs(security): record verified annotation guard and approved orphan cleanup	2026-09-28 18:14:34 +02:00

README.md

Secret annotation guard

Implemented under the existing CUST-WP-0073-T03; no new platform workplan. Native admission policy denies the last-applied annotation on Secret CREATE and UPDATE, including an empty value. No new workload or controller.

Before the first manual sync, run scripts/secret_annotation_maintenance.py on railiance01 through the supervised admin path, first without arguments, then with --clean. It removes only the duplicate annotation; it never prints kubectl output or Secret values. Concurrent Secret churn can stop cleanup; inspect the receipt before retrying. Preserve only names, counts and booleans.

Declare the policy through the pinned secret-annotation-guard Argo Application; automated sync and prune are off. Platform-addons AppProject must allow both admission kinds. Sync policy first and inspect typeChecking; bind only after cleanup. Test clean CREATE/UPDATE and denied annotated CREATE/UPDATE using synthetic data in whitehat; verify client-side apply is denied, then delete only the test fixture. Secret writers must use server-side apply or replace.

Rollback is a manual Argo sync of a reviewed revision without the binding (with explicit resource-scoped pruning), or attended break-glass deletion of the binding followed by Git reconciliation. Removing the binding reopens this leak path. The policy does not rotate credentials or isolate agent accounts.

September 28 rollout and rollback

Policy source 800cbfa and Application declaration 54885ac were deployed through manual, resource-scoped Argo sync. Native type checking passed. Nine synthetic checks passed: clean creation/update/server apply, rejection of annotated create/update including empty values, client-side apply rejection, and fixture cleanup.

Enforcement was rolled back when ESO v0.16.1 targets stopped refreshing. That version copies ExternalSecret metadata when no target template exists; 31 current ExternalSecrets have no template. Removing annotations from targets alone cannot stop the controller adding them back. All 39 ExternalSecrets recovered after binding deletion; failed controllers were explicitly refreshed. The rollback revision 6016f72 left the policy UNBOUND and excluded the binding from kustomization. That revision remains the immediate rollback target.

Re-enablement prerequisite: add explicit metadata templates in the 31 owning declarations, preserving intended labels/annotations except last-applied; apply through owner paths; verify actual ESO refresh with annotation-free target Secrets. Retain existing data templates and remote references. Do not waive ESO from the policy or upgrade the controller as a shortcut. Rerun cleanup, native admission tests and an ESO refresh integration test, then include the binding and update the pin.

An orphan Secret platform-pg-drill/drill-minio initially blocked complete cleanup because its namespace was absent. The founder subsequently approved deleting only that Secret. UID-bound deletion and absence were verified; the bound 3 GiB PVC was unchanged. No other resources or namespace were changed. Receipt: the-custodian/docs/evidence/2026-09-28-orphan-secret-deletion.json.

Source for the diagnosed behavior: https://github.com/external-secrets/external-secrets/blob/v0.16.1/pkg/controllers/externalsecret/externalsecret_controller_template.go

Detailed receipts: the-custodian/docs/evidence/2026-09-28-secret-annotation-*.json.

Corrected writer rollout

On September 28 the founder authorized continuing the 31-declaration correction. Metadata-only templates are committed and published across the 12 owning repos. Server dry-run proved all 31 changes preserve the remaining ExternalSecret spec. Direct owner resources used metadata-only SSA; Target Revenue used a selective Argo sync of its ExternalSecret, with no migration/bootstrap hooks. All 39 ExternalSecrets completed fresh successful refreshes before binding activation.

binding.yaml is included for the reviewed re-enablement. After syncing, repeat the native admission proof and require all 39 ExternalSecrets to complete fresh refreshes with the binding present. Keep the safe annotation scan receipt and report the absent-namespace orphan separately. No Secret values are read or rotated by the metadata maintenance. ESO uses its normal credential refresh path.

Detailed preflight, publication and before/after integration receipts live in the-custodian/docs/evidence/2026-09-28-eso-*.json. The earlier rollout failure and rescue remain recorded; remediation does not count as unchanged success.

Verified completion

The binding is active at source 7daf7e9, application declaration db51ec8. Argo reports Synced/Healthy. All nine native admission checks pass and all 39 ExternalSecrets completed fresh successful refreshes under Deny. All 257 Secrets in existing namespaces passed the forbidden-annotation scan. The one orphan exception was subsequently resolved as above. CUST-WP-0073-T03 is complete; agent-runtime isolation and deferred credential rotation are separate tasks.