railiance-platform/argocd/platform-addons/secret-annotation-guard/README.md

Secret annotation guard

Implemented under the existing CUST-WP-0073-T03; no new platform workplan. Native admission policy denies the last-applied annotation on Secret CREATE and UPDATE, including an empty value. No new workload or controller.

Before the first manual sync, run scripts/secret_annotation_maintenance.py on railiance01 through the supervised admin path, first without arguments, then with --clean. It removes only the duplicate annotation; it never prints kubectl output or Secret values. Concurrent Secret churn can stop cleanup; inspect the receipt before retrying. Preserve only names, counts and booleans.

Declare the policy through the pinned secret-annotation-guard Argo Application; automated sync and prune are off. Platform-addons AppProject must allow both admission kinds. Sync policy first and inspect typeChecking; bind only after cleanup. Test clean CREATE/UPDATE and denied annotated CREATE/UPDATE using synthetic data in whitehat; verify client-side apply is denied, then delete only the test fixture. Secret writers must use server-side apply or replace.

Rollback is a manual Argo sync of a reviewed revision without the binding (with explicit resource-scoped pruning), or attended break-glass deletion of the binding followed by Git reconciliation. Removing the binding reopens this leak path. The policy does not rotate credentials or isolate agent accounts.

September 28 rollout and rollback

Policy source 800cbfa and Application declaration 54885ac were deployed through manual, resource-scoped Argo sync. Native type checking passed. Nine synthetic checks passed: clean creation/update/server apply, rejection of annotated create/update including empty values, client-side apply rejection, and fixture cleanup.

Enforcement was rolled back when ESO v0.16.1 targets stopped refreshing. That version copies ExternalSecret metadata when no target template exists; 31 current ExternalSecrets have no template. Removing annotations from targets alone cannot stop the controller adding them back. All 39 ExternalSecrets recovered after binding deletion; failed controllers were explicitly refreshed. The rollback revision 6016f72 left the policy UNBOUND and excluded the binding from kustomization. That revision remains the immediate rollback target.

Re-enablement prerequisite: add explicit metadata templates in the 31 owning declarations, preserving intended labels/annotations except last-applied; apply through owner paths; verify actual ESO refresh with annotation-free target Secrets. Retain existing data templates and remote references. Do not waive ESO from the policy or upgrade the controller as a shortcut. Rerun cleanup, native admission tests and an ESO refresh integration test, then include the binding and update the pin.

The absent platform-pg-drill namespace still has an orphan Secret drill-minio, a Deployment referencing it, a PVC and Service. Metadata patch fails because the namespace is absent. No orphan object was deleted or namespace recreated. Its disposition stays in CUST-WP-0073-T03; do not report cluster-wide cleanup complete.

Source for the diagnosed behavior: https://github.com/external-secrets/external-secrets/blob/v0.16.1/pkg/controllers/externalsecret/externalsecret_controller_template.go

Detailed receipts: the-custodian/docs/evidence/2026-09-28-secret-annotation-*.json.

Corrected writer rollout

On September 28 the founder authorized continuing the 31-declaration correction. Metadata-only templates are committed and published across the 12 owning repos. Server dry-run proved all 31 changes preserve the remaining ExternalSecret spec. Direct owner resources used metadata-only SSA; Target Revenue used a selective Argo sync of its ExternalSecret, with no migration/bootstrap hooks. All 39 ExternalSecrets completed fresh successful refreshes before binding activation.

binding.yaml is included for the reviewed re-enablement. After syncing, repeat the native admission proof and require all 39 ExternalSecrets to complete fresh refreshes with the binding present. Keep the safe annotation scan receipt and report the absent-namespace orphan separately. No Secret values are read or rotated by the metadata maintenance. ESO uses its normal credential refresh path.

Detailed preflight, publication and before/after integration receipts live in the-custodian/docs/evidence/2026-09-28-eso-*.json. The earlier rollout failure and rescue remain recorded; remediation does not count as unchanged success.