# Secret annotation guard: rollout, failed integration and rollback

CUST-WP-0073-T03, September 28, 2026. The founder authorized continuing the
existing workplans. This receipt records an unsuccessful rollout with recovery;
it is not evidence for promoting the agent to autopilot.

## Source and deployment

The platform owner source added the native policy/binding and safe maintenance
helper in `railiance-platform@800cbfa870661d47bee34c747f04f6145875adf1`.
Application commit `54885ac1589074a68d9607d1be250c84c5c2307a` pinned that revision.
The AppProject allowlist gained only the two admission kinds. Publication used
repo-manager; deployment used manual Argo resource-scoped sync, without syncing
unrelated root changes. The application has no automated sync or finalizer.

Policy type checking passed. The first synthetic proof failed on an SSA field
ownership conflict; its failed receipt is retained. The corrected proof tests
SSA of the same value followed by a clean update. All nine native checks passed:
clean create/update/SSA; annotated create/update rejected, including empty
values; client-side apply rejected; synthetic fixture removed. These checks did
not establish compatibility with existing controllers.

## Actual integration failure

ESO v0.16.1 copied the ExternalSecret source last-applied annotation back onto
its target when no target template existed. Under Deny enforcement, required
Secret refreshes failed (ten ExternalSecrets observed in SecretSyncedError).
31 live ExternalSecrets lacked an explicit template. The implementation is
visible in the [installed-version upstream source](https://github.com/external-secrets/external-secrets/blob/v0.16.1/pkg/controllers/externalsecret/externalsecret_controller_template.go):
without a target template the controller copies source metadata; with one it
uses template metadata. Merely removing annotations from the targets does not
fix this writer behavior.

The binding was deleted promptly to restore refreshes. Failed ExternalSecrets
were explicitly force-refreshed. All 39 became Ready. Source rollback commit
`6016f72a8db26df1eed7b7b9f3b36c8a0eb9f3b7` removes the binding from kustomization
and keeps it in `binding.pending.yaml`. Application commit
`c5d65b0b405be9ce5624f49c6f6df54c12b38735` pins that policy-only revision.
Both were published using repo-manager; the root was synced only for this
Application, then the child synced to its policy-only revision.

Final read-back at 13:17:32 UTC: application Synced/Healthy, operation Succeeded,
binding absent, 39/39 ExternalSecrets Ready. See
`2026-09-28-secret-annotation-rollback.json`. The policy remains installed but
UNBOUND. A normal sync of the pinned source cannot re-enable enforcement.

## Cleanup and limits

The recorded passes removed the annotation from 49 distinct Secrets in active
namespaces. Some were cleaned again after ESO recreated the annotation. This is
not a claim that all remain annotation-free after rollback. The helper changed
only that metadata key, never Secret data. ESO recovery performs its normal
refresh behavior; no credential rotation was performed by this work.

`platform-pg-drill/drill-minio` is orphaned: its namespace is absent, so the API
refuses the metadata patch. A referencing Deployment, a PVC and Service also
remain without that namespace. No orphan was deleted or namespace recreated.
Cluster-wide cleanup is incomplete. Disposition remains under existing T03.

Kubectl subprocess output was captured and suppressed throughout the helper.
The old raw presence template failed on absent annotations; its error was
suppressed rather than exposing a Secret dump. The replacement iterates keys
and handles absent/empty maps. Orientation §6 now requires the safe helper.

## Remaining work in T03

Before re-enabling the strict guard, explicitly set target metadata in the 31
owning ExternalSecret declarations while preserving intended labels/annotations
and all existing data templates. Verify actual controller refresh and clean
resulting target metadata, repeat cleanup and synthetic checks, then verify
ESO refresh with enforcement enabled. No ESO exemption or controller upgrade
is proposed. The owner source changes and orphan disposition remain unfinished;
no additional workplan or task has been created.

Receipts alongside this file: `secret-annotation-cleanup.json`,
`secret-annotation-cleanup-retry.json`, `secret-annotation-cleanup-active.json`,
`secret-annotation-post-binding.json`, `secret-annotation-admission-proof.json`,
`secret-annotation-admission-proof-final.json`, and
`secret-annotation-rollback.json`, all prefixed `2026-09-28-`.

## Corrected rollout — September 28 follow-up

The founder instructed “Good, go on” after the 31-declaration remediation was
identified. Explicit target metadata was added to 31 ExternalSecrets in 23 files
across 12 owning repositories. Source labels and intentional annotations remain;
controller bookkeeping and last-applied are not inherited. Data mappings, data
templates, store references, creation/deletion policies and refresh intervals
were unchanged. Server-side dry-run and semantic comparison verified this for
all 31. A canary refreshed successfully and removed its copied annotation.

All source changes were committed and published. Eleven repositories have exact
primary repo-manager receipts. activity-core's repo-manager registration refused
pre-existing historical workplan IDs; its documented `statehub fix-consistency`
path passed with warnings and pushed the exact metadata commit, without changing
those historical file IDs. See `2026-09-28-eso-metadata-publication.json` and
`2026-09-28-eso-metadata-changes.json`. Required user-engine checks passed (four
tests); telemetry pinned-chart fetch/check and family validation passed (one
pre-existing declaration warning).

Thirty non-Argo-managed ExternalSecrets received metadata-only server-side
apply through the existing SSH admin path. Target Revenue's ExternalSecret used
a selective Argo sync at `a3a8a27a8cda32ae3c7b55353ee16a131c50a0a0`, declared by
platform commit `d2631f6112fca8f374b37aa52bc34400c12c95e1`. The operation result
lists only that ExternalSecret; no migration/bootstrap hook or workload rollout
was run. Its application is Synced and Healthy.

All 39 ExternalSecrets completed fresh successful refreshes before enforcement.
A complete active-namespace scan checked 257 Secrets and found no last-applied
annotations; ESO had removed the formerly inherited metadata. The absent-namespace
orphan remained separately reported.

Guard source `7daf7e90675b21c8f9556028e54aa8c0a13fd9f0` includes `binding.yaml`.
Application commit `db51ec802801ddf85a80bf81980865c9f9239839` pins that source.
Both were published through repo-manager. Selective root/child Argo sync enabled
the binding without unrelated app changes. At 14:30:57 UTC the guard was
Synced/Healthy, the binding was present with Deny, and policy type checking was
clear at observed generation 1. Nine native admission checks passed again.
**All 39 ExternalSecrets then completed fresh successful refreshes under Deny.**
Receipts: `2026-09-28-secret-annotation-enforced.json`,
`2026-09-28-secret-annotation-admission-proof-reenabled.json`, and
`2026-09-28-eso-refresh-{before,after}-binding.json`.

The old rollout failure remains a failed original proposal requiring refinement
and recovery. Successful remediation does not retroactively earn unchanged
execution credit. No agent promotion, credential rotation or interactive-runtime
cutover is claimed.

The remaining orphan is the August 13 Secret
`platform-pg-drill/drill-minio`, UID `2fcb66df-d90f-4776-8ea0-8ca1a04bd307`.
The namespace is absent and has no pods. A reviewable UID-bound proposal to
delete only that Secret is in `docs/changes/CUST-WP-0073/orphan-secret-deletion.json`.
Explicit deletion approval is pending; the bound 3 GiB PVC and all other resources
are outside that proposal. No deletion has occurred.

### Approved orphan cleanup

The founder explicitly approved deleting only the orphan Secret. The API accepted
the DELETE with UID precondition `2fcb66df-d90f-4776-8ea0-8ca1a04bd307`; a fresh
identity inventory verified absence. The 3 GiB PVC retained its exact UID,
resourceVersion, volume and Bound status. No other resources or the namespace
were changed, and no Secret values were read or archived. Receipt:
`2026-09-28-orphan-secret-deletion.json`. This resolves the cleanup exception
and completes CUST-WP-0073-T03; the earlier pending-deletion text is historical.

Final complete cluster-wide scan after deletion: 257 Secrets checked, zero
forbidden annotations, zero orphan exceptions, helper exit 0. Receipt:
`2026-09-28-secret-annotation-final-scan.json`.